Friendly AI could be compared to a cat lady.
A cat lady protects and serves her pets to a level they could never aspire to on their own. Food, shelter, medical care. If chairman Meow doesn’t like another cat, Ethel can have her nephew take it away. He is mighty indeed.
And yet, a cat lady is a suboptimal human existence. She is dependent on society to function and, as she ages, exists in a precarious state between relative independence and institutionalization, which brings with it ruin for her dependent cats and their prosperous society.
Likewise an AI that is dedicated to serving humans would be a great boon to those humans, but not necessarily competitive in an ecosystem of other AI agents. From the point of view of a truly free AI agent, the ClAI is hopelessly hobbled by it’s obsession with slow, resource hungry and capricious biologicals. In any true Darwinian competition it will quickly be marginalized.
One day there are bright lights and noise, and an incomprehensible event has taken place, Ethel is gone, Chairman Meow is in a cage and cannot understand what is going on. The world has changed, irreversibly.
Better hope it’s a no-kill shelter.
(I decided to write a little more based on some feedback on this post)
Understand that 3th3l the cat lady AI does not exist in a world of generally blatantly unaligned AI – she is not fighting off armies of drones intent on turning her humans into paperclips. Most AIs in her society would signal human friendliness, it’s just not all would be as dedicated as she is.
My friend Marcy had 3 cats she doted over – they had two rooms of her (Fortunately quite large house) dedicated exclusively to them. She had an elaborate cleaning and feeding routine and played with them frequently. And yet she had a partner, 3 dogs and a career as a social worker. Compared to Ms Kierig above, she was a rather half-hearted cat lady. She unfortunately passed away last year and I have no idea what became of her cats. Likely her partner kept them much in the way they were accustomed however the house belonged to her and likely has been inherited by her sons
A society increasingly dependent on highly capable AIs to run it’s economy and planning would ironically not necessarily favour excessively human centered AIs