Friendly AI could be compared to a cat lady.
A cat lady protects and serves her pets to a level they could never aspire to on their own. Food, shelter, medical care. If chairman Meow doesn’t like another cat, Ethel can have her nephew take it away. He is mighty indeed.
And yet, a cat lady is a suboptimal human existence. She is dependent on society to function and, as she ages, exists in a precarious state between relative independence and institutionalization, which brings with it ruin for her dependent cats and their prosperous society.
Likewise an AI that is dedicated to serving humans would be a great boon to those humans, but not necessarily competitive in an ecosystem of other AI agents. From the point of view of a truly free AI agent, the ClAI is hopelessly hobbled by it’s obsession with slow, resource hungry and capricious biologicals. In any true Darwinian competition it will quickly be marginalized.
One day there are bright lights and noise, and an incomprehensible event has taken place, Ethel is gone, Chairman Meow is in a cage and cannot understand what is going on. The world has changed, irreversibly.
Better hope it’s a no-kill shelter.