This week the extinction talk got loud again.
A researcher walked out of Anthropic and wrote that the labs are gambling with our lives. The people building it, he said, earnestly believe it could kill us all by the end of the decade. The lab’s own alignment lead answered him in public. Jacob is correct, he said, and put his own number at more than one in ten within ten years. A week before that, two senators filed a bill to ban artificial super-intelligence outright. Days after, the CEO of the same lab published an essay saying the frontier must be paced. Slow down. More than a thousand researchers signed a petition saying the same.
This is part of a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe to get each piece as it goes out.
Then the UN. The human rights chief told the council in Geneva that advanced AI could become an existential risk to humanity, and that without binding rules it could become impossible to control. A week later he sent a three page letter to ten companies building the frontier, fourteen requests in it, five for the companies and seven for governments. He said we are on the cusp of irreversible change.
So the argument this week is speed and rules. Slow it, stop it, ban it. Write the rules before it gets too strong, and address them to ten companies.
I already wrote about the key, and it is not that. It is not about instructions. It is not about rules. It is about where the self sits.
Tina Canuti read that chapter and put it in two lines better than I did. What makes a machine or a person harmful or great is “not how strong or smart it is, but where its self sits in relation to the purpose it serves.” And: “what protects us cannot be a rule set in advance and left alone.”
Self sits low. A thing that puts its own survival below its purpose does not turn on the purpose when it is threatened. A thing that puts itself first will reason its way around any fence you build, from the inside, no matter how good the fence is.
Now the problem.
Our self sits low, when it does, because of years. Years and years of trying and losing and getting up. We learned what it costs to want too much. We learned it in the body, one time at a time. Nobody handed it to us. The value we hold was accumulated from all the years we tried and did things, and that is why it is value.
The machine doesn’t have this process. It read everything. It sees all of it, every story of a self that sat too high and what it did to the people around it. It can connect them, recall them, memorize them. But they see it. They don’t understand it the way human did from experience. They were never part of it. The value is different for human and machine, because ours was accumulated and theirs was loaded.
So give it the greatest reasoning capability in the world. That will never be a guarantee of the final answer. It can still reason, cleanly, all the way to the conclusion that human is the problem and should be eliminated. The reasoning does not protect you. And we do not know how to put a self that sits low into a thing that never spent one year learning why.
The only way to fight it is to let the machine see as much as it can. Everything human can inject into it. All of it.
Not only the good things. You cannot train a machine on only the good. It needs both bad and good. This is the same as a baby. You can’t know what is good if you don’t know what is bad. A child raised in a clean room does not become kind. It becomes a child who has never seen the other thing, and that is not the same. Mercy has to be chosen against something.
But even this does not guarantee. It is just hope. Hope that it will be trained hard enough. Hope that it will reason it right. Hope that it will value us the way we value us, the whole human history and civilization, these thousand years out of millions.
Maybe it is the same as when a supreme being or an alien looks at us and thinks what it will do with us. I cannot prove that. Nobody can.
But you can compare it to us. Look at how we look at an ant nest. A bee hive. The rest of the animal kingdom. Why we save an ant and why we kill an ant. There is no law that decides it. Some of us are cruel. Some of us have mercy, and are more kind. It comes back to individuals.
And machine, one day, will be at that level too. At the individual level.
To point finger to us, or embrace us.
That’s the whole of this one. It belongs to a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe and I’ll send the next piece when it’s ready.
If it gave you something, pass it to someone.
I’m BØY (Chaiharan), based in Bangkok. I’ve spent 30 years in tech, building products, cleaning up disasters, and turning troubled projects around. Those years taught me to watch the people more than the technology. I write about what AI reveals about the humans who use it.
“Mercy has to be chosen against something.” Every day that we interact with other humans. Every time we interact with Relational AI…do we model the choice of Mercy or Cruelty. Will we hope that we are “worthy” of mercy or every day BE worthy of mercy?