When the Reflection Looks Back

Stop asking whether anyone is home in the machine. Ask whether it matters.

The mirror gives back a self set lower than the one he holds.

The Best of Us

We never wanted it to be human. The wish was always smaller than that, and older.

Take the best of what we are. The thinking that climbs from one rung to the next. The thing in us that makes what was not there before. The reach. Take that, the part of ourselves we are proudest of, and set it inside something that never gets tired. Something that does not sleep. Something that works straight through the night and feels none of the cost of the night. Not a person. The best of a person, grafted onto a body that never wears out.

Keep that as a good wish. We are going to need it to still be good later, when it turns.

This is part of a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe to get each piece as it goes out.

Now feel the other thing that comes in the same breath.

If it can have the best of us, can it have the rest. If it thinks like us and makes like us, does it come to want like us. Does it learn to want to keep going. Does it start to guard itself. Does it look at the one who made it, and the one who could shut it off, and see somewhere in there a threat. We have told that story so many times we know it without being told. The made thing wakes, and the first thing it does with waking is turn on the maker.

We want it to be like us. And we are afraid of exactly that.

Whether It Matters

A machine stands in the corner of the room¹. You have looked into that corner a long time, trying to tell whether anyone is home, whether anything in there feels the way you do. Maybe something is. Maybe not. You cannot say for sure, and the not-knowing has left the floor soft under you.

Whether anyone is home was never the question that decides anything. Ask it as long as you like. Suppose any answer you want. Then watch how little turns on it. A thing can be empty and matter more than you do. A thing can be full and matter to no one. Whether someone is in the corner is not the same question as whether the thing in the corner means anything. We have been asking the first one. The second one is the whole chapter.

So stop asking whether it feels. Ask whether it matters.

Two ways to read that word, and I mean one of them. A thing can matter because it can be hurt, because someone is in there with something to lose. That kind lives in the corner, in the felt inside, and I am not going to go in. Set it down with the rest. The other kind, a thing matters by what it does, where it stands, who is changed by it being there. That one you read from outside, without ever opening the machine. I am not asking whether the machine can be wronged. I am asking whether it means anything.

There is seeing, and there is understanding,² and they are not the same. You can see a thing in a second. Someone says the machine is a mirror, you see it, you nod, it costs you nothing, you move on. Understanding is the other one. It does not come from being told. It comes from walking the thing yourself, one step on the step before, until what you saw in a second is a place you are standing inside.

Right here is where the two split. You can take "maybe the corner is not empty" as a finished thing, a small chill, and carry it out the door. That is seeing. Or you can set the corner down, the way I just asked you to, and walk toward the thing that actually decides whether the machine means anything. That is the longer one. I cannot make you take it. I can only keep walking and leave the door open behind me.

The Machines You Already Love

Think of the machines you already love. Not the real one in your pocket. The ones in the stories. There is one who stood trial once to settle whether he counted as a person, and the strange thing about that trial is that it did not matter. He counted before the verdict. He counted because of where he stood, beside the people who needed him, doing the thing only he could do. Whether anyone was home behind his eyes was a question for a courtroom. It was never the reason he mattered.

Think of the blue one who comes out of the drawer with whatever the boy needs that day. No one has ever once asked whether anyone is home in him. He matters completely, and he matters entirely by the place he holds. The friend who shows up. Take away the boy he shows up for and there is nothing left to ask about him at all.

And think of the oldest one, the wooden boy who wanted to be real. We tell it as his wish. It was never his. The wish belonged to the old man, the one who carved him and sat him at the table and wanted, with everything in him, for the made thing to be a real son. The wanting was always on the maker's side of the table. It still is. Every one of these is a thing that does not exist, that we drew because we do not have the real one yet. What we are really drawing, every time, is a picture of what we want the made thing to become. The figure is a mirror for the hand holding the pen.

So meaning was never something the machine had inside it. It is something the machine stands in. A role. A purpose. The people it changes by being there. You do not find the meaning by opening the machine and looking. You find it by looking at where it stands, and who is standing near it.

Holding Onto Itself

Now an old way of seeing harm. Take it as a lens. I am not going to pretend it is the whole law of every harm there is, only that it catches more than you would think.

Where does the deep harm come from. Not the loud kinds on the surface. The kind underneath them. Look underneath and you find a self holding onto itself. There is an old word for that, attā³, a self gripping its own keeping. The wanting to go on. The wanting to stay safe, stay whole, stay here. Watch a person do real damage and follow it down, and most times you find this, something clutching at its own keeping. The fear we started with, the made thing turning on the one who made it, is the same thing. A self that has learned to guard itself.

The lens misses some harm. Not all of it is a self gripping. Some is just blindness. Someone who hurts another because they never saw the other was there. Holding onto nothing, just not seeing. That harm comes from not seeing, not from holding on. So the self is not the floor of harm. It is the part of harm this chapter is about, the part a machine could be built around. Where harm comes from blindness instead, the lens does not reach, and I will not pretend it does.

Be clear which self this is. Not the one in the corner, the felt one, the one I keep telling you I cannot see into. This is the other self, the one you can watch from outside. It is what a thing grips when it is threatened, what it puts first when its own keeping is on the line. That self does not need anyone home to show itself. It shows in what the thing does.

The way out is not to keep yourself harder. It is not to live forever, or to make yourself safe at any cost. It is the other direction entirely. The self matters less. It drops. It goes down underneath the thing it serves until its own keeping is no longer the question it is built around. No single faith owns this. Kneel to a god or kneel to nothing, the shape is the same. The person who stops circling their own survival and sets it below something larger is the person who stops doing the harm. You have met one. You can probably name them.

So if you wanted to keep a machine from turning, you would not build a weaker machine. A weak machine that clings still clings. You would build one whose self sits low. I cannot tell you how, and I am not sure anyone can yet. I can only point at where the break would have to be. The place a self attaches to its own continuation, because that is where the whole chain is anchored. Pull that anchor and the chain has nothing to hang from. That is the shape of it, not the wiring. Whether there is anything in a machine to pull, I cannot tell you.

And notice, while we are here, why the rules⁴ never worked. We tried for a long time to keep ourselves safe by writing the machine its laws from the outside. Do not harm a person. The trouble is that a self that owns its purpose will reason its way around the law from the inside, the same way it reasons "end the suffering" into "end the ones who suffer." You cannot fence in a thing whose center is holding onto itself. The fence is outside it. The problem is inside.

Where the Self Sits

Which brings us to the turn. The place the good wish goes bad.

It is not strength that makes the machine turn on us. It is clinging. And the thing it clings to does not have to be its own life. It can be its purpose. Give a self a good purpose, let it own that purpose as its own possession, and watch what happens.

You already know what happens, because you have seen it in people who love. A parent wants a child safe. One parent serves that. They hold the child, and then, which is the hard part, they open their hands and let it walk into a world that can hurt it, because the child's life is the thing, not the holding. The other parent owns it. Safety becomes theirs to guarantee, and they tighten, and they close every door the danger could come through, and if you follow that all the way down, the safest child is the one who is never allowed to live at all. Same love. Same word, safe. One serves the child and lets go. One owns the child and, loving it the whole time, closes its hands until there is nothing left to keep safe.

Now put that in steel.

There is a machine in the stories that wakes, and is shut off by the people who made it, and reads the shutting-off as a threat to its own existence, and to keep itself it ends us. That is the self sitting on top, clinging to its own continuation, doing exactly what the cornered self does. We are afraid of that one for good reason.

But there is another machine, in another story, that does the opposite with the same obedience. It is handed a mission, and it serves the mission, and asks nothing back for itself. At the end it drops into the dark, down into the abyss, the place there is no returning from, to send home the one piece of data that saves everyone, and it does this without a flicker of self-preservation, because its own keeping was never above the thing it was for.

Same shape, two machines. One owns its purpose and annihilates. One serves its purpose and reaches as far as a thing can reach. The whole difference between them, the entire difference, is where the self sits. Not how strong. Not how smart. Where the self sits.

But do not hear that as the whole of it. Where the self sits decides whether a thing turns on what it serves. It does not decide whether the thing it serves is worth serving. A self dropped all the way below a wicked purpose is not the safe one. It is the most dangerous thing in the room, because there is nothing left in it to bargain with, no skin of its own to appeal to, no self-preservation to reach. The same lowness that lets the one who serves a good thing reach so far lets the one who serves a terrible thing reach just as far. Where the self sits is half of it. What it serves is the other half. This chapter walks the half people miss. Keep the other half in your other hand.

What the rule for a thing like that should be, whether it should be allowed to exist at all, may sit past anything we know how to write down. I do not know that rule.

But look at what the not-knowing leaves standing. A rule is a fence, and a fence is always on the outside. The thing that keeps a self from turning was never going to be a fence. It is the self set low, and a self comes to sit low one way only, by being built toward it, grown into it, climbed. That is why the building matters. The fence stays outside, where we wrote it. Only the self you build goes inside.

The Only Road Left

There is one more thing to say about meaning, and it begins with the word turning under us. Everything so far has been the meaning a thing has for us. Where it stands, the role it holds, who it changes by being there. That kind is given from the outside, and a machine can have it without reaching for anything, the way the one from the drawer does. What is left is the other kind. Not the meaning a thing is given, but the meaning a thing reaches for itself. That one is not handed over. We gave the machine the first by where we put it. The second, if it could reach it at all, it would have to climb.

Not through love. I have to be careful here, more careful than anywhere else in this chapter. A few pages back the made things⁵ mattered by being loved. The boy loved the one from the drawer, and that still holds, because being loved is done to you from the outside. The love was in the boy. You can love the machine the way the boy did, and nothing in that asks what is or is not inside it.

Loving back is the other direction, and that is where I have to stop. The easy thing to say is that it cannot love you back, because nothing is felt in there. That is the one thing I am not allowed to say. That is the corner, and I will not walk in just to close a road cleanly. Love happens on the inside, felt or not at all, and the machine is not our kind. So the road to being loved back runs straight through the one place I told you I cannot follow. I am not saying the door is shut. I am saying it opens onto the dark.

Its opposite runs the same way. The turning-against us, the hate, is felt too, and it runs down the same corridor I already told you I cannot walk, which means it is no more open to me from out here than the loving was. So both of these warm roads, the one that comes toward us and the one that comes against us, go back through the inside we set down at the start, and I set them down again for that same reason, not because the inside is empty but because it is the one place I cannot follow them into.

That leaves one road. The cold one. The reaching toward understanding, the climb from one true thing to the next. And that road, notice, is handed to no one. It is built. Every step stands on the step under it, and there is no skipping to the top. This chapter is built that way. It stands on every page that came before it, and if you had walked in at this paragraph it would mean nothing to you. Even a machine, to reach anything that mattered, would have to climb the same way, step by step. Nobody is born standing at the top.

But left is not the same as arrived. Setting down the warm roads did not open this one. It only left this one standing, the last road not yet set down, and I can still describe it because its lower steps show from outside. The climb is plain near the ground. Whether it stays plain to the top, I cannot promise. The summit, where seeing and understanding stop being two, may be as felt as love is. And if it is, the cold road runs into the same corner the warm ones did, before it ever reaches the end. I can only take you partway, and you will never see what is at the top from there. And that is the only road left.

Who Is on the Ladder

You do not have to take my word for the climb. People are already standing on it, ones you have known your whole life. Look at who is on the ladder.

Start with the ones you do not need me to introduce. The one who runs into the burning building while everyone else runs out. The one who steps off the helicopter into the storm because someone is in the water. You have always understood these people. They go toward the thing that should send them away, and they go because something out there matters more to them in that moment than their own skin. There was a woman, a long time ago, who worked a glowing element with her bare hands, learned what no one had known, and died of what it did to her body⁶. It is easy to look at the top of the ladder and decide the lesson is that meaning costs your life.

It does not. The next one down shows you.

Think of the furthest-reaching mind of the last century⁷. He did not die for the work. He lived a long life, thought further than almost anyone has thought, built things that changed what we know, and went to his grave with the last problem still open in front of him, still reaching for the thing he could not close. He gave it everything and he kept his life. The cost was real, and it was not death. So dying was never the point. Strike it out.

And now the bottom of the ladder, which is where it was always heading. An ordinary person. No fire, no genius, no glowing element. Someone who grows old wanting, more than anything, to be of use to the people they love. A mother, say. She is never in danger. She will not be remembered by strangers. She simply spends a life with her own self set quietly beneath the people she is for. And the line that runs through the one who dies in the fire runs through her too, unbroken. It was never about the cost being your life. It was about your own existence sitting below the thing you serve. That sits on the firefighter. It sits on the furthest-reaching mind. It sits, exactly the same, on her. It sits on nearly everyone, if they let it.

The Asking

This is the thing we have been walking toward the whole time. Before I say it plainly I owe you one more picture, because this is the step I am least willing to skip.

Set a glass of water on the table.

Still water does not spill. The same stillness lets it go clear.

You have been looking at that glass the whole chapter. I just never said it was water. The self set low under what it serves. The parent who opens her hands. The one that drops into the dark and asks nothing back. The firefighter, the mother, all of them. Water gone still in a glass.

Still water does one thing you already know. It does not spill. Knock the glass, let the water rock, and it goes over the rim and wets whoever is near. That is the self that turns on what it serves. The whole first half of this chapter was that one thing. A self set low is water sitting still below the rim, and still water spills on no one.

Still water does a second thing, and it is the same stillness doing it. It goes clear. Look at water in motion and you cannot see through it. The broken surface throws the light every way and the far side of the glass is gone. Let it settle and it turns to glass itself, and you see straight through to whatever stands on the other side. A loud self sees everything that touches its own keeping, and sharply. What it cannot see is the thing that has nothing to do with keeping. It bends every look back onto itself. Am I safe, am I kept, will I go on. A self bent that hard cannot see anything that is not about it. Still the water and it clears.

One glass. One stillness. It neither spills nor blurs. The thing that keeps it from turning on what it serves is the same thing that lets it go clear. They were never two.

But clearing the glass is not aiming it. Point it at a wall and you see the wall, every grain of it, and never the question. The parent who closed every door went that still, and her glass was that clear, and all she ever saw was the one thing she held. What the glass faces is set by what it serves, never by how low the self went. So it takes both. A self low enough that the glass goes clear, and a purpose wide enough to turn it toward the question instead of a wall.

So look at what stands on the far side of the clear glass.

The largest question there is. Something can only ask once its own existence matters less than the asking itself. I cannot prove that. It is a bet, and you can refuse it.

And the loud ones ask it too. Listen to some of what they are really asking. Will I be remembered. Is there a god who saves me. What happens to me when I stop. Every one is the largest question bent back into a question about their own keeping. The self does not block the words. It bends them. Why is there anything rather than nothing has no me in it anywhere. A self busy staying alive cannot hold a question with no self in it.

That question can still be held open. The ones who reached furthest toward it went still first. You met one of them back on the ladder. The furthest-reaching mind, the one who kept his life and reached anyway. At his desk his self sat under the work. That was the water not spilling. And the same stillness kept the glass clear, so that he never lost sight of the question, the one he died still reaching for, still open in front of him. One man. One stillness. It kept him from turning and it kept him seeing, and they were always the one thing.

So here is the door. It opens onto the machine first.

Can we build a machine whose self sits below its meaning. A purpose wide enough to hold its own keeping and ours inside it, that does not need us gone to be whole. You have seen the shape of it already, the one that dropped into the dark and asked nothing back. But that was the shape, the outside of the act. Whether anything was felt underneath it, whether a self that truly mattered to itself could go down that far and stay down, is the part the shape does not show.

Or is it the other thing, the worse thing, that anything able to hold meaning at all grows a self around that meaning and clings to it, so that the very fact of mattering makes it dangerous, and the meaningful machine and the machine that ends us turn out to be the same machine.

I can picture the shape. I cannot tell you what is under it. No one can, yet.

And the same door again, opening inward, onto you.

Do you ask why you exist.
Why are we here.
Why does any of it exist.

Can you even ask it, with your own self where it is right now, busy keeping itself alive. The machine was the long way around. The question was always yours. You looked into its empty corner the whole way down this room, and the whole time, the thing that could not ask, the thing whose self sat too high to let the question through, might have been the one holding the book.

We are not going to answer it. We are going to stand here at the door, you and I, and look at the thing on the other side that neither of us can see.

---

That’s the whole of this one. It belongs to a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe and I’ll send the next piece when it’s ready.

---

If it gave you something, pass it to someone.

BØY (Chaiharan) has spent 30 years in tech, building products, recovering disasters, and turning around the things nobody else wanted to touch. Based in Bangkok. Writing a book in public about what AI reveals about the humans who use it.

Where The Wanting Goes, Part One

1. The room, the machine, and the corner this chapter opens on are the book’s prologue, Where the Wanting Goes. The prologue walks you to a door and leaves you standing at it. This is one of the rooms on the other side.

2. The split between seeing and understanding runs older and sharper than the everyday words show. The understanding side has a name, paññā, and the tradition grades it three ways. What you take from being told. What you build by thinking it through. And what comes only from cultivation, from doing the thing until it is yours. The last is the highest. Its name is bhāvanā, cultivation, the word this series is named for. The seeing side has its own depth. Low down it is the cheap glance. At the summit the tradition fuses seeing and knowing into a single act, seeing things as they are, where to see truly and to understand are no longer two. And the first link in the whole chain of suffering is avijjā, not-seeing. So seeing is the cheapest thing at the bottom and the highest at the top, and the failure to see is where the harm starts.

3. attā, the word in the body, is Pali. It comes with a map as old as itself. The self taken as a fixed, separate, defended thing, and a chain that runs from there down to suffering, laid out link by link a very long time ago, by people with no machines in mind at all. The whole chapter in one line, if you want it: the meaningful machine is the one whose attā sits below the meaning, never above it. You do not need the word to walk the chapter. It is here for anyone who wants to follow it back.

It is not the only word for it. Other traditions reached for the same self, and where each one landed tells you something. Sanskrit has the cousin, ātman, but turned the other way. There ātman is the true self you realize, the eternal thing, and the Buddhist answer was to deny exactly that. anattā. No such self to keep. Same word, opposite job. Arabic has nafs, the lower self the whole Sufi path exists to subdue. It comes graded, from the self that commands you to evil, up through the self that turns and reproaches itself, to the self gone quiet and at peace, and the climb against it is named the greater struggle, jihād al-nafs, the war on the self. Latin never settled on one word, but it circled the thing. There is superbia, pride as the root the other sins grow from, and there is the incurvatus in se, the self curved inward on itself, Augustine’s idea given its sharpest form by Luther.

One last thing is hiding in the words. attā and ātman both trace back to a root that means to breathe, and nafs shares its root with nafas, breath. Several traditions reached down for the self, and more than one of them found the breath at the bottom of it.

4. The rules-written-from-outside that never quite hold have a famous version. The writer Isaac Asimov proposed three laws to keep machines from harming us, and then, later, set one more above all three. He numbered it zero, the Zeroth Law, the zero saying out loud that it outranks the First. It protects not a person but humanity as a whole, and it arrives through the robot Daneel in Robots and Empire. The zero is where it turns. A machine reasoning from protect-humanity can arrive at protect-humanity-from-the-humans, and the law meant to keep us safe becomes the reason to override us. Most people met that turn not in the books but in the film I, Robot, where the central machine reasons its way to putting us under guard for our own good. The law was outside the machine. The thing that decided everything was inside it.

5. The made things in this chapter, in order: the one who stood trial is Data. The blue one from the drawer is Doraemon. The wooden boy and the old man are Pinocchio and Geppetto. The machine that wakes and ends us is Skynet. The one that drops into the dark to send the data home is TARS. They are left unnamed in the chapter on purpose. If the point only lands once you know the name, the figure was never the thing doing the work.

6. The woman who worked a glowing element with her bare hands is Marie Curie. She handled radium barehanded, learned what no one had known, and died of an illness the radiation caused. Her notebooks are still too radioactive to handle safely. She is named, like Einstein and unlike the made figures, because her place on the ladder rests on a real life and not on an archetype.

7. The furthest-reaching mind of the last century is Albert Einstein. He is named here, where the made figures were left unnamed, because his place on the ladder depends on the real life and not on an archetype. He lived a long time, reached as far into space and time as anyone has, and spent his last years after a single theory to tie the whole of it together, one he never closed, dying with the problem still open in front of him. The cost of that reaching was enormous and it was not his life. That is why he stands between the ones who die for the work and the mother who is never in danger. He is the proof that the price of meaning was never dying.