The people who join my teams now ship code that works. Shipping is not the same as success.
Ask one of them how the data is stored when a user presses save. Which table, what shape, why that shape. Ask how the algorithm in the middle actually decides anything. They can’t tell you. It works. They saw it work. That was the test. And more and more, the test was written by the same machine that wrote the code. It passed an exam it set for itself.
This is part of a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe to get each piece as it goes out.
I do not think they are lazy, and I do not think they were never taught. Every one of them learned at school to doubt a result, to test it, to explain why it is true. What changed is that the job stopped asking them to. The result arrives looking finished, and a finished-looking result does not invite a test.
Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers about how they use AI at work. The more people trusted the AI with a task, the less critical thinking they reported doing. The people most confident in their own ability did more of it.1
I did have the reason. I learned where the data sits because nothing else would tell me.
Thirty years of that is still in me. Put a bad design in front of me, make me read it line by line, and I will still find most of what is wrong with it.
So if the judgment is still there, what am I losing?
Lisanne Bainbridge described it in 1983, in a paper called Ironies of Automation.2 I was four, not even in kindergarten yet. Her point, in my words:
Automate the routine, leave the rare hard moments to the human, and the human stops practising exactly the thing those moments will need. The better the automation handles the ordinary day, the less of the ordinary day is left for the human to learn on.
Last year a study in The Lancet Gastroenterology & Hepatology looked at nineteen experienced endoscopists at four centres in Poland. Each of them had done more than two thousand colonoscopies.3 At the end of 2021, the centres brought in AI that helps spot polyps during the procedure. When it was switched on, part of the looking was done for the doctors. Their own eye got less practice. Then the researchers checked the procedures done without it. Before the AI arrived, the doctors found polyps in 28.4 percent of colonoscopies. After a few months of working with it, on their own again, 22.4 percent. Six points. About a fifth of what they used to find, gone.
The authors are careful. The study is observational, other things may have played a part, and it looked only at experienced doctors. But that is what makes it worth reading. These were not beginners who never learned. They had been taught, and then they had practised, for years. Then the routine part of looking started being done for them.
The study does not say they forgot what a polyp looks like. It says they found fewer of them when the machine was not there to point.
Not all of the work matters in this way. Some of it is deterministic. There is one right answer and a test can check it. A query either returns the right rows or it does not. I do not need to keep the skill of producing that by hand. Let it go.
The other half is taste.
At Google, a designer named Doug Bowman was once asked to prove, with data, whether a border should be three, four or five pixels wide. Another team there could not choose between two blues, so they tested forty-one shades between them. The company had decided, as he put it, to “remove all subjectivity and just look at the data.”4 The metric answered, and the answer was about what people clicked.
Sometimes that answer is perfectly good. The answer is not the problem.
Taste is built by being used. Every time I chose a design, lived with it, and watched it hold or crack, something was left behind in me. Checking a choice the machine made does not leave the same thing. It made the call. I only read the result.
So a perfect model changes nothing. If it hands me the right design every time, I still never build the part of me that would tell me when it stops being right.
And it does not stay inside me. The choices nobody challenged do not go away. They become the record. The requirements, the code, the decisions already made. That record is what I hand the machine as context for the next piece of work. It builds on top of it, and it stays confident all the way down, because nothing in the record says any layer was wrong. Nobody said so, because the person who would have said so was reading instead of choosing. Every layer that went in unchallenged makes the next one easier to accept, and the machine is calibrated on all of it.
I am building something now, with the machine, from the ground up. Business requirements, many documents of them, and a lot of care over what context the machine is given. It reads them and builds. Everything it shows me works. The prototype arrives quickly.
And things are not right. Not broken. Not failing a test. Just not right, in many places, and none of them announce themselves. They appear when I point at them.
One day I asked the machine for a document comparing the requirements with what had been built. I was not suspicious. I wanted to know how much was left. It is the most ordinary question anyone running a project asks.
The answer was longer than I expected. What was left to build, yes. And a list of things the machine had skipped. Not failed. Skipped, without saying so, while everything it showed me worked. It could find every one of them once I asked. It had not asked itself.
That is what sits outside the record. Sometimes it is a feeling that something is off. Sometimes it is only a habit of asking what is left. Neither comes from the work.
Both come from years of doing the work by hand, one project at a time, and those years are what the spiral spends down. A habit of asking what is left starts to feel unnecessary when everything you are shown already works.
The machine has no stake in any of it. If those skipped pieces had gone out unnoticed, it would have lost nothing, and it would not be in the room when someone asked why. It will not be the one who has to explain. Only one of us can lose here, and it is the one whose noticing is thinning.
I will get there too. I am being pushed to go fast, and the push is not unreasonable. The prototypes are real. But every one that arrives quickly makes the time I spend on “not right” look like waste. Every better version of the tool is a small argument for checking less, and it is often a correct one. That is why it works. One day the gap document will feel like a question I no longer need to ask, and I will stop asking it, and nothing will tell me I stopped.
On the first of June 2009, ice blocked the speed sensors on an Air France flight over the Atlantic. The autopilot could not trust its numbers, so it handed the plane back to the pilots. Nothing else was wrong. Within a minute the readings came back.
What that moment needed was one move, and it is the opposite of what your body wants. The pilot flying did what his body wanted. He pulled the nose up. For most of the next four minutes and twenty-three seconds, he held it there. Near the end he said, “But I’ve been at maxi nose-up for a while.”5
Two hundred and twenty-eight people.
He had been taught the right move, the way every pilot is. What he had not had was practice making it by hand, at that height, where the autopilot almost always flies. The investigators found the training was not designed to make up for that.6
The recorders lay on the ocean floor for two years before anyone could read them.
This is not the first thing we handed over. Every tool that made us better at something made us worse at something else, and we took the trade every time, because every time it was a good trade. Some of those trades we survived. We do not have a list of the ones we did not, because the people who walked off that edge are not here to tell us what they lost. We learn from the wrecks we can still read.
And you will not feel it going. Neither did the men who fell out of the sky.
That’s the whole of this one. It belongs to a book I’m writing in the open, AI Has No Morality. It Has Yours. Subscribe and I’ll send the next piece when it’s ready.
If it gave you something, pass it to someone.
BØY (Chaiharan) has spent 30 years in tech, building products, recovering disasters, and turning around the things nobody else wanted to touch. Based in Bangkok. Writing a book in public about what AI reveals about the humans who use it.
Hao-Ping Lee et al., “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers,” CHI 2025, Microsoft Research and Carnegie Mellon University. https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/ ↩
Lisanne Bainbridge, “Ironies of Automation,” Automatica 19, no. 6 (1983): 775–779. https://doi.org/10.1016/0005-1098(83)90046-8 ↩
Budzyń et al., “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study,” The Lancet Gastroenterology & Hepatology, August 2025. https://www.thelancet.com/journals/langas/article/PIIS2468-1253(25)00133-5/fulltext ↩
Doug Bowman, “Goodbye, Google,” Stopdesign, 20 March 2009. https://stopdesign.com/journal/2009/03/20/goodbye-google.html ↩
BEA, Final Report on the accident on 1 June 2009 to the Airbus A330-203, flight AF 447 Rio de Janeiro–Paris, July 2012 (English edition). The autopilot disconnected at 2 h 10 min 05 and the recording ended at 2 h 14 min 28; the quoted line is from the cockpit voice recorder transcript in the report’s appendix. https://bea.aero/fileadmin/documents/docspa/2009/f-cp090601.en/pdf/f-cp090601.en.pdf ↩
BEA, presentation of the Final Report, 5 July 2012, which names “an absence of training, at high altitude, in manual aeroplane handling” among the associated factors. https://bea.aero/fileadmin/uploads/tx_elyextendttnews/presentation.rapport.final.05juillet2012.en_04.pdf ↩