It Used to Cost $100 to Fix an AI. Now It Costs $150 Million.

There is one comparison Stephen Kemmerling keeps coming back to, and it explains more about the state of artificial intelligence than most of what gets written about it.

For most of his career, if a machine learning system was doing the wrong thing, the fix was trivial. You threw it away and built another one.

“I’ve worked on production systems where we’re doing that a dozen times a day,” he says. “Just throw the AI away, build a new one from scratch. Twelve times a day. Automate it.”

The cost of doing that, on his systems, was about a hundred dollars. “Just making the actual AI, $100 on a server somewhere for like an hour, and you’re done.”

Then the comparison.

“The latest version of ChatGPT reportedly cost them $150,000,000 to train. So if they had to do that again, it would be another $150,000,000. You just can’t do that anymore.”

Retraining has gone, in his words, “from a really cheap step in your workflow, something that’s easy and quick and not really something you have to think about a whole lot, to by far the most expensive and most time consuming step”.

Everything else he argues follows from that inversion.

A career spent in small rooms

Kemmerling grew up in Germany and studied computer science and mathematics before a master’s in applied mathematics at Stanford. He started a PhD there, and a startup recruited him out of it.

“I’m technically still on leave from the Stanford PhD program. It’s been like 15 years now, so I don’t think I’m going back.”

What followed is a pattern. Lexity, an ecommerce intelligence startup, where he was one of the first fifteen or so employees. Then Kifi, where he was employee seven or eight. Both got acquired, and he had left before either deal closed. “Lexity got bought by Yahoo. I didn’t want to work for Yahoo. Kifi later got bought by Google. Also didn’t go to Google.”

When he finally did join a large company, his reason is worth stealing.

“I ended up joining what was called Facebook at the time, now Meta, because it had the smallest office.”

He stayed a few years and left for a reason he states plainly: “There was a lot of ad targeting and how to squeeze more money out of people. What I was working towards just didn’t feel quite right to me.” Then Indigo, a Boston climate company using agriculture for carbon capture, where he built the AI team from seven people to around 120.

The problem he named before it had a name

What he had been noticing across all of that was something he had to invent language for.

“In my writings, before the terminology started to coalesce, I liked to call that control of AI systems. And what that boils down to is simply, how do you reasonably quickly and reasonably cheaply get your AI system to do what you want it to do?”

The old answer, retraining, has priced itself out. The consequences arrive in two stages. The first is concentration: “fewer and fewer entities actually get to work on advancing the state of the art, because of the cost.” The second is subtler and worse. Those remaining few also get slower.

“Their iteration cycles, every time they want to try something, take longer and longer and more and more money. So you’re kind of stalling out, almost, as you’re making progress.”

Competition as a safety mechanism

Kemmerling thinks market competition is a safety feature, and that losing it is dangerous in a way people are not paying attention to.

“The fewer people work on it, the less competitive economic pressure there is to do better.”

His illustration is deliberately mundane. Imagine a company with the best chatbot in the world and a flaw in it serious enough to cause real problems. Fixing it means retraining, which now costs a fortune.

“If fixing it costs you $200 million, instead you pay $5 million to the lawyers to make the problem go away, then you can live with that.”

Now add a competitor. “If you have the economic pressure of someone else also having a comparable product, then that looks a little different, because all of a sudden you’re losing users.”

The mechanism that actually forces quality is not conscience and it is not regulation. It is somewhere else for customers to go. Concentration removes that.

He is careful to place himself outside the apocalypse genre. “I’m not in the AI doom camp of, hey, it’s going to end the world, and the paperclip maximiser turns the world into machines and the Matrix or whatnot.”

His concern is more concrete, and he asks you to stop picturing chatbots. Think instead about a household robot that does your chores and watches your children, or a self-driving taxi. A text generator producing something offensive is one category of problem. A household robot hitting your kids, a taxi driving into a crowd, or an automated medical system starting to harm patients is another.

“We don’t have the human in the proverbial driver’s seat. Right now it’s very, very hard to intervene. And the more advanced systems get, the harder it gets.”

That is why he started a company rather than joining someone else’s. This problem needs solving, nobody seemed to be solving it, so he would.

The bubble he expects to burst

Kemmerling is unimpressed by how narrow the public conversation has become.

“There’s this belief that language models are kind of the be all and end all for AI, and it is very much not,” he says. “You’re not going to want your language model to drive your car, or operate your surgical robot, or do drug discovery.”

His estimate of the coverage is blunt. “It’s like 90% language models and 10% something else, and that’s completely false.”

The example he wishes people had noticed arrived at almost the same moment as ChatGPT: AlphaFold, the protein folding system from Google DeepMind.

“To me, that’s much more impressive, because it does something that no human can do. Whereas there are plenty of humans who can do what ChatGPT can do, and often significantly better.”

Which leads to a prediction he is willing to date.

“I don’t think the AI bubble will pop anytime soon. But the bubble around language models, I think, will pop,” probably within a year or so. Not because they are useless. He uses them constantly. Because they will not deliver the thing they are being valued for.

“They’re not this magical thing that gets us to artificial general intelligence. Just like human intelligence is not just about language. You don’t just think in language, you think in abstractions, in images and in smells and sounds.”

What it is all for

Ask what gets him out of bed and he goes somewhere unusually large.

“The only two things that have broadly reshaped society in a categorical way across the entirety of human history are war and technological advances.” He pauses on the obvious point. “Not intending to go to war. Don’t like that. So I work on technological advances.”

His framing is defensive rather than triumphal. “I have a very risk-centred mindset. Not in a depressing or negative way, but mitigating negative outcomes. What are all the paths we could take, and some of those paths are not very good. How do we make it less likely that we’ll go down that path?” He has two young children, which is where the abstraction lands.

Asked what he values in people, he names honesty first, then something he distinguishes carefully from cleverness. “By intelligence I don’t mean raw IQ, but more a willingness and desire to think deeply about things.” He likes being argued with, and will argue positions he does not hold for the exercise.

What he cannot stand is a particular internet personality type. “It’s okay to be critical and have a better approach, but this sort of, my brand is dumping on other people. I don’t want that at all. It really rubs me the wrong way.”