LLM reality check

February 2025 7 min read

Last week I had beers in Warsaw with a few guys my age, all in tech. One of them told a story I keep thinking about. His CEO had just sat through a meeting where the tech team explained why their big LLM project wasn't working. The guy looked crushed. "But everyone's doing it," he said. "What are we missing?"

A lot of companies are in that exact spot. They jumped on LLMs because everyone else did, and now they're realizing the thing writes nice sentences but nobody can point to the money it's actually making.

I've been looking at how companies really use LLMs for about a year now. The ones that work are boring. No cool chatbot, no demo you'd show off. Just some small, specific thing that quietly saves time.

One company I worked with is a good example. Everyone around them was racing to build customer-facing AI assistants. They didn't. They just used LLMs to help their frontline people write better reports. Sounds primitive but it cut their processing time by more than 30%. What made it work was simple - they figured out which parts the human should keep and which parts the machine should take, instead of trying to hand the whole thing over.

That's where I think Rama Ramakrishnan's analysis misses something. His cost equation is a good way to judge an LLM project. But it leaves out the costs you don't see coming, and those are usually the ones that sink the whole thing.

I watched it happen. A software company handed their first-level tech support to LLMs. Faster replies, lower costs, looked great. Then about six months in, their senior engineers were drowning in hard problems. Turned out the routine support tickets used to give them early warnings, little hints that something was going wrong. Once the LLM ate those tickets, the warnings disappeared. The problems didn't go away. They just moved down the line and got more expensive.

So what actually works? The good setups I've seen all treat the LLM as a helper, not a replacement for the person thinking.

It's like cooking with a decent sous chef. He doesn't run the kitchen. He does the prep, hands you things, keeps it all organized, and makes you faster. You still decide what goes on the plate.

Use it that way and you dodge most of the hidden costs while still getting real speed. And it's harder for competitors to copy, because you're combining your people's judgment with the machine in a way they can't just buy.

The hard part isn't swapping people out for AI. It's rethinking who does what. That's more work than plain automation.

So the point is simple. Don't let the hype hide the real wins, or the real costs. The companies that come out ahead won't be the fastest ones. They'll be the ones paying attention, able to think, customize for own unique position.

P.S. As the amount of real-value LLM implementations grows, we experience the Jevons paradox in action.

---