LLM reality check
February 2025 6 min read
Last week I had beers in Warsaw with a few tech guys. One story hit home for everyone: the push to adopt AI (LLMs) at any cost, so you don’t miss the wave. If competitors do it, we must be faster!
A lot of companies are in that exact spot. They got into LLMs because everyone else did. Now they see the thing does something, while nobody can point to the money it actually makes.
I’ve been looking at how companies use LLMs for about a year now. The boring applications win. You don’t see flashy stories about AI transformation, just very specific tasks automated.
One company I worked with skipped the customer-facing AI assistant everyone else was building. Instead, they used LLMs to help frontline staff write reports and cut processing time by more than 30%. It worked because they split the job: some parts stay with the person, some go to the machine.
Rama Ramakrishnan’s cost equation is a good way to judge an LLM project. But it misses the hidden costs, and those are usually what kill it.
A software company handed first-level support to an LLM. Replies got faster and costs dropped. Six months later, senior engineers were drowning in hard problems. The routine tickets had been their early warning system. When the LLM took those tickets, the warnings disappeared, and the problems came back bigger and more expensive.
What the spreadsheet shows
What works - is the LLM as a helper, with the person doing the thinking.
Think of a good sous chef. He preps, hands you things and keeps the station clean. You (chef) decide what goes on the plate.
- Decides the dish
- Owns the final plate
- Reads the room
- Judgment calls
- Preps the ingredients
- Suggests options
- Keeps it organized
- Handles the repetitive bits
This way you avoid most hidden costs, and competitors can’t easily copy it. They can buy the same model, but not your people’s judgment.
The hard part is deciding who does what: the person or the model. That’s more work than plain automation.
Don’t let the hype hide the real wins or the real costs. The companies that come out ahead will be the ones that fit LLMs to their own situation, and they won’t always be the fastest.
P.S. As LLMs deliver more real value, we see the Jevons paradox in action: cheaper LLM work means more use, not less.