LLM reality check

Last week, we had beers in Warsaw with tech guys of my age. One of them watched a CEO's face fall as his tech team explained why their much-hyped LLM implementation wasn't delivering the promised results. "But everyone's doing it," he said, confused. "What are we missing?"

This scene plays out in boardrooms these days. Companies rush to deploy large language models with the same vigor that drove the dot-com boom or the initial charm of generating human-like text masks deeper questions about real business value.

Boring, applications that solve specific business problems

I've spent the past year studying companies' LLM implementations, and I've noticed something fascinating: the most successful ones rarely make headlines. They're not the flashy chatbots or the automated email systems that companies love to showcase. Instead, they're careful, sometimes boring, applications that solve specific business problems.

Take the case of a mid-sized company I worked with. While their competitors raced to build customer-facing AI assistants, they took a different approach. They used LLMs to help their frontline staff write more accurate reports. Not exciting enough for a press release, perhaps, but it cut processing time by >30%. The key? They thought deeply about the human-AI partnership rather than chasing full automation.

This brings me to what I believe is the fundamental mistake in Rama Ramakrishnan's otherwise excellent analysis. While his "generative AI cost equation" provides a solid framework for evaluating LLM projects, it misses the hidden costs that often sink these initiatives.

I have first-hand example. It what happened at a software company that automated their first-level technical support with LLMs. On paper, it looked great - faster response times, lower costs. But six months in, they noticed something troubling. Their senior engineers were spending more time fixing complex problems because they weren't seeing the early warning signs that used to bubble up through routine support interactions. The "cost savings" were actually pushing problems downstream.

These hidden costs take three main forms:

The opportunity cost of taking the easy path. When you automate existing processes, you might miss chances to reimagine them entirely. The same as putting a faster engine in a horse carriage instead of inventing the car.

The human cost of fractured work. When LLMs handle the routine parts of knowledge work, the remaining human tasks often become more fragmented and demanding.

The institutional knowledge tax. As more work flows through LLMs, organizations risk losing the pattern recognition and tacit knowledge that come from humans doing the full range of tasks.

Treat these tools as intelligent assistants rather than replacements for human knowledge workers

There's a better way forward. The most effective LLM implementations I've seen share a common philosophy - they treat these tools as intelligent assistants rather than replacements for human knowledge workers.

Analogy would be - learning to cook with a sous chef. A good sous chef doesn't take over the kitchen - he amplifies the head chef's capabilities by handling prep work, suggesting ingredients, and maintaining organization. The chef remains firmly in control of the final product.

This "augmentation first" approach addresses many of the hidden costs while still capturing real productivity gains. More importantly, it creates sustainable competitive advantages by combining human judgment with machine capabilities in novel ways.

The future of LLMs in business isn't about replacing humans - it's about reimagining how humans and machines can work together. That's a harder challenge than simple automation, but it's where the real value lies.

The companies that win won't be those that move fastest, but those that move most thoughtfully

As for that confused CEO? His company changed the strategy with LLMs. After they stopped trying to keep up with competitors and started focusing on augmenting their employees' capabilities in specific, measurable ways.

The lesson is clear: in the rush to embrace LLMs, don't let the hype blind you to the real opportunities - or the real costs. The companies that win won't be those that move fastest, but those that move most thoughtfully.