Tell me you don’t understand software without literally using those words!!!
People who claim “LLMs can write decent code” don’t understand how code works. Sure, creation is much cheaper, but anyone who has run software in production at scale knows that maintenance, reliability, security, scalability, etc. is the majority of the cost. These are commonly known as NFR (non-functional requirements).
In my experience even the Functional Requirements (what the code is supposed to do) is NOT a solved problem yet. There’s a bit of Dunning-Kruger effect at place where the people who don’t read the output are more confident in it.
As a veteran developer holding 2 engineering degrees (hardware and systems engineering), I can list 3 types of products that do not strictly require reading the code:
Personal software: scratching an itch, automation, DYI patches, etc.
POC (proof of concept): demonstrating technical feasibility and product viability
Weaponized AI: acknowledge the risk and deliberately point it at a target to cause harm
Notice the commonality: high risk tolerance.
Most software that requires hiring and paying software engineers has low risk tolerance: healthcare, finance, automotive, aviation, manufacturing, … wherever a mistake can cost money, lives or legal consequences you need accountability. AI cannot be held accountable. It cannot suffer any consequences.
You cannot be responsible for what you can’t control either. That understanding is key to reasoning about system behavior and fix it when the AI inevitably fails.
If you’re toying around, LLMs do a great job. That’s why some of the most aggressive proponents of the “coding is solved” narrative have nothing to show for it. Anthropic accidentally leaked Claude Code (which on further study turned out to have many flaws) and their status page shows orange is the new green!
Why coding is NOT solved?
Contrary to common narrative, coding is actually one of the last areas for the current generation of LLMs to take over!!!
Allow me to elaborate:
Coding is about logic. Anyone who has dealt with compiler errors knows that computers don’t give a f*** about how right you think you are. If it’s logically wrong, it doesn’t compile. Even if the syntax is fine, there are runtime errors.
The reason LLMs are successful in writing code is because we’ve made a feedback loop that feeds the syntax/runtime errors back to the LLM and loops until most errors are solved or hidden.
LLMs can wing it for tasks that are related to natural language (e.g. writing social media posts, reports, articles, etc.) but when it comes to code, the same engine that struggles to count number of R’s in “Raspberry” or suggests a walk to car wash, struggles with simple but important logical issues.
LLMs are stochastic and probabilistic. The only way we could even get remotely close to make them logical is to wrap them in traditional code (known as harness), run tests, and a bunch of other techniques (e.g. CoT) but the core issue remains: LLMs struggle with logic and volume (the larger the input and the more the context window is used, the less accurate they get).
I’m not saying LLMs cannot generate code or maintain existing code bases. They have their utility as a tool and their capabilities are increasing in an S-curve. There is a point of diminishing return where more expensive models aren’t necessarily more productive at the rate of the price increase.
Those who claim LLM-generated software is good enough:
❌ Haven’t written code in ages
❌ Cannot spot if their code figuratively had 6 fingers!
❌ Have a low bar for what good looks like
❌ Don’t care about quality or NFR
❌ Have difficulty understanding an S-curve
✅ Are honest: AI genuinely writes better code than them
But to go ahead and extrapolate that to an entire professional industry requires a level of brain-dead thinking that’s only present in people who spend too much time with sycophantic AI.
Not your lab rat!
I’m not here to convince you. I couldn’t care less. What I do care is that the services I’m paying for (looking at you Google and GitHub) are degrading with stupid bugs.
If you’re in leadership position, please don’t stress your [smart] developers to force AI into every possible surface and workflow.
The tech has some genuine power and is the biggest change in our industry in ages. But “AI overdose” is a thing, and when it hurts the customer, you are accountable.
Stop repeating the half-baked narratives from token sellers about exaggerating the capabilities of AI because we, the consumers pay the end price.
AI is great for POC (proof of concept), Personal Software (a growing category), Map-reduce on human language (e.g. translation, converting different formats, summation, expansion) with varying degrees of success but the current generation of tech has fundamental problems too.
It’s not doom and gloom
I don’t want to belittle how far we have come with harness, SKILLS, AGENTS-md, MCP, A2A, ACP, RLM, OKF, MoE, MoA, self-healing, and various runtimes, quantizations, optimizations, architectures, and memory techniques.
I’ve written about many of those before:
Those are great pragmatic approaches to work around LLM shortcomings and there are probably more to come.
What I’m trying to elaborate is that I don’t want the services (that I depend on) to degrade just because someone pushed AI where it didn’t belong or skipped their job in quality, security, reliability and verification.
AI overdose and AI overuse is a thing.
I believe AI is a bar raiser: if the quality of your output is equal or subpar to AI, upskill.
Other fallacies
"You can create a full spec upfront". If you're that naive, I know a guy in a white van who gives free ice cream! Let me guess, you also believe software estimates are accurate and Santa is real. Anyone with a few years of industry experience knows that it's impossible to spec the software meaningfully ahead of time (unless it's very trivial).
"English replaces code". Hold my beer! Human language is vague and conflicting. That's the primary reason programming languages are created. A compiler or type-checker flags some of those conflicts. How on earth can you be sure that one part of your NL instructions don't conflict with another? The only way to discover those nuances is to ask your agent build what you asked for. But that's much more expensive than a linter or compiler.
"I move much faster". Don't confuse motion with progress. Don't measure progress in vanity metrics like SLOC, PR count or features. Measure service levels, ie. service consumer's happiness. Call me when you can prove a margin between token costs and business value.
"Agent is the new compiler". Ah that one again! Sure! If that's your reality, I let this meme do the work.
Bonus point: an old trick
Pssst! Do you want to know an old trick to make your LLM-generated code instantly superior?
Run multiple-agents in parallel! The sheer volume of code makes it humanly impossible/expensive to review and you give up!
The trick is the same as pre-AI era: if you want a PR to be merged, make it massive because ain't nobody got time for that.
It'll be merged based on "trust"!
We don't exactly trust AI but we have to because the alternative (having to read the output) is too hard for some folks! Instead they come to social media and claim that since UAT (user-acceptance testing) passes, the code is "good enough". Then ship it to me and you to do the rest of the testing.
We're just lab rats after all. Just a friendly advice: have a little AI-free hobby project to keep your coding skills fresh for when you're thrown back to the job market. Cheers!