"Sometimes in the process of writing a good enough prompt for ChatGPT, I end up solving my own problem, without even needing to submit it". AI as a rubber ducky, I think this is good AI use!
I read this article through LLM translation, and I’m also using LLM translation to reply right now. This happens to align with the article’s point (as I see it): you must take responsibility for LLM-generated content. There was an interesting discussion in the Chinese tech community previously; it mentioned that the software industry would later usher in its own Cambrian explosion, but it did not mention another biological event, namely the Ordovician mass extinction. In my view, if software engineers do not take responsibility for the code they run and do not continuously maintain their ability to write code, they are very likely to be eliminated in the visibly approaching “age of mass extinction.” The same applies to software companies: the proliferation of all kinds of software and LLMs will cause the entire software industry to degrade and become oversaturated, and what ultimately remains will most likely be the various kinds of software that still adhere to providing high-quality services and maintaining high availability.
This post is extremely well written, and it summarizes most (or all) of my problems working with software in this AI era. Now I can articulate much better why I'm so sick of this. Thank you!
I could not put my opinions on AI better than this.
As a casualty of Saaspocalypse, I've landed on running local LLMs for everything from code completion/assistance to the 3 AM "Who would win in a fight between a polar bear and a tiger?" Every now and then I try to generate software without actually handling code myself, and no matter what model or harness I use, no matter how detailed the markdown documents, no matter how thoroughly I appease the machine spirits, I always get about 85% of the way before it falls apart.
No amount of fiddling with levers, managing context, etc., produces anything close to what I'd call production-ready code. At best, you get a pleasantly boring UI suitable for a sales demo on top of a rattling heap of spaghetti: The Sorceror's Apprentice meets "best coding practices". If you're lucky, it mostly works, if you don't use it too much. As you say, if you care more about velocity than quality, it's fine, and that encompasses a good number of use cases. It's just not a replacement for engineers and never will be.
Although, I will say it has inspired my decision to move from engineering to a product/program manager role, so there's that.
I had a very similar thought on where many developers are going to take their careers. I also struggle with trusting these AI companies given their track record and you make a good point not to trust them in what they do and say.
I have invested most of the last year in a range range of "vibe coding" projects - so I largely agree with Alex's point of view on the inadequacy of current AI coding tools and methodologies for Real Work. Using observations about the failures in those millions of lines of code and thousands of human hours, I've deduced a set of more foundational causes for the waywardness of LLMs on the "big-code-from-prompts" challenge:
1. Ambiguous specifications — omissions and contradictions in requirements get translated directly into conflicting implementations.
2. Stochastic interpretation — LLMs make many implicit architectural and coding choices, so nominally equivalent prompts can produce divergent assumptions.
3. Ambiguity compounds across passes — repeated planning, decomposition, and implementation steps amplify small inconsistencies into system-level disconnects.
4. No single source of truth — different modules independently reinterpret the same state or requirements and gradually disagree.
5. The code becomes the specification — evolving prompts are not preserved as a coherent spec, so buggy or ambiguous code becomes the reference for future changes.
6. Limited global attention — large systems exceed what an LLM can analyze deeply at once, forcing multiple “peephole” passes that lose global consistency.
7. Local fixes replace architectural repair — debugging tends to patch observed symptoms rather than identify and eliminate the underlying general inconsistency.
8. Code bloat makes the problem self-reinforcing — additive fixes increase size, interactions, analysis cost, and the long tail of subtle bugs.
Qodo found 89% of organizations have had an AI-related incident in production, and only 3.7% of engineering leaders call their processes adequate. New Relic's survey found a similar gap: 94% of US tech leaders rate AI code highly at review, but 78% see more incidents once it ships. The code looks fine until someone is accountable for what it does at runtime.
The firms making this work narrow the blast radius rather than trusting taste. Duolingo lets a risk-scoring bot auto-approve only its lowest-risk 10% of changes, with infrastructure and audited repositories excluded by design. That is the opposite of "coding is solved": it is accountability engineered back in at the boundary, one category of change at a time.
> I can list 4 types of products that do not strictly require reading the code:
You forgot another category - social media/entertainment software that is free to access (or rather pay-with-your-data model). Even before AI they did not gave a f*** if the UI and features work as long as user were kept hooked by main manipulative means.
Maintenance, reliability, and security being the actual majority of software cost is the part every "AI writes code now" take conveniently skips. Creation got cheap, running the thing in production for three years without it falling over is still the same hard problem it always was.
I doubt that AI is here to stay - in the shape it is right now with LLMs (things like computer vision, classification, etc were around long before LLMs and those - yes, are not going anywhere).
In my experience, the code produced by claude with latest opus or copilot with astra has landmines in it 30% of the time and requires correction. This is mostly java/c#/kotlin backend code, and 2 out of 3 are popular backend languages with a lot of high-quality training data available. No one in their right mind will pay for that level of output quality even at the current subsidized prices.
Just to cover their current CapEx the hyperscalers have to make way more money. Here's a back-of-the-envelope calculation for their economical situation. This year their CapEx is said to be $800B while AI-related revenue is $70B. The CapEx structure is said to be 40% on data centres (~15 y lifespan) and 60% on GPUs (~6 y lifespan) that gives us annualized CapEx of ~100B (only this year!). And then there's OpEx eating out that $70B revenue. Goldman says they need $300B to break even.
Then there are AI-labs themselves burning money at enormous pace (it's hard to tell as they are not public and the data they let out is highly manipulative, but it's safe to assume that they are burning tens of billions $ every year, and the pace is increasing). And there are debts accumulated from previous years.
This is just not viable economically. If a breakthrough happens - yes, the whole picture may change, but given that most of the brightest engineering minds have been trying to achieve it for 4 years already - that has a probability of a miracle.
So, my prediction no one here asked for: the bubble pops and starts a crisis worse than the .com crash, along with the third AI winter. Winter is coming!
AI can write code, but someone still has to understand and run it. The person who approves a change needs enough detail to own what happens if it breaks.
"Sometimes in the process of writing a good enough prompt for ChatGPT, I end up solving my own problem, without even needing to submit it". AI as a rubber ducky, I think this is good AI use!
> But they give you one thing that cloud AI can never guarantee: your data stays local.
Until you give it permission to run `curl` 😅
I read this article through LLM translation, and I’m also using LLM translation to reply right now. This happens to align with the article’s point (as I see it): you must take responsibility for LLM-generated content. There was an interesting discussion in the Chinese tech community previously; it mentioned that the software industry would later usher in its own Cambrian explosion, but it did not mention another biological event, namely the Ordovician mass extinction. In my view, if software engineers do not take responsibility for the code they run and do not continuously maintain their ability to write code, they are very likely to be eliminated in the visibly approaching “age of mass extinction.” The same applies to software companies: the proliferation of all kinds of software and LLMs will cause the entire software industry to degrade and become oversaturated, and what ultimately remains will most likely be the various kinds of software that still adhere to providing high-quality services and maintaining high availability.
I just want to say: thank you!
This post is extremely well written, and it summarizes most (or all) of my problems working with software in this AI era. Now I can articulate much better why I'm so sick of this. Thank you!
I could not put my opinions on AI better than this.
As a casualty of Saaspocalypse, I've landed on running local LLMs for everything from code completion/assistance to the 3 AM "Who would win in a fight between a polar bear and a tiger?" Every now and then I try to generate software without actually handling code myself, and no matter what model or harness I use, no matter how detailed the markdown documents, no matter how thoroughly I appease the machine spirits, I always get about 85% of the way before it falls apart.
No amount of fiddling with levers, managing context, etc., produces anything close to what I'd call production-ready code. At best, you get a pleasantly boring UI suitable for a sales demo on top of a rattling heap of spaghetti: The Sorceror's Apprentice meets "best coding practices". If you're lucky, it mostly works, if you don't use it too much. As you say, if you care more about velocity than quality, it's fine, and that encompasses a good number of use cases. It's just not a replacement for engineers and never will be.
Although, I will say it has inspired my decision to move from engineering to a product/program manager role, so there's that.
I had a very similar thought on where many developers are going to take their careers. I also struggle with trusting these AI companies given their track record and you make a good point not to trust them in what they do and say.
I have invested most of the last year in a range range of "vibe coding" projects - so I largely agree with Alex's point of view on the inadequacy of current AI coding tools and methodologies for Real Work. Using observations about the failures in those millions of lines of code and thousands of human hours, I've deduced a set of more foundational causes for the waywardness of LLMs on the "big-code-from-prompts" challenge:
1. Ambiguous specifications — omissions and contradictions in requirements get translated directly into conflicting implementations.
2. Stochastic interpretation — LLMs make many implicit architectural and coding choices, so nominally equivalent prompts can produce divergent assumptions.
3. Ambiguity compounds across passes — repeated planning, decomposition, and implementation steps amplify small inconsistencies into system-level disconnects.
4. No single source of truth — different modules independently reinterpret the same state or requirements and gradually disagree.
5. The code becomes the specification — evolving prompts are not preserved as a coherent spec, so buggy or ambiguous code becomes the reference for future changes.
6. Limited global attention — large systems exceed what an LLM can analyze deeply at once, forcing multiple “peephole” passes that lose global consistency.
7. Local fixes replace architectural repair — debugging tends to patch observed symptoms rather than identify and eliminate the underlying general inconsistency.
8. Code bloat makes the problem self-reinforcing — additive fixes increase size, interactions, analysis cost, and the long tail of subtle bugs.
A more complete and coherent version of this analysis of root causes of vibe coding's failure is found here: https://rowenchris.substack.com/p/why-vibe-coding-doesnt-work
You captured a lot of my thoughts and concerns as a non-developer. Thanks for putting this together.
Qodo found 89% of organizations have had an AI-related incident in production, and only 3.7% of engineering leaders call their processes adequate. New Relic's survey found a similar gap: 94% of US tech leaders rate AI code highly at review, but 78% see more incidents once it ships. The code looks fine until someone is accountable for what it does at runtime.
The firms making this work narrow the blast radius rather than trusting taste. Duolingo lets a risk-scoring bot auto-approve only its lowest-risk 10% of changes, with infrastructure and audited repositories excluded by design. That is the opposite of "coding is solved": it is accountability engineered back in at the boundary, one category of change at a time.
> I can list 4 types of products that do not strictly require reading the code:
You forgot another category - social media/entertainment software that is free to access (or rather pay-with-your-data model). Even before AI they did not gave a f*** if the UI and features work as long as user were kept hooked by main manipulative means.
Maintenance, reliability, and security being the actual majority of software cost is the part every "AI writes code now" take conveniently skips. Creation got cheap, running the thing in production for three years without it falling over is still the same hard problem it always was.
I doubt that AI is here to stay - in the shape it is right now with LLMs (things like computer vision, classification, etc were around long before LLMs and those - yes, are not going anywhere).
In my experience, the code produced by claude with latest opus or copilot with astra has landmines in it 30% of the time and requires correction. This is mostly java/c#/kotlin backend code, and 2 out of 3 are popular backend languages with a lot of high-quality training data available. No one in their right mind will pay for that level of output quality even at the current subsidized prices.
Just to cover their current CapEx the hyperscalers have to make way more money. Here's a back-of-the-envelope calculation for their economical situation. This year their CapEx is said to be $800B while AI-related revenue is $70B. The CapEx structure is said to be 40% on data centres (~15 y lifespan) and 60% on GPUs (~6 y lifespan) that gives us annualized CapEx of ~100B (only this year!). And then there's OpEx eating out that $70B revenue. Goldman says they need $300B to break even.
Then there are AI-labs themselves burning money at enormous pace (it's hard to tell as they are not public and the data they let out is highly manipulative, but it's safe to assume that they are burning tens of billions $ every year, and the pace is increasing). And there are debts accumulated from previous years.
This is just not viable economically. If a breakthrough happens - yes, the whole picture may change, but given that most of the brightest engineering minds have been trying to achieve it for 4 years already - that has a probability of a miracle.
So, my prediction no one here asked for: the bubble pops and starts a crisis worse than the .com crash, along with the third AI winter. Winter is coming!
AI can write code, but someone still has to understand and run it. The person who approves a change needs enough detail to own what happens if it breaks.