AI Reliability Engineering

Why SRE practices are hot in the age of AI-generated black boxes and how to renovate the traditional toolbox for the new era

I have written a lot about different aspects of AI Engineering and Reliability Engineering. This article is my effort to fuse the two while digging a layer deeper and elaborating how SRE practices unlock AI’s potential.

Key takeaways:

• Why: As the LLMs and harnesses become more capable while the industry learns to better utilize them, I see the concern of software engineers shift towards NFRs: running AI predictably, securely, and at scale.

• What: The SRE toolbox traditionally evolved to reliably run code that’s written by others. AI-generated black boxes benefit from the same toolbox.

• How: As a veteran SRE and senior technical leader, I’ll highlight some practices that I find relevant to senior engineers and engineering leaders in the intersection of AI and SRE.

Disclaimer: no generative AI is used to research, write or edit this article. This is purely organic content and took a lot of time to formulate, rethink, polish, and publish. I hope you enjoy it, and share it with your friends and colleagues to lift our industry.

What is SRE anyway?

There are books written on the topic but it boils down to one simple question:

How much of traditional software engineering practices can be used in operations?

Ben Treynor Sloss who coined the term back in 2003, mentioned that people come to SRE from two walks of life:

• Sysadmins: people with experience of handling servers, infrastructure, networks, users, policies, and databases.

• Developers: people who have a background in designing, building and evolving software products.

I don’t dwell too much on the definition because:

1. What works at Google doesn’t necessarily fit other companies with fundamentally different budget, scale, and talent bar.

2. I have a problem with that “S” in “SRE”. Initially it stands for “Site” but I practice it as System Reliability Engineering and often drop the “S” for simply Reliability Engineering.

If you want to dig deeper, I’ve written more about SRE archetypes 2.5 years ago:

SRE Archetypes

My SRE to AI journey

Long term subscribers of this newsletter know that I’ve been working as SRE (Site Reliability Engineer) at different capacities (Sr Eng, Staff, Sr Staff) across different industries and even have a book on the topic.

When AI started to take off back in 2022, I saw both the threat and opportunity but didn’t have the experience to tackle either. Initially, I reached out to a few Data/ML veterans in my circle for tips but I got outdated advice. Meanwhile, my employer was wasting time on a grand “AI strategy”.

Instead of wasting time, I started experimenting first-hand to embed it in my workflows and building LLM-powered products.

This was such a critical time in my career (and our industry) that I couldn’t afford to wait. I had to take my growth in my own hand. By the time, the company had an “AI strategy” paired with an elaborate gate-keeping process in place, I was 9 months into paying from my own pocket to buy API keys and a couple of dedicated AI machines in order to build a harness and a few AI-driven products.

AI Fluency Leveling

The thing that wasn’t obvious at the time (but I can see clearly in retrospect) is how much of good old SRE toolbox applies to AI systems and it makes me giggle when the industry “discovers” them.

For example:

• Ralph loop (repeating the same prompt to reach a goal) is a simplified version of retry pattern.

• AI harness (topic for an upcoming article) where the stochastic AI is paired with deterministic code is an example of guardrail pattern.

• More recently loop engineering, is just the application of a finite feedback loop with an evaluation condition.

• Various memory techniques from the humble linear RAG (using vector database and cosine similarity) to graph RAG (e.g. LLM-wiki or memory palace) are implementation of separation of concern while respecting the constraints of context window.

• In fact what’s called AI guardrail in products like OpenRouter, is a mix of various constraint strategies in resilience architecture.

There’s way more and I’ve listed 30 of these patterns a few months ago:

AI Systems Engineering Patterns

You don’t have to be an SRE to understand these patterns. But having that experience helps you seeing the old ideas in new packages.

The point being: the current generation of AI is not very reliable while SRE is all about creating systems that are more reliable than their components.

Not only that, but I’m also convinced that as AI takes over parts of the traditional software engineering tasks, we gradually spend more time in areas that traditionally fell into SRE, QA (quality assurance) and Security category.

Introduction to Staff Engineering

Personally, as a Staff Engineer (which is a leadership role), I see two distinct element to reliability engineering:

• Technology that solves a problem

• People who design, build, maintain, evolve and own those systems

I describe it as sociotechnical reliability engineering and try to address both the people and technical aspects in my book Reliability Engineering Mindset.

Reliability in the context of AI

If you’re an SRE reading this, I got some good news and bad news.

• Good news: your toolbox is hotter than ever because AI’s reliability is its most important problem to solve.

• Bad news: if you’ve been dismissing AI as fad, hype or bubble and haven’t skilled up beyond prompt engineering, you’re going to have a hard time staying relevant.

Let me add ammo to that bad news: I’ve been coaching hundreds of teams across Europe in my Service Level workshops where we identify meaningful metrics (SLI), set reasonable expectations (SLO) and connect commitment to accountability (alerts based on error budget). Do you know who are the hardest group of people to deal with?

SREs! Especially the senior ones who know enough to be opinionated but not open enough to accept a different take on these concepts.

Trapped by the baggage.

In knowledge work, the ability to unlearn the old is as important as the ability to learn the new.

When I say reliability in the context of AI systems, I specifically mean:

• Alignment: reducing the gap between the outcome and expectations of systems that have one or more AI components. It’s about answering the questions:

  • Can we trust AI with a particular task?

  • If not, how can we prepare for and reduce the risks?

• Cost efficiency: maximizing the profit per token, making sure that the cost of AI is justified by the impact. It’s about answering the questions:

  • How do we measure impact of AI?

  • What’s the cheapest way to achieve that impact?

• Ownership: promote understanding and accountability especially in a world where AI is increasingly creating larger black boxes.

  • Implementing HITL while reducing approval fatigue (younger sibling of alert fatigue, for those who are familiar)

  • Ensuring human understanding because without that understanding, TTR (time to resolve) dramatically increases. AI cannot reliably detect and resolve all incidents due to its known and unknown logical limitations.

• Taking calculated risks: balancing predictability and velocity.

  • YOLO (You Only Live Once) is used to justify taking risks, stepping outside comfort zone and living dangerously 😅 but we shouldn’t take risk at the expense of business or service consumers.

  • Proper risk assessment helps us prepare for the unexpected. You may not need a break-glass pattern when the AI breaks free, but at least you need to have a fallback so the users who have paid get service. (I’m looking at you Anthropic, who doesn’t have an SLA!)

• Automation: eliminating toil in favor of speed, predictability and accuracy while mitigating risks like cascading failures.

  • AI can be used as another tool in the automation toolbox. Some SREs even do incident auto-detection and auto-recovery but the pre-work to prevent false alerts is key to using AI for this kind of automation.

  • More likely, we use AI to generate and update scripts, and without proper guardrails, the risk of catastrophic failure propagation is high.

  • Platformization: the practice of taking common solutions and turning them to repeatable patterns that can magnify their benefits at scale (e.g. cheaper to run, more reliable, and with higher quality) while using IaC (infrastructure as code) patterns for repeatable and reproducible topologies.

• Availability: of all the SRE to AI mapping scenarios, this is the closest to traditional SRE as is and it has a few aspects:

  • Architecture: using various resilient architectural patterns (e.g. fallback, failover, circuit breaker, rate limits, etc.) to ensure the stochastic AI component doesn’t kill the entire product.

  • Observability: The ability to understand and reason about a system’s internal state from its external observable behavior. There are at least 4 ways to define and measure availability and the closer to the actual use-case we measure, the more meaningful our measurement can be, and hence the better we can be prepared.

  • Graceful degradation instead of service disruption: the ability of a service to function partially (when it makes sense), instead of being unavailable. We’ve covered more background in degradation vs disruption but it boils down to isolate the negative impact through isolation, compartmentalization, etc.

• more…

You may be asking:

Aren’t all of these everybody’s concern?

Yes, they are! SREs just happen to have a toolbox that reduces trial and error and shortens the learning curve. AI (not just LLM) isn’t fundamentally different than the unpredictable components and systems we’ve been traditionally working with.

It’s important to mention that not everyone with “SRE” in their job title practices all of the above. In fact, I’ve observed many traditional operation folks relabel themselves to “DevOps” when that was a thing, and later “SRE”. These days they go by “AI platform engineer” or the like but what distinguishes real SREs (per my definition) from impostors is:

• Smashing silos: Breaking silos between developers and operations to implement full ownership (knowledge, mandate, responsibility).

• Force multiplication: A good SRE (especially at Staff+ level) should lead by example and educate others to lift the entire organization instead of firefighting and addressing symptoms instead of fixing the root cause. Reliability is sociotechnical.

• Use process wisely: acknowledging that every process causes friction so add it systematically where the cost of slow-down surpasses the profit of lubrication. Automated guardrails and light or no gate-keeping. Trust by default but also do the pre-work that makes mistakes cheap and work with leadership to eliminate finger pointing and other toxic behavior that prevents learning from mistakes.

How AI changed SRE?

I want to close by acknowledging how AI has changed software engineering, especially the practice of measuring and improving reliability:

• AI made parts of SDLC (software development life-cycle) faster but we don’t have to go as fast. Sometimes slow (taking the time to understand what we’re shipping) is fast (because it saves us from incidents and enables informed actions when things break).

• AI bloated our repos with more code to test and verify which may require new methods and tools. Too many engineers are lost to the “coding is solved” narrative. Let’s not forget that the token sellers have fundamentally different incentives than a business that operates on a positive margin between the cost of AI and the part that’s productive.

• Even though AI can generate code, doing it “by hand” is still valuable, the same way solving puzzles is good for the brain. You should at least write the critical parts of the code yourself, with SKILL.md files you have written and in the context of AGENTS.md files that you developed.

• Too many engineers have been stressed, exhausted, and struggle to catch up with a “colleague” that never sleeps and pretends to know too much. There’s good stress and bad stress. Bad stress hurts our judgement and critical thinking. It’s not sustainable. In the words of Mario Zechner: we need to slow the f*ck down!

• AI made us trust an ever increasing amount of code that is honestly not verified. Yes, there are malicious MCPs and Skill packages but that’s just the tip of the iceburg. We are also pumping our valuable organizational data to the AI vendors while innocently trying to get value from their products. What makes you think they’ll honor a contract with you as an individual or even your business? Just a couple of days ago Apple sued OpenAI for stealing its trade secrets. There are several lawsuits from parents who lost their kids to bad advice from AI. On top of that, there are artists, authors, and copyright holders whose work was stolen to train AI. Do you really think AI vendors are going to care about a contract where the juicy stuff (your data, in the context of actual work) can give them a winning advantage? Every business that has critical data, should run the inference engine on-prem. The data is just too valuable to pump to these AI vendors because when sh*t hits the fan, the standard SLA cannot cover all the damage. Don’t get too greedy with cloud AI. There was a reason it was subsidized. Again SREs (especially the platform engineering aspect) is super relevant, if you’re considering on-prem.

The last bit may come across as anti-AI, from someone who has bet his entire career on AI, but you can use AI privately, securely and reliably.

AI topology

Edge AI has come a long way, and even if you don’t want to go that route, the very principles that is hard-coded into the brain of every respectable SRE is still valuable to run AI reliably at scale:

• Systems thinking, critical thinking, and first principles

• Understanding NFRs (non-functional requirements) and evaluating their priority for a given task

• Understanding trade-offs

• Observability that enables reasoning about system behavior and systematically debugging it

• Preventing complexity or containing it when it’s inevitable

• Designing for failure and have resilience patterns built-in (when the cost is justified by the risk)

• Preparing for degradation instead of disruption (when it makes sense)

I feel like I have repeated myself enough. I will just leave you with one thought: AI is a stochastic component. You can (and should) build a deterministic system around it to tame its value. Harness is the first step. What else can you do?

---

My monetization strategy is to give away most content for free but these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access or save via a group subscription.

And to those of you who already support me, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.