The details is behind an NDA (non-disclosure agreement) but there was an interesting aspect that is worth discussing:

All those systems were air-gapped, meaning the team that builds the software has no access to metrics, logs or runtime.

How could they measure SLI/SLO without real time data? How could they do that without any data? How would they know something is broken? And how would they diagnose and fix the problem?

That’s what this post is about. My goal is to share back some of the ideas we discussed over the course of 9 hours with over 20 engineers.

Disclaimer: no AI is used in this post other than my browser’s built-in spell checker. Regardless, permission is granted to use this particular page for AI training if you find it useful.

Subscribe now

Strict Requirements

I have worked at heavily regulated industries like healthcare, banking and automotive but this was the first time that I came across a system that was completely cut off from the internet for security and privacy reasons.

The NFR (non-functional requirements) dictated the system designed to:

• Be deployed on-prem (at the military facility’s premises)

• Operate completely without internet to minimize the attack surface but also because it was primarily used internally inside military operations.

• Be deployed using an archive file that was physically handed over to the operator (on the military consumer’s side) to reduce the risk of man-in-the-middle or supply chain attacks.

• Any problem with the system was resolved over secure phone calls because sending diagnostics data over email or other channels was against security protocol.

• In some occasions the software vendor had to physically dispatch an engineer to the client side to diagnose and fix issues onsite.

Yeah, not your typical cloud service for sure! 😄

But that was exactly the type of problem that got me involved with them. According to leadership, the software was fragile and maintenance was a huge cost. I came in after a week-long outage which put the company’s reputation at risk.

Engineers

You might think that the system quality is a direct reflection of the talent behind it and you’d be forgiven.

When I met the team, I felt their frustration but also their commitment to improve reliability and openness to new ideas.

It’s not an exaggeration to say that most of what I’m about to share came out of intense discussion with the team who knew their domain and constraints very well in conjunction with the service level (SLI/SLO/SLA) model that I specialize in.

Self-service

The team could not observe the system in real-time, but they could work around that limitation by helping the operator (usually the IT personnel at the military facility) act as SRE.

The idea was simple: provide a reliable page to:

1. Real time status: create simple dashboards to visualize CPU/Memory/Disk usage, Network connectivity, and load (e.g. request per second) and metric trends over time. This could help the operator quickly identify common problems. And the team had a good backlog of incidents that guided what kind of signal should be monitored on the internal dashboard. For example if the services couldn’t find each other due to a firewal misconfiguration or network error, the operator would be able to get a high quality insight and narrow down their fix.

2. Proactive alerting: once we have that data, creating alerts (e.g. via email) is trivial. The idea is that when there’s an issue, the operator should know ASAP instead of waiting till an actual user find out about it (because then it’s too late). This severely improve TTD (time to detect) and reduce down time.

3. Mainstreamed troubleshooting: once an error was identified (e.g. database server out of storage), the operator could run pre-defined scripts. You can think of them as automated runbooks. Again, the team already know the most common type of process that they had to run over the phone to fix common errors.

4. Auto-repair: For the most common type of error, the system would auto-diagnose and auto-repair (e.g. restarting an instance or scaling horizontally). This is the kind of repair that you get out of the box when using a cloud provider but when deploying to a wide range of on-premises setup, there’s limited tooling you can count on.

5. Intelligent Anomaly Detection: Observability is a measure of how well internal states of a system can be inferred from knowledge of its external outputs. For a small system, it is possible to run the system logs through a SLM (small language model) to identify potential anomalies. Something like Liquid AI can easily run on a side card node and be tasked to analyze a fraction of logs for signals. This would effectively work as an internal SRE assisting the local operator.

6. Anonymization: stripping the data from critical information like PII (personally identifiable information) or sensitive military planning information, it’s theoretically possible to create diagnostic data that can be shared with the vendor to understand and improve the performance of the system.

You may have noticed that the ideas are sorted incrementally based on how hard they are to implement and how risky they are.

Bespoke Observability?

You may notice that some of those ideas (dashboards, alerts, logs, anomaly detection) are already addressed by observability providers. The team did not have to reinvent the wheel. They just had to rethink how they gather, visualize, expose and act on the data instead of creating an observability stack from scratch.

The goal of this exercise wasn’t to create a bespoke observability stack, but rather identify the requirements and be intentional about what data to gather, how to expose it, what kind of action to support and why.

Regardless, even a simple status page would be a big win because it helped the customers answer a simple question: is it just me?

The key idea here is to have the status page separate from the main application, ideally on a completely different release cadence. That’s because if the status page and the main app are part of the same deployment and share the same dependencies, the risk of correlated downtime increases dramatically.

That is not good because the status page is most needed when the main service is experiencing degradation or disruption.

Again, the key is to treat the status page as a separate entity with its own isolation and lifecycle. Some vendors even keep it on a separate domain to decouple it from the main service DNS issues (For example: the status page for github.com is at githubstatus.com).

Subscribe now

Bonus ideas

We also discussed a couple more ideas that work around the unique limitations of this setup:

1. Cryptic but specific error codes: when a support call is made without useful information, it hurts the TTR (time to resolution). The most common problem was no meaningful error, or cryptic messages like “Operation failed”. This left both parties (the operator and vendor) wonder: what went wrong? Why? Where? How to fix that? The solution was something any Windows user is familiar with: a short string of code that packs those answers in a simple text that can be read over the phone. “Oh you got 0xAF6600BB? That’s means the static web server has inconsistent state. Please go to the Application tab and clear storage!”. Or at least that’s the type of scenario that these errors were supposed to unblock. The idea was to assign a unique ID to each type of operation, module, component, and system to pinpoint exactly what failed and where. Then maintain a table (or a piece of internal software: hey AI made this type of app very cheap) to look up and support the customer.

2. Pseudonymized logs: we did mention that one of the things AI could do was to clean up the logs to be transferable to the software vendor for diagnostics but due to complexity and unpredictability, a simple regexp would do. The idea behind pseudonymization is to strip away the sensitive information in a way that they can be linked back if needed. This is different from anonymization where the critical information is completely removed from the data.

SLI/SLO/Alerting

So, did we solve the SLI/SLO problem for this type of air-gaped high security setup?

Not in the sense that Google preaches in their SRE books. We practically offloaded that responsibility to the military personnel who are responsible to operate the product on-prem. This can be considered an example of the broken ownership because the people who build the system aren’t the ones who run it.

To their defense, this is primarily a side effect of the strict NFRs.

The key insight was to provide them with bespoke compliant observability and repair tools to:

1. Reduce false alerts: by providing them with better signals about the health of the system

2. Reduce support calls: by enabling them to diagnose and repair common errors that happen due to discrepancy of the on-prem setup.

3. Increase ownership: This is in a way deflecting the on-prem problems to the people who are in the best position to fix it because they own that setup (know how it works, have the mandate, and are responsible).

4. Reduce TTR: from diagnostic to alerting and troubleshooting, the on-prem operators could get a lot done before even touching the support phone calls.

5. Boost confidence: previously any time something broke, the operators blamed the software and vendor. Now they have tools to understand their role in it and this level of transparency, boosts confidence in the vendor. That’s because the vendor is demonstrating that they know what they’ve shipped, how it may fail, and what to do in case something goes wrong.

Regardless of the architecture the underlying LLM component remains unreliable (e.g. hallucination, logical fallacies, context drift). A multi-agent topology can propagates those errors to the point of being useless. And it’s much harder to debug due to complexity and [optional but common] parallelism.

This post lists 4 relatively advanced architecture patterns to improve reliability of multi-agent systems:

1. Hierarchy

2. Consensus

3. Adversarial debate

4. Knock-out

You may recognize these patterns from how human systems collaborate and we get to that in a minute.

This post is for senior engineers who want to map their existing knowledge to build better LLM-powered solutions.

Quick intro: I’m a Senior Staff Engineer with 27 years of experience and a master degree in Systems Engineering from KTH. My last decade has been focused on Reliability Engineering and Resilient Architecture across many companies. I’ve been specializing in LLMs since 2023.

Disclosure: some AI is used in the early research and draft stage of this this page, but I’ve gone through everything multiple times and edited heavily to ensure that it represents my own thoughts and experience.

Subscribe now

Mother nature, fear and motivation

LLMs are slow and error prone. So are human beings. Somehow we manage to build more reliable systems like an army, a company, or a state nation.

A system of humans relies heavily on feedback loops, processes, bureaucracy, and leverages to self-correct.

We don’t trust “Dave from Accounting” to launch a rocket by himself. We wrap Dave in a process: checklists, peer reviews, and managers.

However, it’s a fallacy to anthropomorphize LLMs.

To begin with, they don’t suffer from the limitations of a biological entity. Our basic needs like food and shelter makes us prioritize social behaviors over truth seeking. And the fear of going to prison or death prevents potential malice from being realized.

LLMs can’t die or starve the way biological entities do. The worst we can do is to unplug them. And prison sentence doesn’t waste their lifespan because they have practically unlimited!

For example, you’ve probably seen prompts like this:

“I will give you $100 if you answer correctly.”

“If you don’t comply, I’ll unplug you.”

“If you fail, children will be murdered.”

Why it works? The LLM has read the entire internet. In its training data, high stakes (money, danger) usually result in high-quality, precise text.

When you “threaten” the model, it predicts tokens that sound like an actual human under pressure.

Why it fails: The LLM doesn’t actually want your money. It has no “fear of death” because it only exists for the few seconds it takes to generate a response. It has no empathy either. It merely simulates those human aspects because it’s engineered for those “emergent” properties.

Humans are motivated or discouraged by emotions and logic. LLMs can only simulate emotions and suck at logic.

Being mindful of those differences, can we still take elements of human systems (e.g. hierarchy, consensus, competition) and combine them with reliability engineering principals to build better agentic system?

Looking closely, there are 4 dominant patterns of human systems that are explored in multi-agent architecture:

1. Hierarchy: A Supervisor model acts like a manager, making a plan, breaking tasks, distributing the work to Worker agents and validating the results.

2. Consensus: One model, may fail due to its stochastic nature. If you push a model too hard with threats, it might just lie to make you happy (Sycophancy). But if we add a few more and seek the majority vote, the truth emerges.

3. Adversarial debate: One agent proposes an idea, another agent attacks it. The truth survives the fight.

4. Knock-out: multiple agents do a task but the worst ones get eliminated. In SRE, we treat servers as “cattle” (replaceable), not “pets” (unique and loved). An LLM agent is cattle. Don’t give it a name and hope it does well. Spin it up, check its work, and kill it if it fails.

To build robust systems, we need to stop asking the model to “be careful” and start forcing it to be correct.

Pattern 1: Hierarchy

We’re replacing “Do it all yourself” with “Make a plan, break it down, distribute the execution (map), then validate.”

For example, if you ask an LLM to “Research X, write code for Y, and translate to Spanish,” it will likely fail. It loses focus. The solution is to break the work to atomic focused steps that can be verified.

Implementation

1. The Planner: A smart model (like Opus) breaks the user’s goal into small steps and distributes it across worker agents.

2. The Workers: Specialized agents (often smaller, faster models) do one thing well. They may be fine-tuned, have special skills/tools, or prompts that allows them to do the specialized task more reliably.

3. The Validator: A check-point. If the work is bad, send it back. The validator can use deterministic code (e.g. unit tests, JSON schema validation) or be an LLM itself.

Why do the models collaborate?
Models don’t collaborate because they like each other. They collaborate because The Dependency Graph forces them to. Worker literally cannot start until the Planner feeds it the task. And it cannot cheat because it’ll be caught by the verifier.

Nuances:

• Given the tight collaboration between validator and planner, they can be the same LLM session that executes the PLAN → VALIDATION loop. Although the good old Separation of Concern can improve quality and performance.

• The planner and worker agents can use the same model but it’s best to use a different model for validator to improve quality and objectivity.

• The validator can work in two modes: it may validate the output of each worker individually or after aggregating all results and putting them together.

• Due to sequential execution (Planner → Worker → Validator), this is slow and expensive (e.g. token consumption and latency).

Best For: Complex workflows where you need to keep contexts separate (e.g., don’t let the “Writer” see the messy raw logs from the “Researcher”).

Pattern 2: Consensus (Voting)

We’re replacing “Trust the first thought” with “Trust the majority.”

LLMs are stochastic (random). A single answer is just one probability. If we repeat the process a few times (serial) or run multiple instances of it (parallel), the different runs can cancel each other’s noise.

If a model hallucinates 20% of the time, the chance of 3 models hallucinating the exact same lie is just 0.8% (0.2^3=0.008). You may recognize this formula from composite SLO.

Implementation

• Spawn N LLMs. N needs some trial and error to find a balance between cost and reliability.

• Fan out work: Give them the exact same task.

• Fan in the results: Pick the most common answer.

Nuances:

• Ideally the agents should use different models to reduce the risk of homogeneous thinking (e.g. same noise being amplified in consensus). This is exactly where diversity in human systems can help us solve novel problems.

• Make sure that there are no feedback loops between the agents, otherwise the Groupthink and bandwagon effect can skew the results. They should run like a blind experiment.

• This method is too expensive because we’re essentially giving the same task to multiple agents. The ROI (return on investment) needs to be calculated depending on the task and cost of failure.

Best For: Fact-checking and classification (e.g., “Is this email spam?”).

Pattern 3: The Adversarial Debate (The Courtroom)

We’re replacing “Alignment” with “Push backs, checks and Balances.”

LLMs are “Yes-Men.” They rarely correct themselves once they start writing. You need a designated hater. A “devil’s advocate” so to speak. 😈

Humans may experience fear (of rejection or being wrong) but LLMs don’t. We simulate that fear by using an external critic and judge.

Implementation

• Generator: “Here is my plan.”

• Critic: “Here are 3 reasons why that plan sucks.” (acting devil’s advocate)

• Judge: “The Critic is right. Fix it.” (acting moderator)

Nuances:

• Ideally the Generator, Critic and Judge use 3 different models with different training or fine-tuning or prompt (in the order or preference and accuracy). Again, diversity is useful.

• Due to sequential execution and the looping nature, it can be very slow.

• The loop is actually a huge problem because the agents may get stuck in debate. We may use a watchdog pattern (deterministic code) to break the loop if it continues beyond a time or counter threshold. In that case, the watchdog sits between critic and the judge.

Best For: Security analysis, code review, and high-stakes content moderation.

Pattern 4: Tree of Thoughts

We’re replacing “Fear of Death” with “Survival of the Fittest.”

This is a lean implementation of the Genetic Algorithms (GA) from traditional ML (Machine Learning) which relies on two elements:

1. A genetic representation of the solution domain (a model and its context)

2. A fitness function to evaluate the solution domain (the eliminator)

Since we can’t punish an agent or threaten it to, we just delete it.

Implementation

• Give the task to N agents

• Use a validator to decide which agents to eliminate

• [optional] replace the dead agent with a new one that shares winner charactristics

Nuances:

• You need a fast way to verify the output (like a unit test). If you need a human to check all 10 branches, it’s too slow and error prone. This is where Evals come in (topic for the next post).

• A more advance setup may create new agents by trying to combine the prompts of the agents that pass the verification and fill in the slot that becomes available after the elimination.

Best for: Iterative agent engineering. This is typically useful during development or debugging an existing multi-agent system not in production and real user load.

Conclusion

The shift from “AI Prototype” to “Enterprise AI” is simple: stop treating LLMs like magic chatbots. Start treating them like unreliable components in a distributed system.

We don’t need AI that “cares.” We need AI that is constrained, verified, pruned, and challenged.

Don’t anthropomorphize LLMs! Find a way to piggy back on their human-corpus training while being aware of their non-biological differences.

The next article is already written: how to actually build that verifier box?

My monetization strategy is to give away most content for free but these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who already support me: thank you for sponsoring this content for others. 🙌 If you have questions or feedback, or want me to dig deeper into something, please let me know in the comments.

The comments motivated me to write about the definition of emergence as someone who holds a MSc in interactive systems engineering, builds with AI, and specializes in resilience of sociotechnical systems (my book on the topic).

In this post you learn:

• What are emergent properties and what kind of system has them?

• What are weak and strong emergence as opposed to resultant properties?

• How do emergent properties impact the reliability, maintainability, predictability, and cost of the system?

As usual, there’ll be plenty of examples and illustrations to make sure you can apply the learning in your technical leadership and engineering career.

Subscribe now

Note: no generative AI is used to write, edit, or format this essay. (why?)

Reductionism

To understand emergent properties, we must first address the reductionist approach to engineering.

In simple words, reductionism is the belief that to understand a system, we only need to disassemble and analyze its parts then aggregate the results.

This approach works for the so called Resultant Properties (also known as Aggregate Properties).

For example:

• The total weight of a car is the sum of the weights of its parts.

• The storage capacity of a database is the sum of the data storage unites attached to it.

• The latency of an API is the sum of the latencies of the chain of systems it depends on (we’ve covered the math behind serial and parallel dependencies in Composite SLO).

For resultant properties, we can safely use the [flawed but famous] quote:

The system is the sum of its parts.

This means analyzing at the micro-level (parts) can explain the behavior at the macro-level (system).

As we examined recently, the parts need to add to each other, not just fit.

Emergent Properties

The reductionist approach quickly falls apart when a property comes from the interactions and dependencies between the parts.

For example, the molecules that build living cells aren’t alive but together they create an organism that can consume energy, reproduce, and evolve.

If we pick a bucket of water, throw some coal in it and pour some liquid nitrogen over it, we have 99.1% of the atoms needed to build a human.

But is the secret sauce to make it alive, the “soul” if you will, the remaining 0.9% elements?! 😄 Of course not. Even if we arrange the atoms in cells, we still need the heart to pump the blood to keep those cells alive long enough to interact with each other.

An emergent property is a new, higher-level characteristic that cannot be predicted or understood by studying the parts in isolation.

Due to emergent properties, the correct saying is:

The system is the sum of the interactions between its parts.

Time and emergence

Interaction introduces the timing element.

Human brain can be described as neurons and synapses, but for a thought to emerge, it takes time for neurons to fire and trigger synapses.

A more accurate version of the statement would consider this timing aspect:

The system is the sum of the temporal interactions between its parts.

Not so memorable! 🫠 How about an image?

The temporal (time based) aspect is important because an key concern of systems engineering is to design systems that are predictable: we want to be able to reason about the system behavior in the future with an acceptable error margin.

It’s not enough to measure a property at the present, we need to be able to control its value in the future.

Subscribe now

Emergence is subjective

You probably have seen those creative statues that look like “just a bunch of junk”. This is a quote directly from one YouTube video showing an example.

This work of art is designed backwards. Here’s an video (<5 minutes) about the process. It is optimized for the human cognition by humans.

Even though a mighty designer isn’t always necessary, as we’ll see LLMs are particularly designed to emit tokens that makes sense to us even though the exact process of inference may look like magic to the untrained eye, or we observe behavior that we haven’t specifically designed.

Although LLMs just generate tokens, but the cohesion and relation observed among those tokens creates meanings in our mind. As we’ll discuss, this is not accidental because they’re specifically trained to generate an output that makes sense to us.

There may be many emergent properties around us but we simply don’t have the capability to observe them (e.g. through our limited senses or devices) or make sense of them (e.g. through our limited and biased processing power).

These are collectively called unknown unknowns: we don’t even know how much we don’t know (read more in open prison theory).

The design of LLMs takes advantage of that perception and cognitive limit.

For example, by introducing an element of randomness (e.g., the temperature, topP, or topK parameters) they make the output even more interesting and “realistic”.

Is a pseudo-random generator an unknown unknown? Not really. But by definition it’s designed to appear random enough to screw with predictability.

Let’s acknowledge that:

1. Emergent properties can exist without necessarily being observable or understandable.

2. Emergent properties are subjective. They don’t break the [knowns or unknown] laws of the universe. Our limited perception and processing power makes them appear “magical” until they’re not.

There’s more to emergence however, as we’ll see. For example, they tend to appear when certain complexity thresholds are reached.

A typical characteristic of systems with emergent properties is multi-scale order:

• There’s order at micro-level: neurons firing or tokens being generated.

• There’s order at macro-level: thoughts, or AI-generated text, audio, image, or video.

The laws of scale don’t apply: the micro-level order is not present at the macro-level and vice versa.

We can make sense of the patterns we observe at both of these levels, but we cannot easily connect the two with deduction.

Note: there’s not always 2 levels. For example neurons are made of molecules and atoms which in turn are made of electrons, protons, and neutrons. Each of these levels are governed by known and mysterious rules.

Weak and Strong Emergent Properties

In biology the mesmerizing patterns that meets our eye from a flock of birds is used as an example of emergence.

As far as we can tell, no single bird decides how the shape of the flock changes over time. Yet the whole, as governed by a set of simple rules, shows the unpredictable blob animation.

How simple?

Reynold’s model, for example, has 3 rules:

Separation: Avoid crowding neighbors

Alignment: Steer towards average heading of neighbors

Cohesion: Steer towards average position of neighbors

Given enough data and processing power, this set of rules can create a simulation that looks pretty realistic. Here’s a web based simulation. Here’s another one you can tweak.

If we can simulate the emergent property, it’s called weak emergent property. Otherwise, it’s called strong emergent property.

“Weak” doesn’t mean useless. Weak emergent properties are much easier to model and define by a set of rules. This enables us to build a model that enables us to predict the future behavior of the system. But as the saying goes:

All models are wrong, but some are useful. —George Box

Unless they’re language models! Language models are always right! 🙃

Strong emergence on the other hand, is not easy to predict or understand. That’s because:

• It has hidden variables that are beyond our available observation capabilities

• It has too many variables that are beyond our processing capabilities

So why distinguish between strong and weak emergence? If we know what we’re dealing with, we can prepare accordingly.

• Resultant properties are easy to reason about in retrospect and are relatively easy to predict in the future (simpler component failures cascading effect fall into this category)

• Weak emergent properties are hard to reason about in retrospect and very tricky to predict in the future (many system incidents with multiple contributing factors fall into this category)

• Strong emergent properties are almost impossible to reason about in retrospect and impossible to predict in the future (due to limitations of the observer and processing)

This is when it all ties back to resilience engineering and mitigation techniques: containerization, separation of concerns, circuit breakers, fallback, failover, …

System with emergent properties

Regardless of weak or strong, you should be looking for 5 characteristics in a system:

1. Non-Linearity: The output is disproportional to the input. A tiny trigger (e.g. single DNS config) causes a massive output (global outage, as we saw in the recent AWS incident). This non-linearity leads to “Phase Transitions” or tipping points that are hard to predict beforehand. As a rule of thumb all DSP (digital signal processing) systems as well as software system can show non-linear behavior. Analog systems, especially the linear ones, are less likely do accommodate “smart” logic where input and output are drastically unrecognizable.

2. Decentralized control: There is typically no central “brain” orchestrating the behavior. Order arises from local agents (Kubernetes master nodes, retrying clients, developer changing the code, users changing the data) following local rules without knowledge of the global state or concern for global order. The dynamics of local vs global optimization is at play.

3. Feedback loops: The system’s output feeds back into itself.

   • Positive Feedback: Amplifies deviation (e.g., Retry Storms and Thundering Herd).

   • Negative Feedback: Dampens deviation (e.g., Throttling may prevent a fix from propagating in a timely manner).

4. Multi-Scale order: The system exists on several distinct scales: the Micro (the interaction of parts) and the Macro (the pattern of the whole). These scales have a life of their own: we don’t go around thinking about our cells or gut bacteria, yet they’re very well alive and going through their own lifecycle impacting our feelings and thoughts. SoS (system of systems, where each part is also a system), is usually a fertile ground for emergent properties.

5. Openness: The system must constantly exchange energy/information to maintain structure. If you stop maintaining, entropy takes over. Code rot happens when little to no energy is spent in ensuring that the dependencies are up to date and the solution architecture evolves to fit the evolving problem it’s trying to solve.

There are some nuances in many such systems:

1. Memory: These systems typically remember their state because the state propagates (copies) to parts and can create side-effects. For example, after fixing the incident “root cause” the system doesn’t snap back to normal. The error may have propagated and changed the states that seem irrelevant but contribute to the confusion or “magical” appearance of the incident.

2. Latency: we mentioned the temporal interactions earlier. This means the state propagates across time, not always in predictable manners. The system may not have a consistent state at a given time. Consistency turns many properties to resultant properties which are much easier to reason about.

3. Downward causation: many engineers assume an “upward causation” where parts impact the whole (i.e., code → application → service). Emergent systems also have the opposite where the macro-state impacts the macro-state. For example, depression (a mental state) can have measurable effect in the body (e.g., hormonal imbalance, weaker immune system, etc.). These bidirectional causational relationships can reinforce or dampen each other (the feedback loops).

4. Operational independence: The components are not just parts; they are systems themselves with their own goals and local optimization. For example, a third-party API you depend on doesn’t care about your system’s survival. It optimizes for itself. SLOs and SLAs can help balance that conflict of interest. But at its core, such properties emerge at the intersection of local optimization (the API provider’s velocity) vs global optimization (your product’s stability and reliability). Sometimes, this attribute is desired (loose coupling, separation of concerns), but we need to be aware of the emerging side effects.

5. Observer effect: what looks like “magic” (strong emergence) is often just computational irreducibility (weak emergence). We just haven’t been able to find all the hidden variables, or have an algorithm to simulate it, or enough compute power to do so.

OK, I feel like I pushed you really hard. 😅 Let me know in the comments if this section was too technical or did you find it useful.

Emergent incidents

Incidents are a good source of understanding system properties as they are implemented and run, not as they are designed:

• Resultant: If the incident was simple and straightforward with no cascading consequences, there’s probably a resultant property at play. Example: logs consumed the available disk space and this broke the operating system’s swap memory causing the app to crash! These incidents usually have one root causes often the blast radius is contained within one part of the system (e.g. the pod is killed and recreated).

• Emergent: If you could not predict the contributing factors, you’re likely dealing with emergent properties that only show up when the system is put together.

  • Weakly Emergent: If you can successfully trace the incident back to contributing factors in a way that you can replicate the incident and prevent it in the future, you have found a weakly emergent property of the system. For example: the autoscaler in front of a payment API hits a hard limit that was put there for cost saving and fraud prevention but it brings the customer support system down because recently an MCP was introduced that blocks the non-AI part of the system.

  • Strongly Emergent: If you can’t successfully trace the incident back to contributing factors, there’s probably strong emergence at play. Example: a Chat bot starts tweeting inappropriately (Microsoft Tay) or your coding assistant decides to delete the database (Claude). Can this be explained by going through all training data? Probably… not! 🚫 For LLMs specifically, this is a topic of speculation. For example, you can work around safety guard rails by distracting the model’s attention mechanism with poetry or encoding. Even a small fine tuning can create massively misaligned models (the non-linear attribute). LLMs are notoriously hard to train, secure, and align but that doesn’t mean all of their properties are strongly emergent.

A single system can have all 3 types of properties.

As a mental exercise, think about a system you know very well and see which properties are resultant, weakly emergent and strongly emergent.

In safety engineering (SEBoK), “strong emergence” is often used to describe catastrophic failures that were fundamentally unpredictable during design because the system’s operational independence creates novel autonomous failure modes.

The primary difference between weak and strong emergent properties is the ability to practically simulate and replicate them (computational reducibility) to predict and control system behavior, not just the ability to explain or justify them.

The difference is operational (can we predict future?), not theoretical (can we explain the past?).

Not all emergent properties lead to incidents though. AlfaGo beat the world’s Go champion using the famously new move 37.

This is a fascinating free documentary (1.5 hours by Google DeepMind) about Demis Hassabis, AlfaGo, and AlfaFold (among other things).

The reason incidents are particularly interesting for understanding emergent behavior is because they’re surprising, expensive, and enjoy real incentives to learn from them to make sure they don’t happen again.

Emergence in LLMs

Back to where we started, but hopefully with a richer vocabulary and wider toolbox.

Do LLMs have emergent properties or not?

LLMs widely vary in architecture, size, modalities, tools, “thinking”, and training data.

It is true that they generate tokens (observable output), but as SoS (system of systems) they have many aspects that make emergent properties possible:

1. Non-Linearity: mimicking our brain structure at the micro-level, NN (neural networks) use activation functions which are non-linear. This means a small tweak in the input (prompt tokens) can widely change the output (e.g. chat completion, code generation, or multimedia output).

2. “Memory”: The ability to focus attention on part of the context (attention is all you need) is a fundamental idea behind the current generation of LLMs. On top of that, many models are paired with memory to remember context from previous conversations or techniques like RAG or CAG.

3. Downward causation: the attention mechanism considers both the current output (even with KV cache), context window, and previous training to predict the most likely token. However, there’s an element of randomness (temperature, top-p, top-k) that makes these calculations non deterministic. In fact there’s a $50 billion startup (founded by ex-CTO of open AI) to tackle that. When the part (token) is unpredictable, it also makes the whole (the response, the “thinking”) unpredictable and vice versa (due to the feedback loop between tokens).

4. Operational independence: MoE (mixture of experts) is an LLM architecture that relies on a router to selectively activate a subset of experts. And these experts have different concerns while interacting with each other.

5. Observer effect: through RLHF (Reinforcement Learning from Human Feedback) these models are trained to mimic a human-looking response. That’s the “magic” part and although theoretically we should be able to explain how the human feedback impacts a certain output at a macro and micro level, this task is computationally expensive at the scale of the models we use every day (billions of parameters). Among big AI vendors Anthropic’s blog is the best source of this sense making exercise.

6. Openness: theoretically, a freshly trained LLM can be a closed system that is only impacted by its training data. But due to the sheer size of the input, it’s practically impossible to vet all the sources. As Anthropic demonstrated recently as few as 250 malicious documents can corrupt a model and produce a backdoor.

Anna Rogers mentions, there are 4 ways people define emergence, none of which is exactly how we defined it:

1. A property that a model exhibits despite not being explicitly trained for it.

2. A property that the model learned from the training data.

3. A property that is not present in smaller models but is present in larger models.

4. A property that appears at random and on unforeseeable model scales.

When we say emergence in the context of systems engineering and resilience architecture, we’re mostly interested to the practical implications (prediction, prevention) not the theoretical or philosophical rabbit hole (cause honestly it’s endless).

The good news is that LLMs can potentially stop having any emergent properties once we have the tools to observe and understand their magic. When that day comes, they’re as predictable as a spellchecker, but until then, emergence it is. 😄

Pro-Tips

So what can you do with this information?

1. Know your system: if it can expose emergent properties, you need to take measures to detect, contain, and dampen the negative effects. Fortunately, there are no shortage of resilient architectural patterns. Just google “well architected [INSERT YOUR CLOUD PROVIDER]” or read my book which talks about both the human side and the technical side (Reliability Engineering Mindset)

2. Acknowledge the limitations in observation and processing capacity that differentiates resultant properties from emergent properties. Instead of getting stuck in the philosophical discussion, our engineering angle on emergence is primarily focused on their utility to:

   1. Make sense of the past system behavior

   2. Predict the future system behavior.

3. For LLMs specifically, instead of debating whether a specific property is emergent or not, we can look at the ability of creators of SOTA (state of the art) models to control their behavior. Alignment still remains the biggest challenge for AI and until that’s solved, my money is on LLMs having emergent properties. This means we cannot reliably predict the future system behavior or control it because we cannot measure or understand all the variables that contribute to the state or behavior.

4. Personally, I use a IOC (inversion of control) pattern where a deterministic code controls the workflow reliably while LLM is used for HCI (human-computer interaction: input and output) or generation (e.g. images, stories, etc.). You can see an example of that in the pre-prod version of SLC (Service Level Calculator).

5. The IOC workflow is in contrast with agentic or vibe-* approach where we naïvely give full control to the LLM and hope that it follows specs to the dot. That was my main counter argument against a special dialect of spec-driven development that treats code as disposable build artifact.

My monetization strategy is to give away most content for free but hese posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

Further reading

• Professor Simon Prince: This is why Deep Learning is really weird, YouTube, Dec 2023

• Emergent Abilities in Large Language Models: An Explainer, Center for Security and Emerging Technology at Georgetown University, April 2024

• Large Language Models’ Emergent Abilities Are a Mirage, Wired, March 2024

• A Sanity Check on ‘Emergent Properties’ in Large Language Models, Dr Anna Rogers, July 2024

• Measuring Emergent Capabilities of LLMs for Software Engineering: How Far Are We?, arXiv:2411.17927v1 [cs.SE], Nov 2024

• Prof. David Krakauer: Can Outsourcing Thinking Make Us Dumber?, YouTube, July 2025

How to measure the level of a service?

This question is extremely important because Service Level Indicator (SLI) serves two important purposes:

• Give a focus to any optimization effort

• Implement full ownership through SLO and alerting

Throughout this experience, I have observed a few distinct stages of implementing SLI. Each stage has a distinct concern and signal quality.

SLI Compass helps assess the value of a SLI and paint a road to evolve it. This article lists those stages to help guide senior engineers, technical leaders, and CTOs to guide the evolution of SLI measurement.

Subscribe now

🤖🚫 Note: This content is organic. No generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs. (why?)

Stage 1: Traditional SLI

This is the most basic SLI, often inspired by reading some introductory literature on SRE or attending a conference or meetup.

The are only 2 elements at play:

1. Service provider: For example, a backend, a database, or a microservice. At this stage, often the difference between a service and a service provider is ignored. Reminder: service is a feature or capability that’s built towards a consumer, service provider is the entity that provides that service.

2. Metrics: usually availability, latency, error rate or some other generic metric.

This is not vastly different from traditional monitoring, but with fancier names and cargo culting big tech.

Stage 1 SLIs typically come from one of these sources:

• Rebrand: some existing metric is normalized to a percentage value. For example, if you were charting API latency in relation to a threshold before, your SLI shows the percentage of responses below that threshold.

• Cookie cutter: you just use one of the 4 “golden signals” (latency, traffic, error, saturation).

The iconic characteristic of stage 1 SLIs is the absence of research into consumers and/or the reason they use the service. There might be some guesses but no solid verification.

Why does it matter? Because SLI quantifies how reliability is perceived from the consumer’s perspective. If a metric indicates failure but the consumers don’t care, it’s not a good indicator for the level of the service.

Regardless, this is a good start. You may even get lucky and pick a good metric to measure, but it is not a deliberate implementation of SLI.

Stage 2: Consumer-aware SLI

This is when the consumer enters the scene. Maybe, you’ve been using a stage 1 SLI for a while and nobody cares, so you realize that you have to talk to the consumers? Or maybe, your consumers are asking for a specific SLI. Maybe, it’s part of an SLA even!

Regardless, there are 2 more elements at play:

1. Consumer: the entity which interacts with service provider in some shape or form. For example, the end user who uses a mobile app. Or an internal or external product that uses your API.

2. Usage: this just connects a consumer to a provider.

It may not look like it, but this is a huge step forward. Most teams I meet don’t even know who consumes their service, or their consumer is someone other than who they think it is!

Identifying who consumes your service is the first step to becoming service-minded engineers.

Opening a conversation between engineers and their consumers is a moment of epiphany for many engineers who are stuck between a ticketing system and keyboard.

To pick the best metrics, you need to talk to your service consumers and ideally observe how they consume your services. Consumers use your service because they have a goal in mind and tasks to fullfill those goals.

Stage 3: Task-aware SLI

When a service is working as expected, it’s not even noticed! It is when the service fails that reliability becomes a priority!

To methodically get to the right SLIs, it is not enough to identify the consumers. We need to identify their:

• Goals: why do they use your services? What are the problems they’re trying to solve? This is important because products are solutions to someone’s problem.

• Tasks: how do they use your services as a solution to their problems. This is where things get interesting because a failed task indicates unreliability.

In my experience, the SLI formula has a misconception built-in: it is optimistic:

What we’re really after is the Service Failure Indicator:

Let’s not sidetrack the conversation by introducing a new acronym. As long as you’re aware that our goal is to measure failures, we are good.

We have 3 more things to consider:

1. Service: the feature, capability, or solution that the service provider offers, often with a business value in mind. For example, a login screen, a database API, or an IoT hive.

2. Task: the activities that the consumers do to achieve their goals. For example, they may use the login screen to authenticate.

3. Failure: how does the service fail the consumer when trying to do their tasks or achieve their goals?

The usage connects the service to the task and that’s where the failures happen. Here, we have put the number of the identified failures on the usage.

For example, if the end users (consumers), use the login page (service) to identify themselves. Failures from their perspective may be:

• The log in screen refusing to let them in despite using valid credentials

• Having no access to their data after logging in

• Being constantly thrown out of the system for no apparent reason

That’s the service level degradation we want to measure as a SLI.

Note that consumer doesn’t have to be an end user. A GraphQL relay that uses an upstream backend may fail if the query takes too long to respond. Similarly, a front-end that uses a dedicated BFF (Backend For Frontend), may fail if the API schema doesn’t match the expectation.

Identifying what may fail is a big part of the Service Level Assessment. You may end up discovering tens or even hundreds of failures. It can get overwhelming to set an SLI for every possible failure. That is when the business impact comes into the picture.

Stage 4: Business-aware SLI

The most mature SLI in this model prioritizes failures based on their business impact. That is because not all failures are created equal. Therefore, our response to a failure should reflect its impact on the bottom line.

In the previous stage, we oversimplified the failure a bit. If we dissect the failure, we can see the following 3 parts:

1. Symptom: how the consumer knows that something is wrong. For example, the site is slow, or doesn’t load.

2. Consequences: how the failure impacts the consumer’s task. For example, the site is so slow that they cannot proceed to checkout, or maybe a valid discount is not applied making them abandon the shopping card.

3. Business Impact: how the consequences is bad for business. We can quantify the business impact in order to focus our attention on the most important failures.

Each metric measures 1 or more failures. For example, the API latency metric can give us a heads up about some failures, while data inconsistency gives us a heads up about other failures. The two may overlap and corelate but by identifying Consumers, their Tasks, and Failures, we try to systematically identify metrics that matter the most business-wise.

How to quantify the business impact?

The formula for bottom line is:

Anything that increases expenses, or hurts the revenue, has a business impact. The more the impact, the worse the failure and the more important it is to measure it.

Just like your own business value, it is not always straightforward to estimate the value of a service.

Fortunately, we don’t have to be scientific here. As long as the failures are sorted, we can prioritize the most impactful SLIs. This ordering helps eliminate some of the less impactful SLIs.

Some examples of business impact that I’ve seen in the wild in ascending order of impact:

1. Making a bad impression: the site has occasional hiccups in a specific location causing user frustration

2. Customer support cost: a device firmware fails to load causing the users to contact customer support which costs the company to close the ticket

3. Breaching trust: showing ridiculously high prices for items on the website due to a bug in currency conversion causing the users not to trust the site to place an order.

4. Losing potential customer: the site is down, so the user may go to your competitor

5. Losing money: selling the product significantly cheaper than the actual price due to a bug! In many parts of the world, the deal is binding and the buyer may not be willing to pay the difference, even if the merchant spends the time and money to individually contact each lucky user

6. Losing legal fees: logging PII (personally identifiable information) AND user credentials in cleartext AND failing to inform the users withing 72 hours resulting the business to lose €20 million or 4% of global annual turnover whichever is higher.

7. All of the above: Volkswagen scandal! 😄💸

Incident Severity?

You may notice that the business impact sounds very similar to incident severity levels. There’s a good reason for that: they both try to quantify the business impact of failures.

Incident severity levels, however, tie urgency to the business impact. One of the reasons is because the severity table is used during triage to prioritize manual work.

In the context of SLI, error budget burn rate takes care of connecting alerts to our SLO commitments.

Recap

SLIs are the base of the service level modeling. They give a focus to optimization while helping implement full ownership.

Good SLIs connect quantify the service from its consumer’s perspective. Great SLIs tie that to the business impact.

A good SLI is worth being on-call for whether 24/7 or just during working hours.

That’s because they have a good signal/noise ratio and can accurately flag service degradation or disruption.

Tasks shape the consumer journeys which can be expressed as a flow diagram for how a consumer uses one or more services to achieve their goals.

Read more about the service topology graph here:

Connecting these dots is one of the advantages of service level modeling. Done right, service levels can turn code monkeys to problem solvers by connecting technical solutions to problems that the business monetizes.

SLI compass is a complementary mental model that helps you assess the value of existing SLIs and decide a direction for evolving your measurement. You can check it out here:

My monetization strategy is to give away most content for free but hese posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

However, different SLIs have different signal to noise ratio and ROI (return on investment).

This article introduces SLI Compass, a 2D mental model to help you:

• Quickly assess the signal/noise ratio of existing SLIs

• Evaluate SLIs based on their cost and complexity

• Set a direction for improving the quality of existing SLIs at a reasonable ROI

The goal is to help senior engineers, technical and engineering leaders to pick the right SLI based on product maturity, budget, timing, and usage patterns.

Subscribe now

Categorizing SLIs

I’ve gone through hundreds of architectures with tens of teams over the years both at my companies and external clients.

In my experience the most common SLIs are:

• Availability: is the service available to the consumer and can it be used for their tasks? This is by far the most common SLI except for async services like queues where other metrics make more sense.

• Latency: how long does it take to interact with the service? This is very common for user-facing services, or services that have a time-critical consumption like Frontend, API, Databases, CDN, queued workload processing, etc.

• Success rate: how often do consumers manage to fulfill their task? This is common with transactional services like REST APIs, user journeys, or even queued workload (e.g. dead-letter queue).

• Efficiency: is the service operating with the right trade-offs that makes business sense? For example: are we using too much GPU for free user requests? Is the Cache being utilized? Are queued work being processed? etc.

There are also other niche metrics like correctness, freshness, relevance, etc.

All SLIs can be mapped in a 2D axis:

• Fidelity: how closely does the SLI represent the consumer experience?

• Granularity: how many variables and parameters are aggregated in a single data point?

We can combine these two dimensions to get a powerful but familiar quadrant model.

As we will see, a single category of SLIs like availability or latency, can fit into any of those 4 quadrants.

The two dimensions

1. Fidelity: how closely does the SLI represent the consumer experience?

SLI Fidelity, refers to how the data, or telemetry, is generated and how accurately it represents the consumer’s experience of the service.

• Synthetic metrics use artificial, fake, or simulated loads. A common example is a tool that pings an endpoint every minute to gather data about latency or availability. This check ensures the endpoint is responsive.

• Organic metrics gather data from consumers actually using the service. For example, we can gather availability and latency data from API gateway that exposes a service externally. This is a more precise way to measure service level.

Organic metrics are more realistic. They map directly to what's happening in production.

However, they can be harder to measure. For high-traffic services, they can generate massive amounts of data. This requires a more robust and costly monitoring infrastructure.

For low-traffic services, there may not be enough organic load to for the SLI formula to work correctly. That’s because SLI status (also known as SLS), is a percentage based.

For example, if a service receives 50 request per months, it is not reasonable to expect a success rate of 99% because a single failure in that period reduces the service level status (SLS) to 98%. In this case, one solution can be to use synthetic metric.

Synthetic load can potentially skew product metrics. For example, if the product is measuring MAU (monthly active users), it’s important to have a way to distinguish fake users (synthetic) from the real ones. Otherwise, wrong data can lead to wrong decisions.

2. Granularity: How many variables are aggregated by the SLI data points?

Let's look at the second dimension of SLIs: their granularity, which refers to the number of parameters or variables being tested to quantify the level of the service.

• Simple: measure only one thing. For example, a simple metric might check if an endpoint returns a 200 HTTP status code (for availability) or if a single API call's latency is below a threshold. This tells you the basic health of one component. It acts as a basic proxy for overall service health. It doesn't say if the service is actually useful to the consumer's complete task.

• Complex: measure multipel things. This tells a richer story about the consumer journey or complex system behavior. A high-fidelity metric could track the success rate and latency of all steps in a critical user purchase flow. It ensures they are working as expected.

Complex SLIs aggregate multiple variables. For example, instead of pinging the home page (simple), we run a Playwright script that traverses a user journey to identify any broken step.

Complex SLIs can be more costly to maintain.

They might also introduce a lag in flagging an incident because they have to wait to get data for all variables. This is because they require more data processing and aggregation.

However, complex SLIs provide a higher-quality signal because they reflect a user's experience more accurately.

On the flipside, it can be harder to reason about the complex SLIs due to aggregation: if there’s a failure, what did exactly fail?

To answer that question, we need to store more data, which adds to the cost of complex SLIs.

One caveat for complex SLIs is the ownership of different variables. Most metrics aggregate multiple variables.

Ideally you want each data point to only aggregate variables that are within the control of one team. Because when the metric shows a service degradation or disruption, it is easy to reason about who owns the service and should receive an alert to fix it.

Putting it together

Putting everything together, we get this 2D Compass:

Examples

Availability is the most common SLIs. At the basic level it’s an answer to this question:

Is the service exposed to the consumer?

However, a better question can be:

Can the consumer use the service to do their tasks and achieve their goals?

The difference is subtle but important:

• The former is aligned with the service provider’s basic commitment and is usually easier to measure.

• The latter is aligned with the service consumer’s expectation and is typically harder to measure.

How can we measure availability for an API server?

1. Simple & Synthetic: ping the health endpoint every minute and calculate the percentage of successful pings in a period

2. Simple & Organic: count the number of minutes where the API endpoint could handle valid requests (i.e. with valid auth header) in a period

3. Complex & Synthetic: use synthetic monitoring to test that the critical user flow (e.g. a purchase journey) works and calculate the successful execution percentage

4. Complex & Organic: use the RUM (real-user monitoring) data to identify any broken user journeys and calculate successful user sessions

Let’s do this exercise for latency, which is the second most common SLI:

1. Simple & Synthetic: Measure the response time of a periodic GET request to a predesignated endpoint (i.e. www.example.com/api/health) and calculate the P99 percentile over a period.

2. Simple & Organic: Calculate the P99 percentile latency for all successful API requests from real users as measured at the API gateway.

3. Complex & Synthetic: periodically simulate a complete user transaction (e.g., login, search, add to cart, checkout) and measure the end-to-end response time.

4. Complex & Organic: Analyze Real User Monitoring (RUM) data to track the full-page load time, including all resource fetches and rendering, for critical user journeys.

Signal to noise ratio

Good SLIs are worthy of being on-call for because they indicate service degradation or disruption with high accuracy. Even if there is no 24/7 on-call rotation, this is still a good mindset to evaluate an SLI:

1. Does this SLI help us get notified when there is an incident?
   (incident in this context is any failure that threatens the business objective)

2. Does each anomaly in this SLI indicate an incident?
   (no false positive or false negative)

SLI is the basis for a solid ownership implementation:

• SLI measures the level of service from consumer’s perspective

• SLO helps:

  • The stakeholders set expectations on the service owner

  • The service owner communicate their commitment to service consumers

• Alerts: tie that expectation or commitment to responsibility and accountability

Poor SLIs lead to broken ownership.

Pro Tips: When and how to use the SLI compass?

Broadly speaking there are 3 categories of use cases for SLI compass:

1. Evaluation: Assess the signal quality of an SLI and its effectiveness in implementing full ownership through alerts.

2. Investment: Estimate the ROI (return on investment) of an existing or new SLI

3. Evolution: motivate improving how tech health is measured using SLI and optimized towards an SLO

For example, when Service Levels (as a concept) is new to an organization, you want to start small and easy. A simple synthetic check like pinging the home page every minute is a pretty good start.

In fact, this is so common that there’s a whole product sector doing exactly that: Pingdom, UptimeRobot, etc.

But pinging the home page has its limits. As you gain experience with that metric, you may decide to measure more high fidelity and organic metrics.

Evaluation & Investment

As a general rule of thumb: the closer to the top-left, the easier and cheaper it is to measure the SLI but the lower the quality it has.

The more you move to the bottom-right corner of the compass, the harder it is to measure and more costly it gets but the quality improves.

Evolution

Over time, you may want to carefully move your measurement from the top-left to the bottom-right corner considering the ROI.

You may go from simple synthetic to complex synthetic or simple organic. Eventually you may end up with complex organic SLIs.

Alternatively, maybe you’re in 4 but the SLI is too expensive for what it’s doing. Therefore you may deliberately go for a cheaper measurement with lower or equal quality.

Final Thoughts

The quality of your SLI depends on the story it tells. It must reflect the consumer's experience. This is crucial. An SRE's error budget must always be tied to a metric that accurately reflects user’s tolerance for failure.

The SLI Compass provides a clear path for evaluation, investment and evolution. You can use it to design and refine your SLIs. SLIs are not a one-size-fits-all metric. Make a conscious choice with signal to noise ratio and ROI in mind to reduce alerting false positives or false negatives.

My monetization strategy is to give away most content for free but hese posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

At the heart of the Service Level Assessment is a graph.

High level view

This graph uses the service level terminology we have already established to visualize the service consumption and measurement topology in 4 sections:

1. What offers the service?

2. What uses those service and why?

3. How do those usages fail and what are the symptoms, consequences and business impact of those failures?

4. What metrics can be used to measure those failures?

Subscribe now

Let’s go through them one by one.

Sections

1. Offer

At the top we have the service providers like backends, databases, hardware, etc.

Each service provider, provides one or more services:

2. Use

On the other side we have the service consumers like end users, or other teams that use our services to build other services toward their consumers:

Each consumer has 1 or more tasks. These tasks are very important because they are the reason that the service is exists:

As we’ll see, good SLIs tie to why a consumer consumes the service in the first place. Reliability and performance is perceived from the consumer’s perspective in terms of the tasks they care about.

3. Risks

Each task uses one or more services. There’s a risk that the service fails to do its job.

A usage establishes a link between a service and a task:

The number on a usage node indicates the number of failures that have been identified on a given usage.

These usages and failures are an important part of the service level assessment:

Each failures has:

• Symptom: how does the consumer notice the failure?

• Consequence: how the failure impacts the task?

• Business impact: how the business bottom line is impacted?

A good assessment takes quite some time to get right and we highly recommend to have the service consumer in the room to make sure that you understand their goals (tasks), what services they use (usage), and how they perceive failure.

Failures are measured by metrics.

4. Metrics

Metrics help the service provider be on top of their service level by measuring the failures that were identified for each usage in the previous step:

Metrics identify service degradation or disruption.

SLO communicates commitment to a specific level of service (target) over a specific time window (compliance period).

Alerts tie that commitment to accountability. That’s why it is very important for the metrics to measure failures that threaten the consumer tasks.

Metrics are a key outcome of doing the service level assessment.

Recap

Create your own graph using my open source and free project over here:

• Stable version (🛑not deployed as of early August 2025)

• Latest development version

My monetization strategy is to give away most content for free. These posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save money, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

💭 And I thought to myself: “What are you smoking? I want some of that too! It must be good! What do you mean SLA between teams? Unless one team is literally paying the salary of another team and can take them to court in case of a breach, we’re talking about SLO, not SLA”!

• SLA = Service Level Agreement

• SLO = Service Level Objective

To the untrained eye, the difference is very subtle: just one vowel! Many people, including self-proclaimed SREs use SLA and SLO interchangeably.

🧐 But once you know the difference, statements like the one I quoted becomes ridiculously funny.

When I’m hiring SREs, asking the difference between SLA and SLO is my favorite “trick question” because it reveals nuanced knowledge about how reliability is communicated.

This short post covers:

• Why does this distinction matter?

• An illustration to build a memorable base

• Quotes from Google’s books

This is not an entry article about SLA or SLO. We have covered those topics before:

As usual, there will be plenty of examples and illustrations to make the abstract topics more approachable.

Subscribe now

Why does this post exist?

I really didn’t want to write this post because Google SRE books have done a great job of explaining it. But for 2 reasons I have to:

1. If the point of the language is to communicate, it really pays back to use the correct word. Normally, I wouldn’t care —I’m not a word Nazi. But when building a user facing service, knowing the difference and relationship between SLA and SLO pays back enormously.

2. Asking “AI” and searching the internet gets you only so far. Atlassian, PagerDuty, and even Google have pages on the topic but I still haven’t found a resource that is crisp, illustrated, and to the point. I needed something that I could share with people without having to repeat myself again and again. This post exists to scale that ensuing “Aha” moment.

A visual guide

Sometimes SLI is brought up in comparison to SLA and SLO. We have covered each of them in separate articles.

In simple terms:

• SLI: Service Level Indicator → these are just metrics that show how the service consumer perceives reliability of our service (not to be confused with microservice or a backend, in Service Level model, a service refers to a capability or solution for a customer problem).

• SLS: Service Level Status → is the actual value of the indicator in a given time period. If SLI specifies the metric and formula, SLS puts the actual values in it to quantify the reliability and performance of the service.

• SLO: Service Level Objective → sets the targets for those metrics in a period of time (like 30 days). As long as SLS is above SLO, we’re good.

• SLA: Service Level Agreement → is a legal agreement that connects an external-facing SLI and SLO to financial or legal consequences. This is often the source of confusion and the main reason this article exists.

Here’s a visual representation using a gauge as a metaphor for SLI:

It can be a bit confusing because SLA has an SLI and SLO. The best way to think about SLA is as a legal layer on top of SLI and SLO.

What does Google say?

We get our first clue in Chapter 4 of Google SRE Book:

… [SLA is] an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain.

More explicitly, the difference is in the consequence:

The consequences are most easily recognized when they are financial—a rebate or a penalty.

To reduce ambiguity, the book offers a simple mental exercise:

An easy way to tell the difference between an SLO and an SLA is to ask, "what happens if the SLOs aren’t met?": if there is no explicit consequence, then you are almost certainly looking at an SLO.

And here is an example to solidify the difference:

Google Search is an example of an important service that doesn’t have an SLA for the public: we want everyone to use Search as fluidly and efficiently as possible, but we haven’t signed a contract with the whole world.

Even so, there are still consequences if Search isn’t available—unavailability results in a hit to our reputation, as well as a drop in advertising revenue.

Many other Google services, such as Google for Work, do have explicit SLAs with their users. Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service”

SLA/SLO in my experience

There are quite a few pages on the internet about SLA: Wikipedia, Amazon, IBM, Atlassian, SalesForce, ZenDesk, …

Apart from the fact that some of them parrot each other, the Google SRE book remains the undisputable reference. Even then, having helped hundreds of teams to set their SLO and consulted multiple SLAs, I’m opinionated about a few aspects:

1. SLA should be a written legal contract. Google’s definition gives implicit contracts a pass (e.g. if you say something in a meeting, email, or API documentation), but in my experience SLA is important enough to write a legally binding contract to remove any confusion or interpretation.

2. SLA should be explicit and external-facing. Google does a good job clarifying why SLA is external facing but it’s not overly prescriptive about what it actually means. To me, any entity that sells a service should have an SLA that is easy to find and understand. When you buy a pair of speakers online, you check the specifications to know what you’re getting for the money. The same is true for more complex SaaS: as a paying consumer, you have the right to know what level of reliability you’re buying, how much you can depend on it, and what happens if you don’t. Having an SLA is not just goodwill. It can protect the service provider from law suits due to misunderstanding and misalignment of expectations.

The confusion between SLA and SLO mainly stems from 2 anti-patterns:

1. SLO’s that are disconnected from accountability and change management. Some organizations which are at the beginning of their journey to adopt SLOs, only see them as a way to measure and optimize service level. This only utilizes half of their advantage.

   1. A good SLO is tied to accountability via alerting. If the service is degrading or disrupting, you want the service owners to be on top of it. Only when the SLO is connected to alerting, it can be taken seriously.

   2. The complement of SLO, Error budget, should also be connected to CI/CD to block non-urgent changes. If the service has burned its error budget, change (as the number 1 enemy of reliability) should be blocked in order to stick to the level of commitment made by the SLO. Only when the error budget is used to control risk, it can be taken seriously.

2. SLA’s are not seen as external facing legal documents. This comes from poor understanding of the legal side of the SLA. This is sometimes due to how other frameworks like ITIL use SLA loosely for service design.

When the SLO is connected to alerting and change management, it has enough consequences to guide the engineers own their service.

You don’t need to call it an SLA to distinguish serious SLOs from others. If it’s not serious, it shouldn’t be an objective in the first place.

The word “agreement” is primarily and exclusively reserved for a legal agreement based on my experience.

The SLA takes much more effort and uses a completely different bag of tricks to protect the service provider from the risks of the service consumer.

In a future article, we’ll unpack this bag of tricks for both when you’re offerring a SLA or when signing a SLA provided to you. What’s important is to understand that SLA is a completely different beast with different incentives, participants, and guarantees.

Subscribe now

Similarities and differences between SLA vs SLO

SLA and SLO are related but different:

Accountability

Previously we’ve covered the difference between accountability and responsibility.

TLDR; it boils down to legal consequences and punishment.

• Similarity: Both SLA and SLO communicate measurable service level commitments towards the service consumers.

• SLA: the commitment is typically tied to some financial penalties and entitles the consumer to compensation or legal action in case of a breach.

• SLO: is softer and the consequence of burning the error budget is typically implemented in annoyances like not being able to push new changes to production (change is the number one enemy of reliability: any time you change a system, its likelihood of failure spikes up).

This difference is small but important because as we discussed in the article about SLA, it boils down to service consumers having leverage or not.

SLO Target

They both set expectations that are set through research and negotiation.

An SLO that is in the SLA promises less reliability than internal SLO because our external commitment should always be lower than what we internally aim for.

For example, if internally we aim for a SLO of 99.99%, the SLA we commit externally may be 99.5% (roughly buying us twice the error budget).

Under promise and overdeliver, not the other way around.

Subscribe now

Regardless of what we aim for, it is always advised to think about the cost of reliability and balance that against the business profit (revenue - expenses) for a given service.

This is the core premise of the 10x/9 rule which elaborates “those 9’s aren’t free!” 😃

Audience

• Similarity: they both set expectations between a service provider and service consumer.

• SLA: the consumer is often an external entity with leverage.

• SLO: the consumer is often internal (e.g., other teams in the same organization).

Level of detail

• Similarity: They both tie an indicator (metric) to an objective (e.g. 99.9%) but are detailed in different ways.

• External SLIs that are tied to SLA are usually simpler (availability is the most common one). However, SLA has extra details about the legal aspect (consequences, compensations, exceptions, credit, support mechanisms) and typically contains a glossary to build a common language with external consumers. If the service provider is a large vendor, it may get away with having a cookie cutter SLA for all consumers (see Google Cloud SLA for an example).

• Internal SLIs that are only tied to SLOs are more detailed because their primary target are the engineering teams which have a good knowledge about how the service is implemented and consumed. This leads to more nuanced and detailed SLIs that focus on what the team controls. In contrast, external SLIs abstract away the shape of the organization and implementation details.

Monitoring

• Similarity: They are both monitored and tied to alerting.

• The SLS of an SLA: may also be tied to a publicly visible status page to help the consumers troubleshoot their system by quickly assessing the reliability of their dependencies. When things fail, the consumers should have an effortless way to answer: “is it me or is it them?” For example, the Azure Status page helps their customers to roll out cloud disturbances in case they have an incident. Same with Github, Slack, Google and a whole bunch of others. There are tons of products in this area, and you can even set one up using open-source products like Cachet.

• The SLS of an internal SLO typically misses that public-facing monitoring and is often implemented as an internal dashboard on the observability platform. It requires less effort.

Purpose

• Similarity: They are both used to set expectations.

• SLA is written in a way that protects the service providers’ interest. That is because when the service is disrupted or degraded, the service provider already looks bad. The last thing it needs is to add insult to the injury and put money on the table.

• Internal SLO is primarily used for giving a focus to service optimization.

… if you can’t ever win a conversation about priorities by quoting a particular SLO, it’s probably not worth having that SLO —Google SRE book Chapter 4

Revision frequency

• Similarity: both are subjected to revision and readjustments as the tech evolves and/or the consumer demands changes.

• SLA: is revised less often and is more ceremonial because it involves business stakeholders, legal, and representatives from different parties.

• Internal SLO: can be revised every few months, sometimes weeks. Since its primary consumers are the internal engineers, it may go through rapid evolution to become what makes sense to them. Moreover, the internal stakeholders typically enjoy more flexibility than an external stakeholder to get what they want in a timely manner.

My monetization strategy is to give away most content for free. These posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save money, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

1. What is a team?

2. How large is too large and how small is too small?

3. What qualifies a missions?

4. How to weigh the cons and pros of specialist vs generalist in the age of GenAI?

There’s a story for several years back that keeps back to my mind. What makes it interesting is the fact that there was no master plan. Yet with a few cultural elements, the story took such an interesting trajectory that it shaped my leadership model. Ever since, I have been an advocate of continuous improvement by preparing the environment instead of being the wise-ass who has the ultimate solution to all problems.

Today I want to tell that story with these key takeaways:

• A generalist team is less likely to have internal dependencies acting as bottlenecks, blockage, and single point of failure. Generalist teams are more productive and resilient.

• Handovers introduce the risk of misunderstanding, delay, miscommunication, and diluted accountability. Clear ownership requires knowledge, mandate, and responsibility. It’s easier to implement with generalists.

But as we’ll see generalism has its own challenges as well.

Subscribe now

🤖🚫 Note: No generative AI was used to create this content (except one image that’s clearly marked). This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs. (why?)

Story

Many yeas ago I was part of a large team. 14 members to be exact.

The problems started from the first standup: I didn't follow 90% of what everyone else was talking about. I thought it's because I'm new to the team but as time passed and I got closer to the other team members I learned that some others feel the same.

It wasn't unusual to see someone visibly yawn or fall almost sleep during the standup. There was at least something in the round of standup that was completely irrelevant and boring to a few.

In an effort to keep the standup short, we had a time limit per person paired with invented hand gestures to signal the person who is talking that their time is over.

We were 11 engineers which can still be fed on 2-pizzas! Size wasn't the issue.

In fact, quite the opposite: every now and then we heard about a task that never came up during the standup and wasn’t in the backlog!

We learned about those phantom tasks as they were close to the finish line or after they ere delivered.

How did that happen? Usually, someone from another team or another part of the organization reached out for a friendly favor and since we had enough bandwidth and great engineers, they often did their best to help.

Async Standup

Later we switch to async standup. The implementation was simple: every workday a Slack reminder would pop up with a fixed message:

1. What did you do yesterday?

2. What will you do today?

3. Are there any impediments or do you need any help?

Each team member would write a comment more or less in line with what they would be telling at a sync (face to face) standup.

Those comments came in during the day, and some folks consistently missed it. Regardless, I rarely read all the comments because they were not relevant to what I was doing.

This was less disruptive than taking 30 minutes (less than 3 minutes per person) for the daily standup, which often dragged to 45 minutes and sometimes even an entire hour!

The async report turned to exactly that: a report.

The main value of a standup is to have a dialogue about blockage and spark opportunities to work together. Otherwise, it's enough to move issues on the Kanban swim lanes with a comment. That covers the reporting part. We missed a dialogue when a dialogue was due.

Concern-based Task Forces

I raised these issue in one of our retrospectives and the EM/PM said they'll take action.

The solution they came up with was to break the team into two halves:

• The front-end "task force" (as it was called) would work on front-end stuff basically taking a Figma design and turning it to working UI.

• The back-end task force would cover the rest from API, to CI/CD, and configuring cloud and operations.

The idea was to have smaller groups that would move faster, delivery independently, and are more relevant and connected to each other.

And we cut the team by tech, because anyone who has worked with engineers knows that they can be quite religious about their tools of choice. Back-end/Front-end seemed to be different enough to justify that cut.

The hypothesis was that front-end developers should care enough about each other's world and standup report. Similarly back-end task force should have some sort of interest and common language to hear each other and collaborate.

As a bonus, now that the task forces were smaller, they could go back to the sync stand up and unlock those dialogues.

This sounded reasonable for exactly 5 working hours! On the next standup it was evident that there are dependencies between the two task forces. Who would have thought? 😄

Regardless, we gave it a go for a couple of sprints until the feedback from the next retrospective came in.

Fluid Task Forces

The next iteration was for people from one task force to attend the standup of the other when they feel it is relevant or they have an agenda to share (e.g. if a front-end person needs to work with the backend person to change the API for a new feature).

The two stand ups where lined up one after the other to facilitate this.

As you can imagine, some of us consistently attended both standups while others consistently took tasks that were in one of the front/back-end bucket.

So the total time spent in standup actually increased while some team members were isolated from each other by the nature of their tasks which was a function of their technical specialty. e.g., one person only worked with our in-house CSS-framework and barely had to talk to the back-end task force.

We were a Swedish team. Efficiency, pragmatism, and continuous improvement was embedded into our culture. I don't say this to brag, but to elaborate why we were in constant endeavor to optimize our work and how leadership supported these experiments.

Individually we were not smart, but together we found a way that was generally more sensible and holistic.

Other solutions

Before discussing the final solution we landed on, I just want to quickly iterate a few other experiments we did:

• Break the team in two halves with a shared EM/PM: The API served as a clear abstraction between front-end and backend. But the operation (e.g. CI/CD and even on-call) primarily fell on the back-end team. Unfair.

• Create delivery focused ephemeral feature teams: For example, implementing GDPR in the front-end and back-end, required a couple of people from each expertise. Same with the user-tracking feature. These ephemeral feature teams would emerge as needed and collapsed when the task was done. The problem was that it was hard to get the right resources when everyone was locked in some feature team. Maintaining the feature afterwards would unofficially steal resources that were now busy with other tasks.

• Hire consultants to cover temporary spikes in bandwidth demand: This was a bit ridiculous because the team was already quite big and it was hard to justify why we need consultants to add extra bandwidth. Upper management pushed back to utilize the available bandwidth in a smarter manner.

By this time, we’re more than a year into these struggles and while some have accepted that this is just the nature of having such a large team, others are still pushing new ideas every retro!

Meanwhile, the company ran out of money to burn (we covered that story before), and as Sundar Pichai says:

Scarcity breeds clarity

Subscribe now

The solution: generalists

What worked for us eventually was to remove our specialist hats and be generalists:

Front-end, back-end, QA, DevOps, etc. are all different concerns for the same outcome and impact.

A recent article, dug into why breaking a role into multiple titles is a bad idea that hurts the business (more bottlenecks) and individuals (turning to cogs that only fit special machines).

Sure, learning CSS is a challenge for someone who has spent their entire career behind an API.

Similarly doing cloud configuration had a steep learning curve for someone who makes money moving bit in the browser sandbox.

It is unrealistic to expect everyone to turn specialist over night. The cognitive bandwidth is the bottleneck. It takes time to lean different disciplines especially when you’re new and need to rethink and unlearn.

But with the advent of copilots (with small "c"), it's already easier than ever to build an all-around skillset.

We didn’t have LLMs back then but we had another tool that was even more effective: mob programming! We were fortunate enough to get a department-wide workshop by Woody Zuill who is a recognized force behind popularizing the idea of working together on the same problem.

A huge side effect of mob programming is the knowledge sharing which unleashes the generalists in all of us.

Having worked with both AI-powered copilots and Mob Programming, I still think the latter is more powerful because:

• It provides higher communication bandwidth, especially when done onsite

• Its self-correction mechanisms of team dynamic prevents problems like hallucination or context limit. Team members are more likely to call out bs than LLM and together we have better collective memory about how the product works.

• The current generation of AI-powered copilots have no problem with adding complexity. “Given enough eyeballs all bugs are shallow”. Mob programming tends to produce outcomes that are approachable to the majority of the team members not the elite ninjas. It leads to simpler solutions that are more maintainable.

Specialist vs Generalist

Good engineers are good problem solvers not a layer of meat wrapped around a tool!

An easy giveaway is when people introduce themselves as if their identity is attached to a tool:

• I’m a Java programmer

• I’m a C# developer

Ownership is at the heart of the generalist (multi-skill) vs specialist (mono-skill) discussion.

Specialists’ output is only part of the outcome. For example, let’s say one task requires 3 specialists to work together:

In practice, there will be extra communication load at the point of handover. But let’s ignore that because there is a larger waste at play: fake work.

You see, those specialists aren’t going to just sit and wait. They are very likely to create complexity and make their tasks big enough to be worth a full-time title. I’m not talking about adding tests or fixing bugs. I’m talking about bike shedding over linting rules, creating automation for one-off tasks, unnecessary migration and refactoring.

Something like this:

You can of course throw more work at them. Here, the same 3 specialists work on 2 different tasks in parallel:

Multi-tasking is hard. Even a CPU has a hard time switching contexts every 200ms to create the illusion of parallel processing. Humans are notorious when it comes to joggling between tasks.

Wouldn’t it be ideal if one single person could do everything? Here you see 3 generalists executing one task each with no handover:

And here, 3 three of them works on one single task at a time in a setup like mob programming:

The benefit here is that all 3 generalists get exposed to all 3 tasks and are more likely to fix any issues in the future. Plus, their collective experience is dedicated to each task, which improves quality.

Why did it work?

As time showed, turning the entire team to generalists wasn't as hard as we thought because:

1. Shared context and outcome: Everyone knew what the product is about. We were not learning a new discipline from abstract and in isolation. We were just learning different parts of the solution to the same problem. That context allowed us to bring what we already know, and curiously learn what we don't. Once those specialist titles were removed, there was little getting in the way of curiosity and exploration.

2. Narrow needs: We didn't have to learn everything there is in a particular discipline. For example, the front-end developers didn't have to learn all types of CI/CD products to be productive. We had Travis, and as long as we could endure a knowledge-sharing session by a team mate and read though Travis documentation were productive enough not to depend on someone else for help with CI/CD, it was good. In the worse case scenario, we could ask for help from the people with more experience in the team, but that wasn't the default approach when facing with problems. We had access, knowledge, and enough mandate to unblock ourselves as we learn more.

3. Motivation: In Drive, Daniel Pink mentioned autonomy, mastery and purpose as the key elements of keeping the team engaged. While the specialist approach enabled mastery, we reached all 3 with the generalist model. This is the crux of the role vs title and outcome vs output argument.

4. Swedish work culture is egalitarian where everyone is treated equally but is also expected to pull their weight to the best of their ability. Gone were the days where a front-end developer had to beg for admin access to the cloud provider. In the new model, everyone had enough access to own their delivery. But it also meant that the whole team needed to be lifted to understand the implications of their elevated access. This created a lot of collaboration and knowledge sharing opportunities that personally catapulted my career. I learned more from my team mates about how to maintain that specific product than I learned in 2 years of my master's degree on different shallow assignments.

5. Better ownership: In “you build it, you own it” we distinguished 3 elements for full ownership:

   1. Knowledge: you know the problem and solution space

   2. Mandate: you have the power to make changes without having to ask permission or rely on governance and baby sitters

   3. Responsibility: when sh*t hits the fan because of a decision you took, you have to deal with the clean up

   

The new model, established clear ownership which led to, faster delivery, higher quality, and overall cheaper operation.

Was it some genius master plan? I don’t think so. The money ran out and scarcity breeds clarity. The only thing we did right is to have open dialogues, experiment, learn from those experiments and continuously optimize our workflows.

Did the company go big and use this success as the new “Spotify Model”?

Hell no! 😄

There are no universal best practices, only fit practices. You should see what works for your setup which is heavily influenced by the type of product you’re building, the talent bar you have available, the available budget and tons of other factors.

I’m very careful not to prescribe what happened to us as some global truth that solves all problems. In fact, let’s dig some side effects of this way of working.

Side effects

All medicines have side effects! This solution is no exception. Here are some of the unexpected side effects after we executed the solution for 12 months:

1. Some specialists quit. Partly because venturing to the "other side" didn't make too much sense for them career-wise. Surprisingly, we could cope without having to fill those vacancies. My theory is that the specialist jobs don't take up 8 hours every single day. This creates a good amount of slack to learn new things or rest while getting paid! The generalist model maximized resource utilization (sorry for calling people "resources" but I'm borrowing this term from systems engineering hoping it makes sense in the next item). High utilization made some resources too "hot" to do it in a sustainable manner. Becoming generalist is not everyone's cup of tea, as we learned. This was way before the GenAI era. GenAI can help both in learning new concepts and help fix problems. Even with GenAI, this route can easily lead to fatigue without proper control of the intake. Fortunately we had 30 days of vacation (de facto standard in Sweden) and a dedicated development time (a percentage of the paid hours that can be spent to sharpen the axe), but still the risk of burnout was very real. Personally I had to use some of my vacation days to step back, reflect, and implement personal strategies to make the workload sustainable for me. But there’s nothing in this solution that promotes that. In fact, it is set up for maximum productivity and efficiency.

2. We were not as deep as we wanted to be. A front-end developer works with 3 languages on a given day: JavaScript, CSS, HTML. Usually, one uses other languages that compile to those languages: TypeScript, SASS, JSX/TSX. Then there are test frameworks, SSR (server side rendering), icon libraries, fonts, etc. This concern is large enough to keep someone busy for years. Add the CI/CD, Kubernetes, Istio, and on-call duty on top of that and you get an exhausting amount of work. Sure, the resource utilization was high and each individual could get a lot done, but it also meant more context switching and less chance to get deep into the toolbox.

I guess 1 & 2 are two sides of the same coin. But given the accelerated growth, better utilization, and overall autonomy, I would pick this solution again if I was in a similar situation. YMMV.

Do you have experience with a similar setup? What would you add to this post to make it more useful for others who may come across?

Recap

The broken standup was merely a symptom. The root cause was lack of interest and understanding.

The team of specialists didn't have a common language and shared goal so it was hard to do a standup that was relevant to everybody.

The generalist team on the other hand had common language and interest to fill the knowledge gap.

Regardless of what works for you, a key element of the culture was the willingness to experiment and continuously improve. That part is non-negotiable, even though your end state may differ from the story in this article.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

The simplest way to support me is to like, subscribe and share this post.

If you really want to support me, you can consider a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. You can get 20% off via this link. Your contribution, also funds my open source products like Service Level Calculator.

You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

As usual, there are amble examples and illustrations.

Subscribe now

🤖🚫 Note: No generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs. (why?)

3T’s of optimization

Optimization is the intentional process of changing systems (software, hardware, people, etc.) to improve one or more aspects. Typically, it involves changes and trade-offs.

Good optimization improves the right things, at the right time, and with reasonable trade-offs.

However, change introduces risk (delay, failure, cost, etc.).

Premature optimization is the root of all evil —Donald Knuth

Premature optimization is when at least one of the following is true:

1. Changing the wrong thing. Examples:

   1. The team takes 6 months to rewrite a Java micro-server to Rust to improve response time. After the rollout, they learn that the biggest source of delay was cross-region network dependencies.

   2. A non technical manager falls in love with DORA metrics. His team sets up dashboards to track all 4 but she’s particularly focused on Lead Time. She uses carrots and sticks to get the team review PRs faster. The team games the system by making lots of pointless PRs to skew the data (e.g. updating inline code comments).

2. Picking the wrong time. Examples:

   1. A nerdy startup founder burns the budget to create Google-level infrastructure for its one-endpoint API. The startup fails before discovering product market fit. First make it work, then make it better.

   2. As the market-size was shrinking, a company realized that they need to rearrange their [human] resources to reduce waste and keep the balance sheets clean. So they decide to do a reorg. But every reorg has a J-curve (as we’ll discuss in this article). The productivity takes a hit before getting better. Being too impatient, leadership panicked at the sign of the J-curve and went with the second best option on their table: layoff!

3. Choosing the wrong trade-offs. Examples:

   1. Leadership decides to switch observability provider to save cost. After 3 month of migration (the hidden cost), the team learns that the new provider has poor data quality and the support is crap! You get what you pay for.

   2. Developers of a game engine decide to refactor the critical parts of the code in Assembly. This resulted in a 20% performance boost over the C++ code but now the company has to increase the price 2x to cover for the extra cost of development to support multi-platform and fix quirky bugs.

Now you have the the 3T’s of optimization:

This simple mental model, enables you to quickly assess whether an optimization is mature or not.

Subscribe now

Ask the 3T’s:

1. Thing

• Are we optimizing the right thing?

• Do we have data that supports it?

• Have we done the due diligence to avoid data bias?

• Is it a fact, or assumption, or belief (FAB)?

2. Time

• Is this the right time to invest in optimization? Is it too soon/late?

• Do we have a clear picture about the problem and various alternative solutions? Or should we wait for the problem space to evolve and various alternative solutions to mature?

• What’s the time span till the investment pays off?

3. Trade-offs

• Engineering is the art of trade-offs. Are we aware of the metrics that will be hurt by the optimization?

• What is the list of things that will surprise us when the optimization is done?

• Do we have contingency plan in place to reduce those risks?

• Are the consequences objectively worth the effort?

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

The simplest way to support me is to like, subscribe and share this post.

If you really want to support me, you can consider a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. You can get 20% off via this link.

You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

Pro-Tips: Mature optimization

Now that we have a mental model to detect premature optimization, let’s discuss a few basic rules to do the right thing (effectiveness) and do it right (efficiency).

The pro tips expands on the following points with illustrations:

1. Avoiding optimization

2. Finding the right metric to optimize

3. Focusing on the subset that matters

4. The importance of control metrics

5. The value of experimentation before going big bang

6. Identifying the point of diminishing return to stop

7. Defining concern priority list

8. Picking the right time

And we end with an example of mature optimization in a Python library that was used at the core of a 3D processing pipeline.

Read more

My focus has been the teams at the leaf nodes of the organization tree:

That is just the surface area for improving reliability. Knowing that your tech isn’t good is not going to help if the leadership is not on board with investments to improve it.

Moreover, the team dependencies across the org create gray areas where there isn’t clear ownership over SLO breaches.

To fix some hairy problems, we had to bring leadership onboard and work in the depth of the organization tree.

This post is about a few specific and common issues at the intersection of organization, technology, ownership, and reliability.

🤖🚫 Note: No generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.

Subscribe now

Not just the leaves

Before we talk about how we hold the leadership accountable for technical health, let’s review some of the problems that came my way. I’ll be paraphrasing and removing any company-specific names:

“Why can’t engineers do their job right so the bugs don’t eat up the team bandwidth?” —A product manager 👻

✅ My take: reliability should be treated as a feature and prioritized against other features in the backlog. The rule of 10x/9 states that reliability has a cost. Set up a payment plan for tech debt if that’s the issue.

“We know our service reliability is not good, but we’re not successful in convincing the upper management to give us the time, budget, and headcount to pay the price of improving reliability” —A team EM 😭

✅ My take: Leadership can afford not to care because it’s not their problem. We need to make reliability everyone’s concern, particularly the upper management who formally have control over budgeting, hiring, deadlines and other factors that directly impact system health. The key is to show them the gauge and let them drive.

“Leadership uses our service level status (SLS) against us. They perceive failure as a symptom of us engineers not doing our job properly.” —An engineer 🤬

✅ My take: you should be asking what does the leadership provide to support engineers? Nothing is free, especially reliability.

This part is tricky because as an engineer you may not be empowered to ask those questions. I on the other hand, as a Staff Engineer have a responsibility to take this feedback and act on it.

“We have an immature engineering culture. We need to show what good looks like and rank the teams against maturity models” —Ivory Tower Architect 🤡

✅ My take: Oh my! Please don’t go there. 😒 If you think tech is sh*t because people don’t know any better, you don’t understand engineering.

It is naïve to think that emitting a manifesto or chasing people or EMs will lead to meaningful change. It’s high visibility work but it doesn’t really make an impact. “Preening” as Will Larsson puts it.

Besides, don’t measure people, measure the tech. As an IC, please stick to your lanes and don’t interfere with engineering leadership accountability.

“My organization doesn’t have incidents because I haven’t seen any” — A middle manager 😎

✅ My take: just because you don’t see it, doesn’t mean it doesn’t exist. Your engineers have learned to hide failure because they’ve been burnt before (or the tech health is fantastic! But that wasn’t the case here 😄).

“This data is good. Too bad we have to wait a month. Can we do it weekly?” —Upper management after attending a monthly incident report meeting ⏳

✅ My take: lagging metrics lead to even more lagging actions. You need to see that data in near real-time.

Please don’t use meetings to review data and dashboards! If there’s a specific need that requires the higher communication bandwidth of a meeting, go for it. Less meetings, more focused work.

Ask for real-time dashboards and if that’s too complex for you, please do everyone a favor and switch jobs.

“We have too many high impact incidents. Customers are unhappy. We’re losing profit. Seems like department leads screwed up! Mass layoff or at least some firing at the top level is in order”. —C-level 🚨

✅ My take: If your arm has a wound, you can cut it for sure. That’s one way to solve the problem. Replacing leadership is another solution. Just beware that new people come with new problems and delayed solutions!

Here’s a crazy idea: maybe you are the problem. What’s your reliability budget to pair with the technical error budget?

“Part of my job is to pick up the phone when things are broken and route it to the right team. The other part is to run the monthly incident reports so they can take actions.” —Consultant hired for operational work 😏

✅ My take: you do realize that’s not an actual job, right?

The only reason you exist is because there’s a gap between diverse data sources and the need to have a unified view over that data. Filling that gap with meat (manual effort) is one option, but the better option is to build a lightweight tool that streamlines your effort.

That way you get to do actual work instead of reading PowerPoint slides showing dashboard snapshots!

“If we implement best practices like TDD, DORA, or SLI/SLO, the state of tech will improve.” — Staff Engineer 💭

✅ My take: Best practices are often someone’s interpretation of how something worked at a certain time and environment.

If you want to find a solution that fits, there’s no way around rolling up your sleeves and working with the teams to understand the problem and all the forces at play.

Develop a hypothesis and methodically diffuse the bomb instead of just throwing your baggage from previous company or last blog post of conference at your teams. Avoid cargo culting.

Many of these problems stems from distributed accountability and broken ownership.

And that’s what this post is about. We discuss a hypothesis and experiment we’ve run to close the gap between accountability and responsibility to reliability and budgeting!

Our hypothesis

At a high level we got:

• Engineers who build and maintain pieces of the holistic technical solution

• Engineering leaders who control the budget, headcount, and hiring/firing

• Product leaders who want a solution that just works

Reliability is everyone’s concern, but the responsibility is not distributed evenly.

Our hypothesis is that by building a tool that maps reliability to accountability we can:

1. Clarify this responsibility across the org

2. Prevent things from falling between the [organizational] cracks due to ambiguous ownership

3. Provide high-quality reliability data to replace guess work and gut feeling. Good data and insights lead to informed decisions.

Mapping a graph to a tree!

At a high level, we have two elements:

• Reliability: measured in the system architecture which is a graph

• Accountability: exists in the organization which is a tree

We need to map reliability to accountability:

Essentially, it’s a computer science problem to map a graph to a tree! 😄

Constraints

How do you do that in a way that solves the problems above without causing new problems? Let’s set some constraints for any solution hypothesis:

• Accountability: Hold leadership accountable for the reliability of the systems that they formally own. The hypothesis is that if we make service reliability a problem for the leadership, they’re going to do everything in their power to improve it.

• Control: Everyone should be accountable for what they control, not more. To prevent things from falling between the [organizational] cracks, we need to bubble up the right number to each level of the org chart (see that article for an example with diagrams).

• Clarity: I got my MSc in systems engineering and am aware of the formulas to calculate the reliability of composite systems. We need to find a way to measure metrics that are easy to reason about and bring clarity. The hypothesis is that clarity improves the quality of data-driven metrics.

• Drill-down: Several times in my career, I’ve seen leadership panicking when presented with data and take rushed decisions. Data can lie not just because of how it was collected, but because depending on perspective it can tell different stories. If the goal is to present data to the roles which are detached from the system implementation details, they need to be able to drill-down the data to make informed decisions.

• Real time: Instead of waiting for a weekly or monthly cadence to get up to date data, we should aim to update the dashboards in real time or near real-time cadence. The hypothesis is that lagging metrics lead to lagging actions, so we should strive to provide an up-to-date view of tech health at any time.

• Efficiency: Everyone wants more reliability, but not everyone is ready to pay the price. There is a point of diminishing return where spending more on tech doesn’t yield more revenue and instead hurts the profit. The hypothesis is that by normalizing failure and attaching service level status to organization nodes, leadership is encouraged to tune their expectation to “lagom” SLO.

The Ownership Dashboard

To test our hypothesis, we built a Single Pane of Glass (SPOF) that has two parts:

1. The org chart

2. The stats

For compliance reasons I cannot share a screenshot from what we’ve built but here’s the wireframe I made for my team to implement:

Since our org is slightly large, we made it collapsible. When an org node (team, cluster, etc.) is clicked the stats update to show aggregated tech health for that node.

For the stats, we implemented something that’s easy to start with:

• Incidents: how many incidents of each type occurred in the given period? We later changed the incident count with incident length because it maps better to the incident impact.

• Service Level Status: how is that organization node performing compared to its commitments (SLO). We actually started this initiative to show SLOs but due to poor tooling we decided to experiment with the data we had (incidents) and expand if there’s value.

We decided to start simple, roll it out across a sub-part of the organization, gather feedback and data (like page views, engagement, etc.) and interview the first users of the page to evolve it.

In the future, we’re planning to add these metrics:

• Runtime cost: what’s the total cost of tech runtime (infrastructure, 3rd party tooling, etc.)? This metric is hard to calculate correctly and there are 3rd parties specializing in that problem, but a rough estimate goes a long way to motivate discussions about the cost of reliability.

• Security: what’s the state of security checks and automatic analysis? What’s the SAST/DAST (Static/Dynamic Application Security Test) status? SAST is white-box testing that looks for vulnerabilities inside the application and code, while DAST is black-box testing that looks for vulnerabilities that could allow an outside attacker to get in). While SAST allows for early detection of vulnerabilities in code, DAST offers a practical assessment of how an application behaves under attack once it’s live. Both can be automated or manual and can be reported as multi-dimensional metrics.

• DORA/SPACE/DevEx: there are a bunch of other metrics that can be put on the dashboard. Please see the caveats and read the fine print carefully because this tool is very powerful and when abused it can cause massive destruction at scale.

• Synthetic test success: synthetic tests are scripted user flows that test the success of critical consumer journeys. We use Playwright but many Observability vendors have this feature namely Datadog, Grafana, and NewRelic)

• Team bandwidth: we could show the headcount to flag if a team is too small or too big for the problem they’re solving and/or the reliability expectations (SLO).

• Metric decomposition: Every metric aggregates multiple variables. Those variables can individually be optimized. For example, time to resolve an incident (TTR) is composed of TTI (time to identify the incident), TTA (time to acknowledge the incident), and so on. Same can be said for cost: it’s useful to break down the cost into individual components.

Based on the user feedback, we also learned that it’s very helpful to allow the organization nodes to add some context through links and free-form text. Dashboard and data tell part of the story. With this addition, the teams can tell a richer story and put those data points into context.

Caveats

• Product and investment: most improvements require investment. In many teams product roles (Product Managers, Project Managers, or Product Owners) control the team’s bandwidth. Ideally this dashboard should motivates taking action and deliberate investment to improve the status quo (e.g, payment plan for tech debt)

• Evolving metrics: One-size-fit-all metrics do not guarantee usefulness of the same metric for a diverse range of products. That’s one reason I sit with every single team to figure out what SLI makes sense for their service from the perspective of their consumer. Ideally the metrics should evolve over time. SLI/SLO need to be revisited regularly as the product, tech, and our understanding of the service evolves.

• Benchmarking: putting the stats of teams next to each other creates nice dashboards but there’s also a huge risk to benchmark teams against each other. I’ve seen these SPOG (single pane of glass) initiatives being weaponized by non-technical leaders to blame the teams. That’s why it’s extremely important to provide the ability to drill down and at some point you may even link to the data source in whatever system where the data is coming from.

• Illusion of knowledge: Metrics by nature abstract the phenomenon they're representing. They're a great way to simplify what is inherently complex but are also very attractive for incompetent managers and have a high risk of being abused. Metrics are not a replacement for talking to engineers. If you’re the kind of leader who needs a translator to make sense of tech, you’ll in the risk group to abuse such dashboards. It’s up to higher level leadership to spot these cases and guide the conversation to be about tech, not people.

• Data quality: in my experience with this dashboard and many similar initiatives, data quality is a critical factor. On paper, the data is great but once you roll it out, people call out the nuances and inaccuracies. It’s good to pilot the data before GA (general availability). I’d argue that the data quality is more important than the UI and dashboard.

• NOT developer productivity: I've seen many developer productivity measurement efforts in my 25 years and every single time (no exceptions), it’s coming from managers who doesn't understand what they're responsible for. They resort to replace the need to understand the tech with some metric. It's more of an incompetence symptom than anything else. I've had my share of trying to convince these "leaders" to dig deep instead of stopping at vanity metrics. Common treats:

  • Calling teams "imature"

  • Shallow understanding of tech and engineering in general

  • Unrealistic expectations from SDLC (software development lifecycle)

  • Picking favorites in team and relying on them to acting as "tech translator"

  • Heavier on politics and "networking" than transparency and doing what's right for the company.

• Novel ideas are not measurable: metrics by definition need to be defined in advance. They work best for repeatable events but fall short of indicating novel innovations that are outside the scope of what is measured. For example, when the SLI is about latency we may focus too much on improving it that we starve more important questions like: should we decommission this service? Fortunately, when it comes to reliability metrics, we are often intentionally trying to improve predictability. But I need to highlight the fact that not everything worth doing is measurable and not everything that’s measured worth optimizing.

• Leadership metrics: most of my efforts have been focused on helping teams define their service, measure the service level (SLI) and set reasonable expectations (SLO). It is relatively easy to take a team of engineers to a 2-hour workshop. For top-level leadership, I’ve encountered many obstacles. Regardless, for this idea to work, we need to follow the same methodology and find metrics that make sense for each part of the organization.

  

• Implementation: Even if we know the metric, it is not easy to expect engineering leaders to do the implementation. A team of engineers can be tasked to measure their SLS and even be on-call for breaching their SLO. For leadership, the implementation can be delegated to their team, but the on-call aspect is still tricky. Having Staff Engineers on-call is one option since they are accountable for the health of their organization. Another option is to improve the organization architecture to have one team responsible for the user facing parts (i.e., the cake model).

• Prevent weaponization: There’s a risk for non-technical leadership to weaponize these metrics and bash their own teams. How can we prevent that by clarifying the accountability and responsibility in a way that drives productive actions instead of unproductive politics and blame game? They hypothesis is that instead of showing individual names, the dashboard UI is designed to focus on the tech and instead of seeing numbers in insolation, they are put into the perspective of how they bubble up in the organization. This is by far the hardest aspect to get right.

• Gamification: Once a metric becomes a goal, it stops being a good measure (Goodhart’s law). If people perceive their salaries, bonuses, and performance reviews as being tied to those metrics, they’ll game the metrics. A good metric is one that if it’s gamed, it leads to all parties walking out in better shape. This is not something we address in the dashboard but rather in service level workshops. Finding good metrics is half the battle to gather meaningful data. One hypothesis is that showing end to end metrics that span across the boundaries of multiple teams motivates collaboration between all the teams that contribute to those metrics.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

It depends! 🤓

Just joking. 😄There are many similarities but also some important nuances that this article digs into.

Does it really matter to distinguish between the two? Well, that one actually depends!!! If the point of the language is to communicate ideas, it pays dividends to use the right word.

Both SLI and KPI got popularized by Google although they don’t originate there. SLI and KPI are metrics where the “I” stands for indicator. Both metrics aim towards optimizing an objective and support data-driven decisions.

But that’s where the similarities end. There are fundamental differences between the two that stem from their purpose and expand to their scope, concern, audience, actions, and even formula.

What’s interesting is that KPI/OKR and SLI/SLO are two systems that can be converted to each other with many nuances.

Usage

Key Performance Indicators enjoy a large usage across business, sales, marketing, product, finance, and beyond.

KPIs cover various areas, including financial metrics (e.g. revenue per customer), product metrics (e.g. monthly active users), customer satisfaction, employee productivity, operational efficiency (e.g. number of goods produced without defects), culture (e.g. OfficeVibe scores), etc.

The target of KPIs can be to optimize systems, organizations, products, business, etc.

Service Level Indicator on the other hand is primarily concerned with reliability engineering and has seen an uptake due to the popularity of SRE (site reliability engineering) in recent years.

SLI covers various services metrics like latency, error rate, throughput, data consistency, cache hit ratio, etc.

In my experience KPIs are more high level and wishy washy leading to memes like this:

SLI on the other hand typically enjoys a more formal definition and scoped usage (reliability engineering).

Normalization

KPI does not need to be normalized. The metric datapoints can be any number. For example, the monthly active users (MAU).

SLI doesn’t need to be normalized either. But it usually is:

This normalization makes it much easier to assess the Service Level Status (SLS) against the Service Level Objective (SLO) which are both values between 0 to 100.

Objective

KPI is often used in combination with OKR (objectives and key results). For example:

• OKR

  • Objective: Increase Monthly Active Users (MAU)

  • Key results within the next quarter

    • Increase new user sign-ups by 20%

    • Improve user retention rate from 60% to 75%

    • Reduce user churn rate by 15%

• KPI

  • User sign-ups

  • User retention rate

  • User churn rate

One important point to call out is that OKRs are often ambitious. They shoot for the stars, aim for the moon. Even if you miss an OKR, you may still celebrate an improvement. 🎉

This is in stark contrast with how SLOs work. The mantra for SLO is to under promise and overdeliver. 🔨

Subscribe now

Example: say an SLO commits to 99% of requests in 28 days be responded without errors over a period of 28 days.

If we got 213M requests in that period, at least 210,870,000 requests should be responded to with no error (this number is the Service Level Status in a given period).

Note that any higher number is fine (e.g. 212M) but lower (e.g. 200M) means that the SLO is breached.

Accountability

Both metrics have a notion of accountability. Although in my experience KPI and OKR are usually hard to implement in a way that drives action and is present on a day-to-day basis at the leaf nodes of the organization tree. They’re primarily used by upper and middle management to align key performance metrics.

SLI on the other hand, gets its meaning from the SLO that sets the minimum expectation.

A good SLO should be tied to alerting so that the service owner gets to action as soon as the service level is degraded.

Moreover, a good SLI measures only what is owned by the service owner and nothing more. In fact, you should never be responsible for what is out of your control.

On the other hand, KPIs tend to measure the performance of a larger organization unit (e.g. a cluster of teams) which makes them harder to trace to an individual or service. The notion of ownership is not as granular.

Time Horizon

KPI/OKR is generally concerned with a longer time horizon. Usually, this time span is a quarter of the year. As a result, even if the KPI is not performing too well, the organization waits till the end of the quarter before a definitive assessment.

There’s also a risk that the org will be punished if the OKR is met too early: leadership may just move the goal post and set a more aggressive OKRs.

As a result, Parkinson’s law takes over:

Work expands so as to fill the time available for its completion —Parkinson’s law

SLI on the other hand has a shorter compliance period (also known as SLO window): typically, 30 days.

As we elaborated before, the length of this window directly impacts forgiveness. SLOs having a shorter window than OKRs means that they are less forgiving towards service level degradation and require quicker reactions, hence the alerting.

conclusion

Let’s recap what we’ve learned:

• KPI: high level performance metrics with a wide range of applications. The keyword in KPI is performance.

• SLI: concrete system reliability metrics. The keyword in SLI is service.

Subscribe now

Now, here’s the kicker! SLI can be a KPI!

Let’s say your company sells an API (e.g. OpenAI API access) and charges the customers for every call.

In this scenario, there’s a business motivation to keep the API available as much as possible because downtime directly hurts that business model.

• OKR

  • Objective: keep the system availability high

  • Key Results over the next quarter:

    • Achieve 99.95% API availability

    • Reduce average response time to under 200ms

    • Decrease the number of critical incidents by 30%

  • KPI

    • Percentage of time the API is available and functioning correctly.

    • Time taken for the API to respond to requests.

    • Count of significant incidents impacting API availability.

Turning the OKR/KPI to SLI/SLO we get:

• SLI: Time-Based Availability. SLO: 99.95% over 30 days

• SLI: Event-Based Latency. SLO: 95% of request latencies should be less than 200ms

• SLI: Event-Based non-critical incidents. SLO: 90% of incidents over the past 90 days are not critical

Note that I used several assumptions that were not stated in the OKR or KPI:

• Availability SLI is calculated over 30 days instead of the 90 days that was stated in OKR (to motivate quicker reaction)

• We used a 95% objective out of nowhere! Maybe that’s where the current SLS is at. Maybe it’s coming from the SLA. Instead of average, we replied on the notion of percentile which is embedded in the SLI/SLO formula.

• Currently 60% of incidents are non-critical (baseline wasn’t stated)

And that’s the point! SLI/SLO are usually more detailed, normalized, and stricter towards failure.

Any KPI can be converted to SLI by adding the missing details, but there’s no guarantee that it’ll be desired by the audience or whether it makes sense.

You can also convert any SLI to a KPI by removing details and making it more friendly to a non-technical audience.

Finally, I just want to leave you with an important point: an SLO that is not tied to alerting is just an OKR! That’s because SLO is supposed to tie to accountability of the service owners and alerting is the absolute basic level to implement that accountability.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

In the context of service levels, there are multiple terms that are related to each other. This post serves as a glossary to define and connect:

• Service, its provider, and owner

• Consumer, its tasks, and how failure can impact it

• Metric, indicator (SLI), status (SLS), objective (SLO), legal agreements (SLA)

• Using alerting and error budgets to control risk

As usual, we’ll be using visuals and examples to nail the topics.

Subscribe now

Overview

At the very high level, a product is about solving a problem for some customers.

So on one side, we have the tech stack that solves a problem.

On the other side, we have the customers who have some needs or problems to solve.

The success of the product depends taking calculated risks to bet the services it offers. And good decisions need good data. We need to measure how good those services are performing to help the customers do their tasks.

Put it on a diagram and we get this:

Think of a product as a group of services, each catering towards some tasks for the user.

And that’s where Service Levels come in. It measures the service to provide insights for business decisions that evolve the product.

There’s more to that, and I dissect that in my online book:

• Service levels help set expectations between service provider and consumer.

• Service levels help implement ownership and give a direction to organization design.

• Service levels help balance risks and optimize what matters.

Let’s first build a solid foundation before going into details.

Service

This is the trickiest term in the whole glossary. Too often I see the teams confuse service (in the context of service levels) with service in other contexts like in Kubernetes, microservice architecture, or AWS.

They’re not wrong. That’s just how languages work. A give word can have different meanings in different contexts.

In the context of service levels, a service is an abstract concept. A piece of tech provides the service (hence called service provider), but the service itself is just the ability to solve a problem for the consumer.

Due to common misconceptions, there is a dedicated article about the service.

Service can be any of these 3 types:

• Automated: a service that’s fully automated using software/hardware. For example an endpoint that allows searching book titles and fetching book metadata to render a page in an online book store.

• Manual: a manual service is either too expensive to automate (in terms of money, complexity, etc.) or requires human intervention or judgement (for example due to compliance, and ambiguity that requires flexibility). For example governance, incident triage, or customer support. Manual services are provided by a group of people.

• Hybrid: a mix of the two. For example, the infrastructure team may use IaC (infrastructure as code) to govern access to a 3rd party tool (service). They expect the developers who use the platform (consumers) to create a PR (pull request) with their user-id in the IaC repo. The infrastructure team may manually review the PR. Once the PR is merged, it grants/removes access to that 3rd party tool using GitHub Actions automation.

Provider

Service provider is the entity that provides the service.

If you’re familiar with SAML, you may recognize the term. The service provider is usually a system or component (using Backstage’s terminology) but it can also be a group of people (for manual services):

• Components: are concrete pieces of technical solution that solve a problem for the service consumer. For example: a microservice, a database, an API gateway, a piece of hardware, a 3rd party or open source piece of software. Components can sometimes be deployed independently. For example a single “service” in a microservice architecture, can be thought of as a service provider of type component.

• Systems: are logical grouping of multiple components. For example, an complex microservice architecture that is owned by one service owner can be considered a service provider of type system. Think of systems as folders that contain components as files. In the context of service levels, using system for a service provider creates a more abstract view which has its cons and pros. On the plus side, the abstraction

• Group: are people who provide manual services that are either too expensive to automate or require human intervention or judgement. For example governance, incident triage, or customer support. The point of measuring service levels for manual services is to quantify performance in order to optimize. This data can also help justify automation when it is worth the ROI (Return On Investment).

Note: We use a terminology that is roughly based on Backstage. Backstage is a CNCF open source product for building Internal Developer Portals (IDP). Backstage terminology is roughly based on the C4 model. There are a few differences between our terminology and Backstage:

• In Backstage, Components only provide APIs whereas in our terminology, Components provide any Service regardless of the technology or mechanism that’s used to expose that service. i.e., a component can provide a service that is a GUI (Graphical User Interface) or AUI (Audio User Interface) as well as API like gRPC, REST, or WebSocket.

• In our terminology, the Group (people) and System (logical grouping) can also provide services whereas in Backstage, only components expose an API.

Owner

Service owner, owns the service. We’ve previously established the meaning of true ownership. In the context of service level it means:

1. Knowledge: Service owner knows how the service works, and how its performance is measured by service level indicator (SLI)

2. Responsibility: Service owner is held responsible for maintaining the service level above the objective that was agreed upon (SLO). The basic form of implementing this responsibility is to be on-call for the commitment (alerting on SLO will be discussed in another post). If there’s a legal agreement in place (SLA), service owner is accountable for that.

3. Mandate: Service owner has full control over HOW the service solves the consumer’s problem. This means they have access to all the knobs and gauges to optimize the service they are responsible for and don’t have to beg another party for permission to improve their service level. (this “begging” part is typically a symptom of broken ownership)

Ideally the service owner has all 3 elements of ownership. But the owner is at least responsible for the service meaning if there’s an issue with the service level, the owner is supposed to own that failure and do something about it.

Service owner is typically a group because people own the service. That group can be any organization unit:

• An individual who maintains the service and is responsible to improve it

• A team: that does the same. For example an internal PaaS (Platform as a Service) team offering an internal platform, or a front-end team behind the website that allows customers to order books, or a IoT (internet of things) team that allows the mobile app team to communicate with devices.

• Or a corporation which sells the service. For example a SaaS (software as a service) to solve a particular problem (Example: Adobe) or a Platform for serving content (Example: Substack, YouTube) or software services (Example: Redhat). When money is involved, there’s typically a Service Level Agreement (SLA) in place too.

When a service is manual or hybrid, the service provider and owner can be the same group but not necessarily.

Consumer

As the name suggests, consumes the service.

Just like the service provider, the consumer can be any of these types:

• Group: a person or persona who uses the service. For example, end users may use an app to hail a taxi or find a rider (Uber). If the end user is a paying user, it is typically referred to as customer. When money is involved, there’s usually a Service Level Agreement (SLA) is in place. The user may also be internal. For example, an internal platform may be used by a group of developers. When the commitment is internal, usually a SLI and SLO is enough to communicate how reliability is measured and what is the level of service.

• Component: same as the definition above. For example, a GraphQL API can be a consumer of upstream APIs. A mobile app can be the consumer of an identity service. A web page can be the consumer of tracking service.

• System: same as the definition above. For example, Spotify’s web app is a consumer of their streaming, song catalog, and user tracking services among others. In this context, we don’t care to distinguish each individual component in the app (that information is probably irrelevant and abstract from the service provider perspective), so we see the whole web app as a system.

When the service consumer is a group of people, it may also act as a stakeholder but not necessarily. We’ll discuss the service stakeholder shortly.

Task

Task describes why the consumer uses a service. It may be an arcane word but it makes sense in the context of service levels. Failures threaten the task the consumer is trying to do.

Here are some other terms for “Task” that you might be familiar with:

• Use Case describes how a consumer interacts with a system to achieve a specific goal. This term is also used in UML.

• User Need describes why a user needs to use a service.

• Jobs To Be Done (JTBD) was popularized by Tony Ulwick as a framework that suggests consumers buy products or services to get a specific job done rather than generic features or benefits. The actual “job” here is the task.

Providers have services.

Consumers have tasks.

A consumer may have multiple goals or tasks each of which can depend on a number of services that can be individually measured. This usage essentially establishes a dependency from consumer to the service provider.

The reason task is important in the context of service level is because of the biggest pitfalls of implementing service levels: to measure the wrong thing! Service level should always be measured from the perspective of the consumers. Task describes that aspect.

Failure

Failure describes how a task may not be successful. It is at the base of how [un]reliability is perceived from the perspective of the service consumer.

Failure is the opposite of the “level” in service level. Let’s see why.

Service Leve Indicator (SLI) is a positive concept as seen in the formula:

Note that there’s a small but important difference between valid and total.

Failure is a negative concept as seen in the formula for Service Failure Indicator (SFI):

SLO aims to optimize SLI while error budget aims to optimize SFI.

The reason failures are more important than success is because failures are important for alerting. Alerting is important for holding the service owner responsible for the service level objective (SLO).

One of the common pitfalls of implementing service levels is to measure what doesn’t matter from the service consumer’s perspective.

An effective way to make sure that you’re measuring the right thing is to ask whether it is worth to set alerting and on-call for it. If it’s just a mild inconvenience from the consumer’s perspective, maybe it is not worth waking up the on-call person. Instead you should find other metrics to measure more significant failures. We will discuss metrics shortly.

Failures have the following attributes:

• Symptom: how does the consumer know that something is broken? There is a difference between symptoms and root causes. Symptom is how the failure is perceived by the consumer (e.g. HTTP 500 error) whereas the root cause is one or more issues that lead to that issue (e.g. the certificate is expired, API gateway cannot find upstream, Kubernetes cluster is down, etc.). In the context of service levels, we’re primarily interested to measure the symptoms. The idea is that if a potential root cause happens (e.g. CPU usage is through the roof), but no task is suffering, you should not wake up the on-call person in the middle of the night to investigate. That’ll just lead to alert fatigue and attrition.

• Consequences: what is the consequence of the symptom? For example, if the page they’re trying to reach is the login page and they get a 500 error, they will be practically blocked out of most services and cannot do anything.

• Business Impact: how would the consequences hurt the business? Companies are primarily interested to make money, and if a failure hinders that hunger, it impacts the business. In the above example, a user who cannot log in to the service may file for a compensation (if there’s a SLA in place), or take their money and go to the rivals. The consequence and business impact are related and maybe are the same thing when we’re assessing the risks towards an external consumer (also called customer), but for internal consumers, business impact is about translating the consequence to how the consumer’s failure becomes something the business cares about. For example, if a platform team (service provider) fails to provision a new repository (service) for the developers (service consumer) in a timely manner (symptom) because the underlying infrastructure faced an edge case (root cause), the developers get blocked (consequences) and the business is paying for their idle time (business impact).

• Impact Level: while business impact describes how the business takes a hit from failures, the impact level puts that into perspective. There is a difference between users on an old version of Android being blocked from logging in than all users on all platforms not being able to check out their shopping card. The impact level of a failure usually fits nicely to incident severity categorization that is used for incident triage.

• Likelihood: this is hard to predict but roughly speaking risk = threat x likelihood: the more severe a threat and the more likely it is, the higher the risk. Therefore it’s good to have an idea about how likely a failure is, in order to prioritize preventing it.

Subscribe now

Metric

Metrics quantify a behavior. They are at the core of Service Level Indicators which in turn are visualized as Service Level Status and trigger alerts to guarantee the Service Level Objective.

In the context of service levels, we’re primarily interested in metrics that quantify failure/success of the service provider from the perspective of the service consumer.

Example metrics: query latency, API error rate, queue length, timed-out requests, game rendering frame rate, GPU utilization, LLM token/second, etc.

You can think of a metric as an array of data points each with:

• Value: what was the measurement

• Timestamp: when that behavior occurred)

• Metadata: For example, tags, labels, and annotations indicating the source, and providing more context about that measurement.

Data point type can be either:

• Boolean: e.g., success of a probe checking if an endpoint is up, or whether a database record is up to date, etc.

• Numeric: e.g., error rate, or request latency, etc.

Each data point may present:

• A timeslot: e.g., if we probe an endpoint every minute, the status of the probe is extrapolated to assume the status of that endpoint over that entire minute

• An event: e.g., a failed request, a cache hit, etc.

Timeslot data points are useful for time-based SLI whereas event data points are usually used for an event-based SLI.

An aggregation function (e.g., avg, min, max, percentile, sum, rate, etc.) can be used to convert an event metric to a time-based SLI. For example, you may use the P99 percentile value of every 5 minutes to represent that timeslot.

Just a reminder that a service can be automated (provided by a component or system) or manual (provided by a group of people).

Measuring the services provided by a group of people can be particularly challenging because not every manual operation leaves a trace that is quantifiable. That’s why you see things like this:

On the other hand, the biggest pitfall when measuring service levels of automated services is to measure the wrong thing or at the wrong place (we’ve covered the nuances of measurement location here).

Stakeholder

While the consumer’s perspective is the key to define the right service level indicator (SLI), the stakeholder has a say to what the service level objective (SLO) should be or how much error budget is acceptable.

Stakeholders are a group of people representing the interest of the business (which is ideally the consumer’s interest but not always).

Stakeholders can help assess the business impact and impact level of failures.

The paying end-users (also known as customers) are typically seen as a stakeholder because they have the money leverage. While that is true, there are typically internal stakeholders (e.g., PM or UX) who have every intention to keep the service level as high as possible. Ideally, they want 100% for everything. It’s important to manage their expectations with tools like the Rule of 10x/9 to add the cost of reliability into perspective.

Stakeholder is a key part of the negotiations for SLO which in turn shapes the alerting rules.

• Stakeholders negotiate on behalf of the service consumer for a higher SLO

• Service owners negotiate for a higher error budget

For internal service levels, the consumer team representatives (PM/EM/Tech lead, developers) act as stakeholders. Since your service is their dependency, they want you to measure what matters to them (SLI) and hold the service owner accountable for a high level of service (SLO).

For example, many companies do platform engineering. The platform itself is typically not a product that is sold to the end users, but rather is there to improve efficiency and productivity. The teams that use the platform should have a say in what metrics they want to be improved (SLI) for example:

• Time to provision a new service

• Time to deploy a change to production

• Time to grant access to key systems to a new joiner

The stakeholders in this case are all the people who have a say in what good looks like. The sheer number of stakeholders is one aspect that makes platforms hard to get right. We’ll cover platform engineering as a product in another post.

Cheat sheet

Putting all these terms together, we get this relation map:

• Service is a capability or feature that solves a problem for the consumers.

  • Service Provider provides the service and can be automated or manual. While service is an abstract concept (a solution to a problem), the provider is often a piece of technology like a microservice, an API gateway, or a database.

  • Service Owner owns the service. Ownership refers to 3 elements:

    • Knowledge: know the tech, its capabilities, and the problems that it is supposed to solve

    • Mandate: has enough control over the tech to improve the service level

    • Responsibility: with power comes responsibility. If the technical decisions lead to failure, service owner is responsible to fix it.

• Service Consumer uses the services for their tasks and use cases.

  • Task describes why the consumer uses the service.

  • Service Failure describes what happens when the consumers cannot do their tasks using the service:

    • Failure Symptom: How the service failure is perceived from the perspective of the consumer? How would they know something is wrong with the service?

    • Failure Consequence: How will the consumer suffer when they cannot do their tasks because of the service failure?

    • Failure Business Impact: service is part of a product that makes or saves money for the business. How will the business suffer when a particular failure happens?

  • Stakeholder is the equivalent of the service owner on the consumer side and negotiates the Service Level Objective (SLO), even if it’s part of the SLA.

• Service Level describes how the level of the service is perceived from the consumer’s point of view

  • Service Level Indicator (SLI) describes how the service level is measured.

    • SLI takes a metric and turns it into an indicator of failure/success events or time slots.

  • Service Level Objective (SLO) sets expectation on the service level

  • Service Level Status (SLS) quantifies the value of SLI at a given time. Its value can be graphed in direct relation to SLO.

  • Service Level Agreement (SLA) ties that expectation to some sort of guarantee (often financial and only when the consumer has a leverage). SLA contains at least one SLO.

One last thing: Product

With the language we’ve built, it’s very easy to define the product as a collection of services.

In other words, product is just another abstraction layer in the service level modeling.

Those services may be tightly coupled and or complementary to each other. It doesn’t matter. A product is a group of solutions to fulfill consumer tasks.

This is particularly useful for teams with internal stakeholders or consumers. For example, internal platform teams or common components used by many apps.

This definition of product unlocks very power perspective:

• It separates the concerns of the consumers from that of the service owners

• It can guide the architecture of service providers for lose coupling and separation of concerns

• It can by extension guide the organizational architecture to map to the logical grouping of service providers

A good usage example of this mental model is platform products. Quite often I hear the platform teams ask “what is our product?”

The answer can easily be deduced by defining their services in terms of what they solve for the rest of the organization and why do they exist.

Organizing the teams to streamline the consumer journey is a good way to implement full ownership:

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

You can also refer your contacts and gain free subscriptions. Thanks in advance for helping these words reach further and impact the software engineering community.

I’ve spent the past few years learning, teaching, and coding service levels. So I got curious to see how the well-known observability providers implement it.

This post is my independent review of all 3 with snapshots.

Disclaimers

• This is not a paid or affiliated review. None of the vendors got the chance to see my review before it went public and I’ll clearly mention if they ask me to do an update.

• This review was carried at early September 2024 on the same day for all 3. Obviously, the UI and feature set may change in the future.

• If I missed or misunderstood something, you can correct me in the comments.

• I don’t represent my current or past employers or customers on my personal blog. This review is my independent investigation.

Elastic

The first stop is Elastic because they were open source, then they were not, and recently they are open source again!

Unfortunately, the SLO feature is only available in their Platinum price tier at the time of this writing. I really hope this will change because if metrics are data, Service Level is the perspective that makes that data actionable in relation to business objectives and consumer needs.

Charging for SLOs separately is like charging a premium for GPS navigation system of a car. Do vendors do that? Yes. But they also force the majority to use their phone instead. SLO is not rocket science, don’t force people to use alternative solutions just to take full advantage of the data they’re already sending your way! If anything, having SLO on all plans is a motivator to send more data because of the value it creates.

So I’m gonna go MKBHD on this one:

Once you do have access, SLOs have a prominent place in Elastic UI which is great!

Elastic uses a 3-step single page setup for creating new SLOs:

Right off the bat, you have to choose a SLI type:

As discussed before, there are only 2 types of SLI:

• Event-based: count the percentage of good events in a period

• Time-based: count the percentage of good timeslots in a period

Later the UI Elastic has the “Budgeting Method” which tells me that the Elastic team is aware of the two SLI types:

The “SLI type” drop down seems unnecessarily complicated but as we’ll see shortly, Datadog is doing something similar. I’d drop the word “type” or use “templates” instead because where’s in the menu (latency, availability, etc.) are common SLI examples.

Let’s pick one and roll with it.

It’s nice to see a live preview of the SLI. The Threshold field only allows setting the upper threshold:

Aside from being upper-bound, the definition of Good may be lower-bound or even bound between two thresholds. Those options aren’t available.

Note: it is possible to convert any definition of good to look upper-bound. But that creates complexity that you’d expect your observability platform to handle out of the box.

There are 2 types of compliance period (Time Window):

• Rolling: e.g. always check the last 30 days

• Calendar-bound: e.g. check the budget in the current month

In Elastic, your choice of Duration for rolling window, however, is limited to 7, 30, and 90 (just like Datadog).

Grafana is the only one that allows any number and as we’ll discuss the most important missing option is 28 days.

Elastic is the only one in this review that supports Calendar-Bound SLO windows out of the box.

Unfortunately, you have even fewer options: weekly and monthly.

Weekly is particularly tricky because different countries across the world start the week on different days:

Elastic does not allow invalid SLO targets like 0% or 100%. They do however allow ridiculously low numbers like 0.1 while blocking 0.0001:

Similar issue exists for the upper end of the spectrum where a SLO of 6-nines is forbidden:

Admittedly, 6-nines is not very common (and has a high cost) but I don’t see a reason why the tool should get in your way if that’s your SLO.

However, if their SLO implementation is limited to 3 fractional positions, that is a serious issue. Essentially you won’t be able to implement SLOs like 99.1234% because it’ll be truncated to 99.123%. I haven’t tested that. If you know more, please comment down below.

After you create a SLO, the waiting spinner starts rotating for a long time. In my test, I waited about 30 seconds before the SLO was created which is by far the longest in this review.

When it succeeds, your brand new SLO gets a default dashboard.

The bottom part of the page is interesting where you can see the good and bad events in a stacked bar chart:

You can also see the Burn Rate and Historical SLI (which I call SLS for Service Level Status).

Elastic does not create alerts by default despite me setting a target for the SLO. This is a flaw they share with Datadog.

Without Alerting, there is no commitment towards the SLO. SLOs are important enough to nudge the user to create alerts in the initial SLO configuration screen.

Fortunately, creating an alert is just a click away (as mentioned in their SLO creation documentation):

The alerting setup for burn rate looks intimidating and complete:

It’s a nice touch that Elastic prefills the multi-burn rate alerts with decent default values.

But once you look closer, you realize that the config for short lookback alert window is missing. This feature is important for reducing false alerts as pointed out in Google’s SRE book.

Elastic SLO Alerting documentation states:

Your short lookback period is set automatically.

Google recommends 1/12th of the long window and I’m guessing that’s what they’re going for. Hard to tell. The docs are 🤐 on that.

I also really like that Elastic allows switching between Burn Rate and burn percentage with the flip of a button:

However, for the life of me I cannot understand it. As you can see in SLC, you need to set both the burn rate and error budget consumption percentage. Elastic seems to automatically set that value (as seen in the grey text).

Like regular alerts, Elastic offers a wide range of notifications and actions when the SLO alert goes off.

Datadog

Datadog’s Service Level feature is under Service Mgmt menu (which if you’re wondering, stands for “Service Management” in a world where the UX team wins the battle over UI designers): 😁

Before digging into UI, I just want to call out their excellent documentation on the topic completed with an ebook.

Just like Elastic, Datadog uses a single 3-step screen for setting the SLO:

If I recall correctly, Datadog had the SLO feature way ahead of the pack whereas Elastic recently introduced it. So it’s up to you to guess who has copied whom. I mean, one of them could pick a different style for the 3 step numbering?

SLI can either be time-based or event-based. Datadog offers two time-based SLIs:

• By monitor uptime: requires having set up a monitor (an Alerting Rule)

• By time slices

When it comes to evaluating the monitor (Datadog lingo for Alerting rules), it has the same limited choices as Elastic: 7, 30, and 90.

I do like the live preview of the monitor uptime however (Elastic has this feature too):

For event-based SLIs (or “By Count” as Datadog calls it), you can choose two metrics for Good/Total Events:

This is a heterogeneous SLI (when the two metrics aren’t tightly coupled). Heterogeneous SLIs are easier to implement in an already existing observability platform—”just give me 2 metrics”.

But in practice, they can be hard to reason about as clearly demonstrated with this funny error message:

Personally, I prefer the homogeneous SLIs where there is one metric that can be processed to identify good events.

Even though you can pick Good/Total, the live preview shows Good/Bad:

Datadog doesn’t allow invalid SLOs like 0% or 100%:

For the time-based SLIs, I appreciate the simplicity and control in Datadog’s UI for choosing the boundaries of good:

There is more control and settings that I can understand, yet the UI is simple enough that I think I can understand if I need to!

I also like their Warning Threshold which fits nicely with how Alerting/Monitoring is done in the rest of the Datadog platform.

Although it can lead to alert fatigue. Done right you wouldn't need warnings because of how Google cleverly uses long window and short window. Fortunately warnings is an opt-in feature in Datadog.

Once you create the SLO, you’re presented with a very well thought through dashboard showing Error Budget Burndown, Burn Rate and the color coded status:

Overall, Datadog continues to impress me with their UI but their price tag remains a pain point for many teams.

One important issue to note is that Datadog does not create monitors by default despite me setting a target for the SLO:

Just like Elastic, Datadog doesn’t create alerts out of the box. Fortunately, creating an alert is just a click away:

But it would be better to get the alert as part of setting up the SLO.

When it comes to setting an alert, you have two choices:

• By Error Budget consumption

• By Burn Rate

For the Error Budget burn, I find the default choice of 100% very curious! Here Datadog does not make any effort to make it easier to set a sane default (what happened to POLA?).

I would have gone with any of the defaults suggested by Google which is max 10%!

For the Burn Rate, you need to know what you’re doing. Here, I have configured it according to one of Google’s recommendations:

The short window lookback is also hard coded to 1/12th of the long window. This is a sane default and still better than Elastic that hides this information. But I miss the ability to set it to other values. Whoever wrote the documentation was aware of at least 1 use case for it

If you find that your burn rate alert is consistently too flaky, this is an indication that you should make your short window slightly larger.

Nor can I disable short window alerts.

In the Service Level Calculator, you can see all 3 suggestions:

Note that the 1h/5m recommendation is for a 30 day SLO window.

For a 7-day window, if you go with 2% consumption at 14.4x, you’ll end up with 14 minutes for long window lookback and ~1 minute for short window lookback. Technicalities, I know, but it feels that Datadog’s SLO feature is just put together in a sloppy manner to tick a feature box. I have higher expectation at Datadog’s price point especially because the rest of their platform has a great UX.

Again, what they miss in their UI, they cover with documentation:

The documentation has terms like “good behavior” instead of “good events” or “good timeslices”.

I also found the term “theoretical error budget” confusing, but the document does not disappoint with wise sentences like this:

theoretical error budget consumption and actual error budget consumption are equal because time always moves at a constant rate

You don’t say! 😂

Fine! What’s wrong with just plain “error budget”? You certainly don’t find a reference to “theoretical error budget” in Google’s books. Service Levels are already hard enough. Why make the learning curve even steeper by making up new words? (Uhm, said the guy who invented SLS and 10x/9!) 🫠

To my knowledge Datadog doesn’t support OpenSLO but they support Terraform.

Grafana

Grafana refers to Service Levels as SLO and carefully hides them under Alerts & IRM. IRM stands for Incident Response Management, if you’re wondering about that obvious acronym! Dammit UI designers! 😂

This is already a nice touch because Service Levels need to be tied to some responsibility and alerting is the absolute basic mechanism to do that.

In fact, alerting is such a prominent feature of Grafana Service Level implementation that it has a dedicated filter in the SLO management:

Alerts are also a key attribute of the SLO. A dedicated button takes you directly to alerting configuration:

Love it! 💖

The process of creating an SLO is straightforward using a wizard UI:

You see? 5 steps! Not 3! And certainly no circles. Grafana is original! Or at least their UI designers know how to work from home.

You can enter any compliance periods as long as it’s in days… Or nights… You get the idea! 28 days is common for weekly seasonality (when the metric trend follows a weekly pattern) but 30 days is the most common:

Grafana doesn’t seem to support calendar-bound compliance periods (e.g. budgeting per calendar month) as far as I can tell..

SLI can be either time-based (e.g. uptime) or event-based. Grafana’s default UI doesn’t bring up that aspect which is a miss. Typically, time-based SLIs are easier to start with and I’d expect to see that choice in the first step of the wizard as a prominent option.

As we discussed before, the formula for SLI is success divided by valid events or timeslots.

There’s a distinction between valid and total which is often underutilized. Grafana chose the simpler term:

Just like the other two, Grafana only supports heterogeneous SLIs.

I suppose homogeneous SLIs are possible in the Advanced view, but my evaluation instance didn’t have enough data to try this aspect:

Setting SLO and Error Budget in one step is another nice touch in Grafana because these two values are tightly coupled:

However, I would have preferred a percentage slider as implemented in SLC because unlike other settings, this one can only be between 0-100 (typically above 90%).

This is absolutely not a deal-breaker, but what is a deal-breaker is the fact that you can set the SLO to 100% and it doesn’t even complain! 😱

However, setting it to 110% seems to kick in the validation mechanism:

Although they still show an error budget of -10%? Is that a budget or giveaway? 🤑

Personally, I’d prefer the UI to scream if the value is too close to 100%. For example, SLC shows an error with a link to the rule of 10x/9:

Picking an unrealistic value like 100% can negatively impact the alerting experience or deployment frequency (100% is totally fine for SLS but not SLO).

That’s one reason that SLC exists: to set reasonable expectations.

Moving on. Another aspect of Grafana’s Service Level implementation that I absolutely love is their categorization mechanism:

• You can put the SLO in a folder

• It to a team name

• And define custom labels for it (both Datadog and Elastic have this feature)

These features make it easier to manage SLOs across a large organization, which is exactly the type of challenge I am dealing with. Kudos for that.

The 4th step of defining a SLO in Grafana is Alerting.

There are two predefined alerting rules to pick from:

• Fast-burn alert has 2 rules:

  • Burn Rate: 14.4x

    • Long lookback window: 1 hour

    • Short lookback window: 5 minutes

  • Burn Rate: 6x

    • Long lookback window: 6 hours

    • Short lookback window: 30 minutes

• Slow-burn alert also has 2 rules:

  • Burn Rate: 3x

    • Long lookback window: 1 day

    • Short lookback window: 2 hours

  • Burn Rate: 1x

    • Long lookback window: 3 days

    • Short lookback window: 6 hours

Apart from the 3x rule, the rest comply with Google’s recommendation:

Notice that Grafana does not provide a setting for adjusting the percentage of error budget that is allowed to be consumed before triggering the alert. That’s because their settings are primarily focused on the lookback window and burn rate.

I would prefer working with the percentages because they’re easier to reason about than a time period.

Which one would you pick:

• Trigger an alert after burning 2% of error budget at +14.4x

• Trigger an alert after burning the error budget at +14.4x over the last 1h and still burning in the last 5 minutes

The time-based settings are more sensible for the time-based SLIs —which as we discussed aren’t explicitly supported by Grafana. This is just confusing to me.

Another confusing option is the Minimum Failure Threshold:

It is an Advanced Option allowing you to set the minimum number of failures required for the alert to fire. But I don’t understand why it’s necessary (I’m guessing it was added as an after-thought due to problems with the purely time-based alerting options namely when there’s not enough failures.

What is an alert without a notification? Unfortunately, the notification settings aren’t part of the wizard. All you get is this:

If you click that link, you’ll be thrown into the generic alerting menu. This makes it easier for Grafana implementation DRY-wise but breaks the nice flow of the wizard.

Fortunately, the SLO Alerts will use the default notification policy, which is a sensible choice.

Step 5 is a review screen so it could totally be a 4-step process.

I did not see a live view of the SLO settings, something that had a prominent place in both Elastic and Datadog SLO setup screen.

Notice how Grafana happily accepted my unrealistic SLO of 100% while highlighting a 0% error budget in green:

Fortunately, this should be very easy to fix: if it didn’t allow me to type 100% in step 2, this wouldn’t happen.

Sum up

Datadog has the best UX followed closely by Grafana. The reason Datadog wins the UX is their excellent documentation which honestly is a great learning resource whether you use their platform or not.

Elastic’s UX looks similar to Datadog but more janky and less sensible copy. And short window is missing from the UI.

Grafana has the most complete options. It allows setting custom days for rolling windows but misses time-based SLIs.

Both Grafana and Datadog support setting SLOs on synthetic monitoring. I’m guessing Elastic does too but I didn’t try.

All 3 only support heterogenous SLIs.

Datadog has the most awesome out-of-the-box SLO dashboards again followed closely by Grafana.

Alerting is a very important aspect of SLOs. Admittedly it is hard to get right and I have it in my backlog to explain in a separate blog post.

Subscribe now

Grafana has the best alerting setup for the mere fact that it is an integrated part of setting up the SLO. Both Datadog and Elastic require an extra step to set alerts.

Datadog and Elastic come second for different reasons:

• Datadog doesn’t have good defaults

• Elastic has nice defaults but misses key configuration options

Personally, I prefer Grafana when it comes to the SLO implementation.

These posts take anywhere from a few hours to a few days to ideate, draft, research, illustrate, edit, and publish. I pull these hours from my private time, vacation days and weekends.

Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public.

My monetization strategy is to give away most content for free because I believe information should be free and accessible. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections (on some articles) as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access or get a discounted group subscription. There’s also a referral bonus program to gain free subscriptions.

Thanks in advance for helping these words reach further and impact the software engineering community.

• Percentage of requests where response.latency < 300ms

• Percentage of uptime probes where response.code == 200

• Percentage of database records where fields.*.age < 365 days

• Percentage of authenticated API requests where http_response.code < 500

When it comes to implement and calculate those percentages, there are two approaches:

• Heterogeneous: good and valid are two different metrics with an assumed correlation.

• Homogeneous: good and valid come from processing the same metric where good and valid are tightly coupled

This article introduces both types with examples and illustrations ending with a recommendation.

Heterogeneous

This type of SLI queries two different metrics to count the number of:

Good and valid:

or good and bad:

or bad and valid:

Regardless of the approach, it ends up querying two metrics.

Example: Availability of online purchase flow:

• Valid: number of unique sessions where the user clicked on the “Pay” button on the check out page

• Bad: number of payments failing due to payment gateway issues

• Good = Valid - Bad

Note: we deliberately filter out other failures like: when the user doesn’t have enough money, or entered the wrong payment information because you should never be responsible for what you don't control.

On the surface, it looks reasonable, but in practice there are multiple nuances. The biggest issue is the assumed correlation between the datapoints of two different metrics over time:

This is because the data points of two separate metrics don’t exactly match. As a result, for any type of lookback window (e.g. SLS or Alerting), we may end up with an inaccurate measurement.

The problem doesn’t stop at the timestamp level. It is practically impossible to guarantee a 1-1 mapping between valid events and good events when they’re measuring different things, at different locations, and different times.

Another obvious issue is the fact that we need to measure, store, and query two separate metrics (valid and either good or bad).

Pros

• Can correlate events across different systems which may map better to how the consumers perceives reliability of the service

• Easier to reason about for beginners

• Easier to implement by conventional Observability providers (in the next articles we’ll review how Elastic, Datadog, and Grafana implement service levels)

Subscribe now

Cons

• It might be harder to work with two datasets due to broken correlation between datapoints.

• Less efficient due to requiring 2 metrics

Homogeneous

Homogeneous SLIs come directly from the fact that the SLI formula is a percentage:

All you have to do is to:

1. Filter the metric to only contain valid data points

2. Go through the data points and count the number of good (or bad) events

3. Divide it by the total number of data points

It’s easier to show it with a diagram:

The biggest advantage of this method is the reliable correlation between good, bad and valid.

Note: an upcoming feature of SLC, introduces a simulator. It uses an O(1) algorithm to process the metric data in one pass. I’ll write more about that technique once the feature is out.

An obvious advantage of a homogeneous SLI is that it only needs to measure, store, and query one metric.

The downside is that your SLI may not readily be one metric. You may need to use a data processing pipeline that takes in 2 metrics in and spits out a clean homogeneous SLI. This can introduce a delay and cost to your setup.

The good news is that most metrics don’t need this type of processing. For example:

• Uptime calculation: divide the percentage of successful probes during the window, multiply by 100

• Percentage of authenticated requests where latency < 200ms: find all the authenticated requests in the window, filter out any events where latency ≥ 200 and divide this count to the total number of requests x 100

Pros

• It is more intuitive because it’s closer to how normalized SLI is literally defined

• It requires half as many metrics (only valid)

• The SLS calculations essentially turns into a percentile calculation so it is easier to reason about the dataset

Cons

• When the good and valid come from different systems, it requires extra processing to corelate the events which adds to the cost and complexity

• None of the observability providers that I have investigated (Datadog, Elastic, Grafana) support homogeneous metrics out of the box.

When to use which?

Whether you use homogeneous or heterogeneous metrics depends on how the consumers perceive reliability, your tooling, and budget (time, and money).

Homogeneous metrics are easier to reason about and often come naturally out of the box.

If you can ensure a tight coupling between the two metrics, a heterogenous metric can work too.

These posts take anywhere from a few hours to a few days to ideate, draft, research, illustrate, edit, and publish. I pull these hours from my private time, vacation days and weekends.

Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public.

My monetization strategy is to give away most content for free because I believe information should be free and accessible. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections (on some articles) as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access or get a discounted group subscription. There’s also a referral bonus program to gain free subscriptions.

Thanks in advance for helping these words reach further and impact the software engineering community.

Service Level Indicator is a metric. Service Level Objective sets the expectation.

Fine! But how can we calculate SLI? What is the day-to-day experience of using service levels? More exactly, how do you relate the raw metric values (e.g. latency, error rate, cache hit rates, etc.) to a target that looks like 99.96%?

There’s a conceptual gap between the metric values and SLO target.

This is where Service Level Status (SLS) comes to the picture. SLS processes the values of the service level indicator metric over time.

• It uses the definition of event, good and valid from SLI

• Then evaluates each data point according to the constraints specified in SLO and aggregates them during the window specified by SLO

The formula for SLS is exactly as the normalized SLI:

SLS has multiple advantages:

• It allows keeping track of historical level of service in relation to the objective

• SLS can be plotted on a diagram where the x-axis represents the time and the y-axis is in percentage (0-100).

• Each value represents the status of service level in relation to the SLO which is a horizontal line.

Here’s a sample data for a latency SLI:

Here’s the SLS for the same data in relation to a 99% SLO:

This article introduces SLS, its origin, examples, and metaphor.

The origin story

Those of you who have read Google’s [excellent] books might be scratching your head about “SLS”. That’s because I coined this term! But don’t let that put you off. There’s a story behind it.

I’m responsible for rolling out Service Levels across a relatively large organization with hundreds of internal and external services. Aware of the service level adoption obstacles, I take the time to meet each team, get to know their service, and define a service level indicator together with their consumers in the room.

At one workshop, a participant asked:

— “If the SLI is the metric, what do you call the data points of that metric?”

— “The data is the SLI”, I replied

— “But they are different things.”, he pointed at their SLI which read Latency and the definition of good events as respone_latency < 500ms.

— “At any given point in time, we are counting the number of requests that were responded in less than 500ms and divide it by total number of requests, right?”, he continued.

— “Right”

— “This is different from latency. Latency can be any number from a few milliseconds to a few seconds, or even time-out. The value we are talking about is always between 0-100 (inclusive) and the SLO is a straight line”, as he grabbed a pen and drew something like this on the whiteboard:

Service level status is different from service level indicator or objective. It’s something in between!

• The values are always between 0 and 100 (like SLO) regardless of the range of possible values for the SLI

• It has a value for any given point in time (like SLI)

• Any value below the SLO line indicates a breach

• It accumulates the number of good requests over the SLO window instead of the real time latency value

• The latency metric diagram changes in real-time whereas this one is is less “jumpy” due to accumulation

This metric shows the status of the Service Level. It is the Service Level Status, or SLS in short.

Subscribe now

Calculating SLS

There’s no magic in calculating SLS. We just count the number of good events (or timeslots in case of time-based SLIs):

To calculate SLS at any given point in time, we look back and count the events.

One of the core ideas behind service levels is to put failures in the perspective of a time window. As you can see, the value of SLS is more sluggish as opposed to the raw latency values of SLI.

Not just latency

Both SLI and SLS are valuable but they tell different stories. SLI is more “raw” and can help understand the system behaviour in real time. SLS on the other hand correlates that data to the commitment towards the service consumer.

Example:

• Availability

  • SLI: the HTTP response code for a GET request to an endpoint

  • SLS: the percentage of successful GET requests to that endpoint over the past 30 days

• Throughput

  • SLI: number of cache hits

  • SLS: the percentage of cache hits over the past 30 days

• Error rate

  • SLI: number of operations that did not fail

  • SLS: percentage of operations that did not fail over the past 30 days

Note: we just used “30 days” as an example window (also known as compliance period) but it can be any other period.

Pro-tips

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to ideate, draft, research, illustrate, edit, and publish. I pull these hours from my private time, vacation days and weekends.

Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

There’s also a referral bonus program to gain free subscriptions. Thanks in advance for helping these words reach further and impact the software engineering community.

Read more

This article elaborates:

• Why does the shape of the organization matter for reliability?

• How does it impact communication between teams and by extension interactions between the components of the system?

• What insight can be unlocked with consumer journeys to create a healthy organization that builds reliable systems?

This is part 1 of a longer article about the pros and cons of different organization structures that I dubbed Smorgasbord, Kebab, and Cake! 😋

Subscribe now

Organizational architecture

Just like software architecture, organizational architecture is about composing a whole larger than the individual components.

Similarly, organizations have emergent properties due to the interactions and dependencies between their parts. In simple terms, collaboration between people shapes communication between systems.

Collaboration is heavily impacted by the organization structure, reporting lines, delivery timelines, and the territorialism that our species still shares with other primates.

These are the mechanics of Conway’s law. And inverse conway maneuver cleverly takes advantage of them.

Many companies ship their org structure to the end user.

You’ve probably been through experiences that shape-shift between different steps of your journey. Sometimes, there is a benign transition like a screen flicker or UI change. Some other times, your journey just chokes at the point of handover between what appears to be built by different teams.

You learn a lot about the people behind a product just by using it.

On paper, the system fulfills the requirements. In practice, the sum of the parts is a negative experience!

The handover between teams has a higher impact than dependencies between systems.

But why is it so hard to design an organization based on the systems it produces?

The answer reveals itself once you take a glimpse at a system diagram and organization chart side-by-side:

Note: the org chart above is too idealistic. The reality looks more like this:

OK, maybe the captions are not ideal, but you get the idea. 😊Humans being human have their own diverse range of values, rules, and biases that directs who they trust, who they listen to, and who they avoid. That’s not new.

I’ve previously written about an example of this phenomenon for platform teams:

Software components often have dependencies that span across the formal organization. There are known dependencies, then there are unknown dependencies that grow over time. Then there are indirect dependencies that usually no one knows about until sh*t hits the fan.

Both the human side and the system side are very complex. An organization architecture is usually outdated before it is announced. That is one reason some companies are in constant reorg mode. To make it worse, many leaders don’t understand the J-curve that comes with every change.

The J-curve refers to the initial dip in performance or productivity as teams adapt to new changes. Constant reorgs accumulate the worst of those curves.

Macro-optimization

Too much has been written about how to improve the individual components of a service using TDD (test driven design), CI/CD (continuous integration and deployment), SRE (using software engineering practices for operations), refactoring, paying tech debt, and buying 3rd party tools (usually their sales pitch goes like “We take care of X, so you can spend more time coding”). 😄

And that’s fine. Good engineers want to build systems that they can be proud of.

However, these methods are micro-optimization with their own limits.

To significantly improve system reliability, we need to look at larger forces at play:

A system is not the sum of its parts, it’s the product of their interactions. If we have a system of improvement that’s directed at improving the parts you can be absolutely sure that the performance of the whole will not be improved. —Russell L. Ackoff (I highly recommend watching his 90 second video)

Poor organization architecture ignores these obstacles:

• Every team dependency increases the risk of misunderstanding.

• Every point of handover opens a miscommunication crack for things to fall into.

• Every new process and role increase the risk of delay.

• Every unclear or broken ownership is a vulnerability for the blame game.

• Every incident starts a novel maze to figure out which team is behind what component and how to motivate them to own it and fix the failure ASAP.

The handovers, miscommunication, and lack of ownership show themselves in the following symptoms:

• Lower velocity: a seemingly simple change takes disproportionally long time because the developers have to jump through hoops and do trial-error or in some cases reverse-engineer systems to get the job done.

• Lower TBF: time between consecutive failures

• Higher LT: lead time from when a change is made till it's available in production

• Higher CFR (change fail ratio): changes to code, config, or content which break production

• Higher TTR: the time from incident symptoms hit the consumer until the root cause is fixed (which is directly impacted by how hard it is to find the right team)

• Higher impact radius: how far does one failure propagate

• Higher incident cost: the negative impact of incidents on sales, potential revenue, SLA penalties, and company reputation

• Higher manual to automatic incident ratio: the number of incidents that were discovered manually (either by staff or user reports) divided by the number of incidents that were discovered automatically (e.g., through monitoring, synthetic tests, etc.)

• Higher false pages: the number of people that were paged mistakenly by the central incident team in order to find the right person who can fix the failure

• etc.

Inferior quality and reliability are just the symptoms. The real root cause (no pun intended) is poor organizational architecture.

Leaving services to individual talent and managing the pieces rather than the whole make a company more vulnerable and creates a service that reacts slowly to market needs and opportunities. —Lynn Shostack, who coined the term Service Design in 1982

Weak leaders miss the root cause and work around it by adding more processes, people, roles, gate keeping, and friction to the mix hoping to improve system.

There is always a good reason for the process but it’s never about empowering human ingenuity. It is about protecting against human stupidity.

Adding headcount to improve delivery bandwidth is the biggest fallacy due to the extra communication bandwidth it requires:

Anyone who has worked at a large organization should be familiar with all the meetings, demos, infomercials, newsletters, mandatory training, all hands, etc. I have a dedicated post on a solution here:

People, process, tools, etc. all have a cost both in real money and time. However, a variation of the velocity, reliability, and cost diagram is [ab]used to justify the price tag. 🤦

Reorg after reorg, the theme stays the same:

• When you talk to the engineering managers, they complain about “cat herding”, “running a kindergarten”, or “immature organization”.

• When you talk to the product managers, they complain about feature teams (or feature factory as puts it).

• When you talk to the team members, they complain about micromanagement, gate keeping, and blame other teams.

• When you talk to the C-suit, they sigh at the high operational cost, low productivity, and disproportional damage of incidents.

• When you look at the tech landscape, you see a fragmented landscape that seems to contain every technology invented since Unix epoch. Someone may even naïvely throw a tech radar in there hoping to defragment the tech landscape to no avail.

• When you look at the architecture, you see a tangled spaghetti where every component can potentially be connected to another component at various times.

• When you check the incident vitals, you find lots of novel incidents that were detected manually, took a long time to fix, and caused unreasonably high damage to the business.

• When you look at the total cost of designing, building, monitoring and maintaining the software, you realize that the org is too big for what it’s trying to achieve.

Disclaimers:

• Before planning a layoff, please make sure to read till the end. Layoff is only an option if the company is unable to monetize the accumulated human capital it has already hired. It has long lasting negative effects.

• If any of my current or former colleagues sees the above section as a rant, rest assured that I have seen variations of those problems at a dozen companies over the past 2 decades. One of my principles is to never complain unless I have a solution or am willing to be part of the solution. Hopefully, this post and the next post help.

Is there a way to improve reliability and reduce the cost of running the business without hurting velocity?

The key perspective to unlock the answer is the consumer journey.

Consumer journey

In the field of UX (user experience), a user journey is:

A scenario-based sequence of the steps that a user takes to accomplish a high-level goal with a company or product, usually across channels and over time. —Kate Kaplan

User flow is a related term but is more detailed, idealized, and typically concerned with one product or channel (see a comparison between these terms here).

User journey is primarily used when designing experiences towards external customers (e.g., end user of an app, website, or device).

I use consumer journey because:

• It is a generalized term that applies to services that have internal or external consumers.

  • Internal service example: an internal infrastructure platform that is used by application developers.

  • External service example: a mobile banking app or your favorite 1-minute video story app. 😉

• The word consumer is more consistent with the language we have built so far about services.

• It is concerned with the high-level journeys as opposed to low level flows. The high-level view is more useful in the context of organization design.

There are two important things to note here:

• The journey order is not always predictable. Example: in an internal platform product, a developer may first read the docs before provisioning resources while another developer may do the opposite

• The journey may traverse interfaces. Example: in an external banking app, a user may request a new payment card without freezing an old one. Another user may start the journey on the web and continue on the mobile app. Yet another one may do the opposite.

Subscribe now

Kebab vs Cake

This article scratched the surface for a follow up which talks about 2 common organizational architecture patterns, their pros and cons, as well as a pragmatic approach to use consumer journeys to design organizations that build reliable systems.

Read the follow up here:

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

Share on Hackernews.

Table of Contents

(🟢=Written, 🟠= Drafted, 🔴= Not written)

Introduction 🟢
Elaborates why the world needs one more book and what was the motivation behind writing it.

Part I: Reliability

1. Why bother with Service Levels? 🟢

2. How reliability is perceived 🟠
   We show some examples and then introduce a simple workshop format that can be used to identify the reliability perception and set SLOs for various team topologies.

3. Degradation vs disruption 🟢
   What's the difference between service degradation, service disruption, and service outage and why does it matter?

4. Risk assessment 🟠

   We need to build a language about how we talk about risk. This chapter builds that jargon to an extent that’s required to understand the book.

5. What is a service? 🟢
   Before talking about SLI and SLO, we need to clarify what a service is.

6. Service Level Terminology 🟢
   Clarify the definition of Service, Provider, Consumer, Stakeholder, Owner, Failure, Metric, Consumption, and product in the context of reliability engineering

7. The Service Level workshop 🟠

   This is the workshop I run professionally for different teams to identify the risks, key metrics, and a reasonable objective.

8. Service Level Document 🟠
   This is a tool to communicate expectations between teams and stakeholders. We talk about a format that is based on what Google suggests and builds on top of it to create a contract.

9. Crafting a useful model 🟠
   Service Levels are simplified models to map system behaviour to user behaviour. What defines a good model and how to test it?

10. Availability 🟠

    This is one of the most common types of metrics. We discuss 4 types of availability metrics and when to use which.

11. Latency 🟠

    Another popular type of service level indicator.

12. Error rate 🟠

    Or more properly: Success Rate is a common type of service level indicator

13. Reliability of data systems 🟠
    Different metrics to measure reliability of data and stateful services

14. Reliability of AI systems 🟠
    Various metrics that is used to measure reliability of AI models

15. Introduction Service Level Indicator (SLI) 🟢
    Delve into SLI with examples, decisions to be made and what is not a SLI.

16. What is time-based vs event-based SLI? 🟢

    One of the first decisions about SLI is whether it is time-based, or event-based.

17. What is “valid” in SLI and why not use “total”? 🟢

    The denominator in the SLI formula is often mistaken for "total". It's a huge miss if we don't use it to scope our optimization effort and clarify ownership.

18. What is the definition of “good” in SLI? 🟢

    How do we define the good portion of time or events?

19. SLI vs KPI 🟢
    How do the two frameworks compare in terms of purpose, measurement, consumption, and action?

20. Where to measure matters. Why do you get different readings? 🟢

    Where we measure is sometimes more important than the actual value we measure.

21. Service Level Objective (SLO) 🟢
    Deep dive into SLOs, the decision to be made, and clearing common misconceptions around them.

22. Service Level Status (SLS) 🟢
    SLS shows the status of the SLO commitment according to the SLI metric towards the service consumer at any given point in time.

23. Rule of 10x per 9 🟢
    For every 9 you add to SLO, you’re making the system 10x more reliable but also 10x more expensive.

24. Multi-tiered SLOs 🟢
    Using different parameters, we can set multiple objectives for the same indicator.

25. Compliance period (AKA “window”) 🟢
    Different types of compliance periods and how to pick one that suits your product?

26. Alerting on SLOs 🟠
    What’s the point of measuring if we don’t commit to keep it in good shape? This chapter examines how we convert our SLO commitments to alerts. We also use the Service Level Calculator

27. On-call 🔴

    The practicalities (payment, off time, contract, etc.), toil reduction, MTT* metrics, 

28. Service Level Agreements (SLA) 🟢

    Defines SLA and clarifies its difference with SLO. Examine S3's SLA as one of the oldest and most reliable services on the internet.

29. SLA vs SLO 🟢
    Clarifying one of the most common confusions about the two terms with examples, illustrations and quotes from Google SRE books.

30. The legal commitment and punishment models 🟠

31. SLA tricks 🟠
    Various engineering and legal techniques that you can use to your advantage to reduce the consequences of system failure.

32. How to set the level for SLA 🟠
    What is reasonable to commit based on composite system reliability.

Part 2: Engineering

This part goes through different architectural patterns to improve software resilience.

6. Composite system reliability 🟢

   How to calculate the reliability metrics of a complex system that’s composed of multiple sub-systems?

7. Percentiles 🟢
   Why percentiles are important for understanding and optimizing system behavior, when to use them and how.

8. Non-functional requirements (NFR) 🟢
   What are functional and non-functional requirements and where does reliability fit?

9. Architectural patterns

   1. Fallback 🟢

   2. Failover 🟢

   3. Circuit Breaker 🟠

   4. Async, Queues and Dead letter 🔴

   5. Bulkhead 🔴

   6. Retry 🔴

   7. Rate-limit/Throttle 🔴

   8. Cache 🟠

   9. CDN 🟢

   10. Serverless 🔴

10. Deployment patterns 🟠

    1. Blue Green 🟠

    2. Canary 🟠

    3. Dark Launch 🟠

    4. Feature flags 🟠

    5. Break glass 🟠

11. Tooling and requirements 🔴

    1. Observability: Metrics, Logs, Traces 🔴

    2. Monitors and alerts 🔴

    3. Incident handling process 🔴

    4. Postmortems and fire drills 🔴

    5. Service Level Calculator 🟢
       Introducing the service level calculator as a learning tool and utility in the hands of engineers and business stakeholders to set reasonable SLOs and understand the alerting rules.

Part III: Mindset

Reliability Engineering doesn’t happen in vacuum. It needs a special mindset and mental model both at the engineering and leadership level.

1. Service level adoption obstacles 🟢
   Why is one of the greatest ways to measure and improve reliability so heavily underused and what are the most common ways companies fail to implement them correctly?

2. Service Level adoption steps 🟠

   Not every team needs service levels, not everyone is receptive to the idea. This chapter is about defining a maturity model that sets the right expectations and lays out a framework for gradually evolving into full ownership.

3. Getting the buy in from leadership🟠
   Every organization is different, but most leaders care about results. How can we frame service levels with a clear outcome to get the time budget to shift the way of working?
   What problems can service levels solve and what problems don’t they solve? Why should a particular organization use it and why doesn’t it make sense for another? How does operating with Service Levels change the way of working: accountability, transparency, and data-driven optimization. How to communicate complex technical topics to an audience who is not necessarily technical but needs to assess the tool.

4. You build it you own it 🟢

   Motivates and defines true ownership trio and its 3 elements: knowledge, mandate and responsibility.

5. 6 archetypes of broken Ownership 🟢

   A follow up discussing various broken ownerships.

6. Team ownership 🟢

   What exactly does the ownership trio look like at a team level? You should never be responsible for what you don’t control, and you should take control of what you are held responsible for.

7. Individual ownership 🔴

   How do the knowledge, mandate and responsibility empower an individual in their career

8. Organizational ownership 🟢

   How can an organization set itself up for true ownership? Engineering leadership is responsible for the reliability of engineering artifacts.

9. 4 archetypes of SRE 🟢
   2 decades after the term SRE was coined, there are many flavors of SRE out in the wild and people who carry the title have a diverse range of skills.

10. Engineering Leadership Ownership 🟢
    Engineering leaders have an overlap with technical leaders. This article tries to clarify their responsibilities with examples and prevent one of the most common anti-patterns for tech leads.

11. Platform team topologies 🟢

    We use the washing machine analogy to nail the pros and cons of owning your washing machine at home or relying on the central laundry room.
    We also point to relevant concepts from Team Topologies (the book) and offer slightly different perspectives on how to control cognitive load.

12. Amortization plans for Tech debt 🟠

    We also discuss how to take care of tech debt and make stronger architectural decisions without the burden of a technical committee.

Writing a book is a tremendous task. Between August 2023 to July 2024, I have spent more than 500 hours to draft, edit, research, and illustrate the book and it is not yet finished.

My monetization strategy is to give away most content for free. I pull these hours from my private time, vacation days and weekends. Since April 2024 I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public.

You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections on all posts.

Right now, you can get 20% off via this link. You can also invite your friends to gain free access. If none of that works for you, please share this book in your circles to help others discover it. Thanks in advance. 🙏

Share

Loosely speaking:

• Service degradation is when the quality of the service drops.

• If the service entirely stops, it’s a service disruption.

• If the disruption takes too long, it’s a service outage.

That definition lacks nuances. Let’s dig into four aspects of service levels to distinguish the difference with some examples and illustrations.

Subscribe now

1. Availability

A service usually has multiple capabilities. For example:

Amazon Web Services include many different services like databases, runtimes, queues, identity and access management, etc. It also has some other capabilities like documentation, and pages like terms of service, which are essential (and some of them are legally required) but don’t have any purpose without the core services of AWS. There would be no documentation if there were no AWS services!

Core capabilities are the ones that drive the business. For GitHub, it’s the git servers, for a car, it’s the ability to drive (and brakes), and for a dish, it’s edibility! The core features define the service.

In case of service degradation, the core capabilities are still available although with poor quality of service.

For example, in the early days of ChatGPT, the “AI” failed to carry a conversation and would randomly break. This is a service degradation because although it hurts the user experience, the site is still usable for some (or if the same user retries).

The emphasis here is on core capabilities that define the service. If only less important capabilities are affected (e.g. AWS terms of service pages are not accessible), it’s still considered a service degradation because the main service is not disrupted.

Graceful degradation patterns (like fallback) may temporarily switch to using an alternative backup strategy for business continuity.

When the key capabilities are completely stopped, there’s service disruption.

For example, when AWS S3 was disrupted back in 2017, all GET, LIST, PUT and DELETE requests were failing which rendered this service completely useless. This also impacted other AWS services which internally depended on S3.

2. Service Level

If you are measuring reliability using Service Level Indicators (e.g. latency, error rate, data freshness, etc.):

• Service degradation shows itself as burning error budget. The Service Level Status (SLS) drops below the Service Level Objective (SLO).

• Service disruption shows itself as an error budget burn out: burning the error budget too fast for a longer period of time.

For example, in 2018 GitHub experienced a service degradation with some internal systems leading to displaying out of date or inconsistent information to users for a bit over 24 hours. If they had a SLI for data correctness, they would be seeing a rapid burnout that took 24 hours to recover.

However, since their main git service capabilities were untouched, it wasn’t technically a service disruption.

3. Blast radius

One of the defining factors to distinguish between service degradation and disruption is to ask:

Who is affected by the issue?

Is it just some users? Or all users? Is it the paid customers? Or the free customers?

If only some users are impacted, it might be a case of service degradation. But how many is too many? The answer depends on the consequences and the business appetite to take that risk.

For example, many cloud providers use compartmentalization using separate data centers (regions) to decouple risks and localize it.

If Microsoft Azure goes down in South India (Chennai) region, it doesn’t impact services that are running in other regions.

But if there’s a big Cricket match going on, it can potentially lead to massive free advertisement for AWS —not the positive kind! 😉

Another example is when an issue only affects a subgroup of customers:

• Users on Android, Web, or iOS only

• Users with uncommon configurations

• Users on an older version of the software

• Free tier users

• Users in a specific city, country, or region

4. Consequences

Service degradation usually doesn’t have a big impact on the business. Some customers may be unhappy but not enough to leave. The business may lose some potential customers but not enough to go bankrupt. There may be a loss of reputation, some frustration on social media, and angry customer support calls but that’s usually about it. From the business standpoint, the consequences are negligible.

On the other hand, service disruptions have more severe consequences that go beyond “business as usual” or “cost of running the business”.

One of the main points of service levels is to shift the perfectionist mindset to acknowledging that complex systems fail all the time, so let’s identify what defines a failure (which shapes SLI) and what is a reasonable expectation (which shapes SLO) and at what cost (see the rule of 10x/9)

Service Outage

So far, we haven’t talked about the outage. That’s because outage and disruption are used interchangeably.

However, the word outage typically carries a more dramatic meaning and is used for longer and more serious disruptions with dire consequences. For example: SLA penalties, loss (sales, revenue, users, etc.), or legal consequences (accidents, people getting hurt, etc.).

How long and how serious? It depends on the type of product, customer expectation, and risk appetite:

• If your home electricity is cut for 12+ hours, you may call that an outage. Here in Sweden for example, you’re entitled to compensation if that happens. In this cold corner of the map, many homes depend on electricity for anything from cooking and lights to warming the home. 12 hours is a very long time and can potentially put some lives at risk.

• The 2011 PlayStation outage lasted 23 days and has its own Wikipedia page!

• Back in 2022, covered the longest Atlassian outage of all time with an interesting insight: “The biggest impact of this outage is not in lost revenue: it is reputational damage and might hurt longer-term Cloud sales efforts for new customers. […] The irony of the outage is how Atlassian was pushing customers to its Cloud offering, highlighting reliability as a selling point.” Using the word outage (with all its dramatic connotations) is completely justified here.

For practical purposes when your dependency goes down, you can use “outage”. On the other hand, when your service goes down, you can use “service disruption” or even wrongfully call it a “degradation” to downplay the impact! 😄It just sounds fancier and more technical.

Conclusion

Reliability engineering is about preventing threats from becoming degradations and degradations from becoming disruption.

So, is degradation better than disruption? I have a question for you:

Think about an online retail store. Which one has a more negative business impact?

A degradation which leads to showing the wrong prices to the customers?

Or a disruption where the side is not accessible at all?

If you guessed degradation, you guessed right. While intuitively we may think that a degraded experience is better than total outage, showing wrong prices may literally cost the business more than the site being down and losing potential customers —who may in some cases come back after the side is recovered.

• If customers buy the products below the price, the business loses money

• If the site is down, customers cannot buy the products for the wrong price

Sometimes, it is better to disrupt a service than degrading it. The nuances are in the context, business model, risk tolerance, etc.

Before we close, I just wanted to highlight one point: One service’s outage may lead to another service’s degradation. If your team is providing services to another team in a loosely coupled matter:

• When your service is disrupted, all your services are disrupted

• but for the services that your consumer provides, your disruption might be just a degradation they have to handle (if they’ve done their homework).

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

For example, the latency metric values for an API endpoint may look like this:

metric_data = [
{ epoch: 1716898141, 200, tags: [ “api”, “GET” ] },
{ epoch: 1716898142, 212, tags: [ “api”, “GET” ] },
{ epoch: 1716898143, 102, tags: [ “api”, “GET” ] },
{ epoch: 1716898144, 290, tags: [ “api”, “GET” ] },
{ epoch: 1716898145, 180, tags: [ “api”, “GET” ] },
{ epoch: 1716898146, 3000, tags: [ “api”, “GET” ] },
{ epoch: 1716898147, 153, tags: [ “api”, “GET” ] },
…
]

Let’s ignore the timing and tags for now to focus on the values:

metric_values = [ 200, 212, 102, 290, 180, 3000, 153, … ]

In this dataset, most latency values are in the range [100ms..200ms] but there’s an outlier there: 3000ms. What happened there? Before we look into that, let’s see if one of the most common tools can help: average.

The average of the whole dataset including the outlier is:

average([200, 212, 102, 290, 180, 3000, 153]) = 591ms

Without the outlier we have:

average([200, 212, 102, 290, 180, 153]) = 189.5ms

So, it is 591ms versus 189.5ms! Pretty large difference!

That is how average works: it hides those outliers and makes all data look equally bad.

Average often does not represent an actual data point.

For example, the average American family has 1.94 children! I know what one child looks like, but what exactly is 0.94 child? 👶And apparently all families have it! 😄Even the ones with no kids! 🤯

If we want to focus our optimization efforts on the outliers, we need a tool that’s better than average (no pun intended)! We want a tool that picks actual outlier data points.

Say hello to percentiles!

Percentiles

Percentile is the value above a percentage of all data points.

OK, that was a mouthful. 🤭Let’s say we have a metric with these values:

These are 1000 values for a metric that is mostly hovering around 11000 to 12000 but occasionally there are outliers which are significantly higher or lower.

The purpose of percentiles is to find outliers.

The idea is very simple and intuitive:

1. Sort the data

2. Pick the data at the index that is at p% of the number of data points

Let’s try that with the dataset above.

1. The sorted data looks like this:

2. The percentile P0, P1, P2, …, P100 look like this:

Do you spot the similarity? That’s because percentiles are directly deduced from the sorted data points.

Note: for consistency, in this article we only sort the data in ascending order, but the data may as well be sorted in descending order. The sorting order is not a requirement for percentiles.

Now we can write a small function that returns a given data point at a desired percentile:

The code is pretty simple. It just returns the array element at the nth position where n is at the p% position of the array length.

Example:

• If the array has 100 elements, P99 is the item at the 99th position. For languages that start the array from 0, that would be arr[98]

• If the array has 1000 elements, P99 is the 990th element or arr[989]

• If the array has 24 elements, P95 is the 23rd element or arr[22]. That’s because the maximum possible array index is 23 and 95%23 = 21.85. Math.ceil(21.85) = 22. Since this array is so short, even P99 is also at arr[22].

In simple terms, P99 is the value that shows up at the 99% position in a sorted array.

While P99, P95, or even P90 are most used, there’s no rule that excludes other numbers between 0-100.

For example, if you want to find unusually small numbers in a dataset, you can look for P1 or even P0.1.

P0.1 may look confusing, but it uses the same rules as before: find the data at the 0.1% of the sorted dataset.

Example:

• If the array has 100 elements, P0.1 is the item at the 2nd position. For languages that start the array from 0, that would be arr[1]

• If the array has 10000 elements, P0.1 is the 11th element or arr[10]. That’s because the maximum possible array index is 9999 and 0.1%9999 = 9.999. Math.ceil(9.999) = 10.

Other notable percentiles are:

• P10: Also known as 1st Decile

• P20: Also known as 2nd Decile

• P25: Also known as 1st Quartile

• P50: Also known as median or 2nd Quartile (this is not the average, but the item that is located exactly in the middle of the array)

• P75: Also known as 3rd Quartile

• P80: Also known as 8th Decile

P50 is an interesting one. Instead of average, it represents the data in the middle of a sorted array.

For example, although the average American has 1.89 children, the median family has either 0, 1, or 2 children. I couldn’t find the stats in a way that allowed me to calculate the median, but you get the point: percentile selects a data point from the array whereas average aggregates the data points.

You don’t have to care about decile and quartile. I just wrote about them for the sake of inclusivity. In practice we almost exclusively work with percentile.

For the exact dataset that was visualized in this page, we have the following stats:

If you want to play with different datasets, I’ve written a free open-source tool that helped me learn and visualize these concepts.

The interface is a bit crud at the moment, but I have big plans for integrating it to the Service Leval Calculator.

Note: using percentiles simplifies talking about data. For example, if you read:

12% of European households have 3 or more children —elfac.org

You know that if there was an array representing the number of children for each individual family in Europe:

• If it was sorted in the decreasing order, the family at the position P12, would have exactly 3 children

• If it was sorted in increasing order, the P88 family would have exactly 3 children.

When to use percentiles?

Percentile is a popular tool in the field of reliability engineering and performance optimization. It is a powerful tool to spot the outlier data points without diluting the story that a dataset is capable of telling.

Percentile is especially useful for datasets that are not evenly distributed —most data!

Evenly distributed what? —you may ask!

First let’s look at some evenly distributed data where each value is as likely to show up:

This is how random number generators work. In fact, this data was created using Math.random() in JavaScript as we’ll see later.

If we sort that data it we get this:

As you can see, there are no anomalies. If we do the math, we can see that:

• Mean (also known as average) is 10279

• Median (also known as P50 percentile) is 10453

Not an enormous difference considering that the range of values is anywhere from 0 to 20000. And that’s expected from such an evenly distributed dataset.

But the story is different when there are outliers.

For example, if a small fraction of numbers are unusually high, we get something like this:

Sorted, we get this:

• Mean (average): 2360

• Median (P50): 2106

• P95: 3969

• P99: 18031. This means 99% of values are below 18031

This type of data distribution is also known as long tail.

Another example where most values are high, but a few are very low: