You're thinking about architecture, tech bets, team direction, influencing product roadmaps, and ensuring your technical initiatives deliver real strategic value for the business.

We dive into Wardley Maps and Pace Layering, not as abstract theories, but as practical tools that can help you answer questions like:

• What perspectives do these tools add to connect technical landscape to the business model? How can we reach clarity?

• How can we make smarter bets on our technological investments?

• How to make compelling cases for architectural changes, refactoring, or technology choices?

• Where should our teams be focusing their bandwidth as our tech landscape evolves?

• How to distribute and balance the budget and headcount around foundations, products, and features?

• What's a lightweight way to start applying these strategic ideas today?

• How can these lenses fill the communication gap between C-suit and the edge nodes of the org tree to speak the same language and pull in the same direction?

And if you’re a senior engineer, you may still find these tools valuable. Ever feel like you're just churning out code without seeing the bigger picture? Or wondered why some parts of your system are ancient monoliths while others are shiny microservices? How about when your great ideas getting blocked or go under-funded because no one up there could understand the value of the problem you’re trying to solve?

Subscribe now

Note: I have used Gemini Advanced over the course of 7 hours combined with manual editing, annotation, illustration and adding resources to create this post. I also had to rewrite a few entire sections. You can dismiss it as “if AI knows this stuff, I don’t need to read it”, and that’s respectable. Sure, AI can generate content but it cannot understand it for you. As a senior technical leader myself, I found this exercise insightful, and I believe the final result is richer. I would read a post like this if it existed (and believe me I did my research). 😊

Wardley Maps: the GPS for Tech Landscape

Imagine you're navigating a complex open world video game like GTA. You wouldn't just wander aimlessly, right? You'd want a map.

That's kind of what a Wardley Map is for your business and tech strategy.

Created by Simon Wardley, it's a visual way to understand what you're building, why you're building it, and where the real challenges and opportunities lie.

A 2D User Centric Map

Forget abstract business goals for a second. A Wardley Map always starts with your users and their needs. This is your "North Star" or the main quest objective. Everything else on the map exists to support the user needs.

Y-Axis: Value Chain - How Visible Is It to the user?

Think of this as your dependency tree or tech stack, laid out vertically. Stuff at the top is what the user directly sees and interacts with (e.g., the UI of your web app).

Stuff at the bottom is the plumbing – the databases, servers, power supply – less visible but totally essential.

This shows how different components rely on each other to deliver value. If your database (bottom) is slow, the user experience (top) suffers.

X-Axis: Evolution - How Mature Is This Tech?

This is where it gets really interesting. Components aren't static; they evolve over time, driven by good old supply and demand. Think of it like tech maturity levels:

1. Genesis (The Wild West): This is brand new, experimental stuff. Think cutting-edge algorithms, the first version of a crazy idea. It's uncertain, likely to break, but potentially revolutionary. Dev parallel: That weekend hackathon project held together with duct tape and dreams.

2. Custom-Built/Bespoke (Your Special Sauce): You're building this yourself because nothing off-the-shelf quite fits. It's unique to your needs, maybe a bit bespoke and quirky. Dev parallel: Your company's proprietary rendering engine, bespoke platform or that complex internal API.

3. Product (+Rental): Someone else has figured this out and is selling it. This is sometimes called COTS (for Commercial off-the-shelf). Think commercial software, SaaS products (like Stripe for payments, Okta for authentication or Akamai for CDN), or well-established libraries (like React for front-end rendering). You buy or rent it. Dev parallel: Using a popular frontend framework, a cloud database service, or an authentication provider.

4. Commodity (+Utility): This stuff is so standard and well-understood, it's like a utility (like electricity). It's expected, reliable, and you pay for what you use. The vendors often have a thin margin since this space is very competitive. Dev parallel: AWS S3, basic compute instances, open-source databases, or even electricity itself.

Here's a quick comparison of these evolutionary stages to help you assess your own tech components:

Wardley Mapping Example: A news site

I used this example from several years ago because I know it intimately well and it doesn’t leak any info from my current company.

Putting the main parts of the architecture on a Wardley Map, we get this:

A few notes:

• Adding everything to the map quickly clutters it to the point it is not useful. This is a high level view showing some of the pieces that may be required in a discussion about why we should build our own CMS for example.

• For product and community pieces, I’ve removed the brand names.

• We built our own headless CMS because we needed a powerful tool for storytelling that wasn’t matched by anything on the market.

• Although the Cloud Infra was based on a purchased product, we were only using a subset of its features that was commodity (a managed Kubernetes cluster)

Using Wardley Maps to for tech decisions

• Spotting red flags: See that critical UI feature (high on Y-axis) relying on some janky, bespoke solution (in Genesis stage) when a solid "Product" solution exists? That can be a red flag for tech debt or a missed opportunity.

• Giving a direction to budgeting and headcount: On the other hand if nothing in the market fits the bill, you may deliberately choose to invest in that area to turn it to a proper Bespoke solution.

• Finding opportunities for migration and cost reduction: It is completely normal for a company to build something that doesn’t exist. But as more companies run into the same problem, some startups may create products that solve those problems better and at a lower price point. Reassessing the tech landscape can pop those opportunities where a migration (tech bet) can save costs on the long run. Security is hard to get right. Instead of building a Bespoke authentication solution, you’re typically better off using a battle-tested “Product” like Okta or Keycloak.

• Setting SLOs: Service Level Objective sets the reliability target for a system which in turn sets the error budget controling the pace of change. Generally, components early in their evolution stage can use lower SLO . Higher reliability commitments create frictions against the experimentation and flexibility that these solutions need to evolve rapidly. On the other hand, solutions that are Productionalized or Commodity need to be held accountable for higher SLOs. In case of 3rd party vendors, these SLOs are often part of a SLA that sets clear legal or financial consequences for the breach.

• Identifying spin-off opportunities: have you created an elegant Bespoke solution to solve your company’s problems? Do you know if other companies have the exact same problem and there’s nothing on the market to address it? You may be sitting on a gold-mine to spin-off a new business model and monetize that solution by turning it to a Product that can open a new revenue channel for your company.

• Communicating with Non-Techies: A visual map is a great way to explain complex tech dependencies and strategic choices to CPO, CTO, or even CFO (when the total cost of ownership is questions). Wardley Mapping is particularly useful because it surfaces the value of the invisible components in the tech landscape.

Beyond these basics, Wardley Maps has a few power-ups that allows it to deliver value at scale:

• Climatic Patterns: These are like the "rules of the game" – predictable forces like "everything evolves" or "success breeds inertia" (meaning, sometimes your past wins make you slow to adapt).

• Doctrine: Think "best practices" or SOP (Standard Operating Procedures) that help you play the game better, like "know your users" or "use appropriate methods" (Agile for new stuff, Six Sigma for optimizing commodities).

• Gameplay: These are your "special moves" or "power-ups" – specific actions you can take to change the game, like open-sourcing a component to speed up its move to commodity and mess with competitors.

Now, let's switch gears to Pace Layering for another map and perspective.

Pace Layering: Mapping the speed of evolution

While Wardley Mapping can help set a direction for tech evolution, Pace Layering helps make sense of decoupled architectural layers. It can be used as a tool for deliberate allocation of resources (budgeting, headcount), governance, reliability, compliance, and stability.

The concept of Pace Layering, or "Shearing Layers," was originally introduced by Stewart Brand in his work on how buildings evolve. He observed that different parts of a building change at different rates. His key insight was that because these layers change at different speeds, they "shear" past each other. This loose coupling between layers is crucial for resilience and adaptability.

…in the early 1970s, the English architect Frank Duffy wrote, “A building properly conceived is several layers of longevity of built components.” He identified four layers in commercial buildings:
- Shell: lasts maybe 50 years
- Services: swapped out every 15 years or so
- Scenery: interior walls, etc. move every 5 to 7 years
- Set: furniture, moving sometimes monthly.

Previously we have covered why using building and construction metaphors in the world of software leads to fallacies but stay with me, especially on the loose coupling between these layers.

In the tech world, you might hear of a specific three-layer model for applications:

1. Systems of Innovation (SOI) - The Skunkworks (Fastest Layer):

   • What they are: New, experimental things (e.g., hackathon projects, prototypes).

   • Rate of Change: Fast.

   • Dev Mindset: Rapid iteration, MVPs, learning quickly.

   • Dev parallel: AI model integrations, new app feature prototypes, A/B tests.

2. Systems of Differentiation (SOD) - Your Competitive Edge (Medium Layer):

   • What they are: Applications enabling unique company processes (e.g., custom pricing engine, unique CRM integrations).

   • Rate of Change: Medium.

   • Dev Mindset: Agility, delivering business value, configurability.

   • Dev parallel: Your company's unique pricing engine, a custom CRM integration, recommendation algorithms.

3. Systems of Record (SOR) - The Bedrock (Slowest Layer):

   • What they are: Core, foundational systems managing critical data (e.g., core financials, HR databases).

   • Rate of Change: Very slow.

   • Dev Mindset: Stability, reliability, security.

   • Dev parallel: Ancient, critical database schema; the mainframe; core network infrastructure.

Pro-Tip: Think in Layers, Start with Business Capabilities When applying this, don't just slap labels on your existing apps. Think about what your business actually does (its capabilities) and how quickly those capabilities need to change.

Then, see which systems support them and how they fit into these different speed layers. This helps ensure your tech strategy truly supports the business strategy.

Here's a quick table to compare these layers:

You may have noticed a similarity between Pace Layering (1D) and the X-axis of the Wardley Maps. We’ll shortly cover this similarity and when to use each tool.

Pace Layering Example: That same news site

I often say business of news is the business of speed. The same rush that goes into writing about breaking news is omnipresent on the development floor where the latest tech is adopted. Or at least that’s my experience.

Before slapping labels on different parts of the architecture, we need to identify the business capabilities and the pace of their change:

• To tell compelling stories, the front-end was constantly changing in tight collaboration with UX and journalists. A bug in the front-end wouldn’t last too long due to the sheer number of eyeballs that would see it. This product was optimized to have several releases per day (including cache invalidation that was needed to ship the latest code).

• The Business Logic layer was more static but it still needed to change from time to time to support new features that were added in the front-end. This layer could potentially change at the same speed but often it was released every 2-3 days. Significant changes were pushed every few months in close coordination with its main consumer: the front-end.

• The CMS hardly changed and most of the effort was put on maintenance, refactoring, and scalability. The shape of the API hardly changed. In fact in my 5 years working with these systems, I had probably seen a handful of changes only one of which was a breaking change.

Note that we didn’t talk about Products or Community components that is surfaced in Wardley Maps.

In this example, front-end is the SOI, Business Logic layer was SOD, and the CMS could be considered SOR.

Using Pace Layering to manage resources and expectations

Why Should Developers Care about Pace Layering? (And How Does It Impact Your Work?

• Identifies best places for implementing loose coupling: Helps categorize old systems and encourages designing for "shearing": loose coupling for adaptability and identifying architecture “seams”.

• Clarifies requirements: different layers have different security, compliance, reliability and other NFRs (non-functional requirements).

• Justifies Different Dev Practices: Explains why your SOI team can CI/CD daily, while the SOR team has quarterly releases with lots of ceremonies, manual controls, and process friction.

• Guides resource and budget allocation: Justifying work on an SOR isn't about new features, but about ensuring resilience, security, scalability, and reducing operational risk. A decaying SOR puts the whole business at risk, no matter how fast your SOIs are. Use this to argue for that crucial refactor or tech debt pay-down.

• Manages Expectations: Sets clear expectations for stability vs. speed based on the pace of change: Innovation vs Differentiation vs Record.

Pace layering for SLI and SLO

Pace layering is particularly useful for Service Levels. Change is the number one enemy of reliability: any time you change a system, its likelihood of breaking spikes up.

SLI identifies how we measure reliability and SLO sets expectations. The unsung hero of the Service Level model is the error budget: it allows the teams to take risks (changing the system) in a controlled manner.

In practice, this means Pace Layering can set different levels of SLO for different systems and make a deliberate decision about the reliability vs speed tradeoff:

• SOI (Systems of Innovation): could use a lower SLO (e.g. 95%) to support a faster rate of change.

• SOD (Systems of Differentiation): could use medium SLO requirements (e.g. 99.6%) to balance reliability and speed.

• SOR (Systems of Record): could set high SLO expectations (e.g. 99.99%) to guarantee a stable base for business-critical services.

Wardley vs Pace Layering: different perceptions

So, we've looked at Wardley Maps for understanding the competitive landscape and Pace Layering for decoupling our internal systems by their speed of change.

You might be thinking, "Hmm, these sound a bit similar, especially Pace Layering and the Wardley evolution axis. Are they just different words for the same ideas?" That's a great question, and it's a common point of reflection!

They definitely share some DNA because both are tools in the technical leadership and architectural toolbox. They both deal with how things change and mature over time.

That’s where the similarities end.

Pace Layering (SOR, SOD, SOI) feels somewhat one-dimensional, focusing on the rate of change. The X-axis of a Wardley Map also describes a form of evolution (Genesis to Commodity). So, what's the real difference?

• Driving Force & Scope:

  • Wardley's Evolution (X-axis) is about the market-driven commoditization of any component in your value chain. It's pushed by supply and demand competition (market forces). Everything on a map, from a tiny piece of code to a whole platform, is subject to this evolutionary pressure. It helps you understand where a component sits in its lifecycle relative to the market.

  • Pace Layering is about the business-dictated change cadence for an entire system or application based on its role in your specific organization and the business capability requirements. An SOR changes slowly because the business needs that stability, not necessarily because all its underlying components are ancient.

• Focus of Analysis:

  • Wardley Maps: Analyze the entire external competitive landscape and your position within it. The evolution axis is one part of a two-dimensional map (the other being the value chain/visibility). The goal is situational awareness to make strategic plays in the market.

  • Pace Layering: Analyzes your internal IT portfolio and how to govern it. It's about managing internal resources and development approaches effectively based on different required speeds of change.

• What They Describe:

  • Wardley's Evolution: Describes the maturity and ubiquity of a component. Is it novel and rare (Genesis), or standard and widespread (Commodity)?

  • Pace Layering: Describes the required operational tempo and governance model for a system. Does this system need to be rock-solid and change rarely (SOR), or does it need to be experimental and change often (SOI)?

Wardley Maps + Pace Layers: gaining perspective

Imagine you have a System of Record (SOR), like your core accounting system. In Pace Layering terms, it changes very slowly. On a Wardley Map, this accounting system itself might be considered a "Product" or even "Commodity" if you bought it off-the-shelf. The components it uses (like the database it runs on, or the programming language it was written in) would also have their own positions on the Wardley evolution axis. The database might be a "Commodity," while a specific custom integration module within that SOR might be "Custom-Built."

Conversely, a System of Innovation (SOI) is, by its Pace Layering definition, fast-changing and experimental. The components it's built from are almost certainly in the "Genesis" or "Custom-Built" stages on a Wardley Map.

This is where it gets really powerful:

• Wardley Maps give you the "lay of the land" – the strategic "why" and "what." They help you see opportunities, competitor moves, and user needs.

• Pace Layering then helps you organize your internal response – the "how." A Wardley insight might identify a new SOI, which then gets agile funding and rapid iteration.

By using both, you get a richer picture for smarter decisions on market strategy (Wardley) and internal system design/governance (Pace Layering).

Other lenses worthy of mentioning

Here are a couple of other widely recognized frameworks that help in designing and organizing complex software systems and the teams that build them:

Domain-Driven Design (DDD) is an approach to software development that emphasizes a deep understanding of the business domain. It focuses on creating a rich "domain model" that reflects the business's complexities and uses a "Ubiquitous Language" shared by developers and domain experts.

• Why it's relevant here: DDD helps you understand the "what" and "why" of the systems you're building at a very granular level. This deep domain understanding is invaluable when creating Wardley Maps (what are the true user needs and core components?) and when classifying systems for Pace Layering (what makes a system truly differentiating for our business?). Concepts like "Bounded Contexts" from DDD naturally lead to more modular systems that align well with the idea of "shearing layers" in pace layering

Team Topologies is a model for organizing business and technology teams for fast flow and adaptability. It proposes four fundamental team types (Stream-aligned, Enabling, Complicated Subsystem, and Platform) and three core interaction modes. (learn more here)

• Why it's relevant here: Once Pace Layering helps you identify systems that need to change at different speeds, Team Topologies can provide concrete patterns for how to structure your engineering organization to effectively support them. For example, you might have Platform teams providing stable SOR-like services, while Stream-aligned teams rapidly iterate on SODs and SOIs, consuming those platform services. It helps answer the "How can these frameworks improve our team's structure or mission clarity?" question from our intro.

City Maps (also known as IT Landscape Diagrams): are visual representations, that provide a holistic overview of an organization's IT landscape using city metaphors. Think of them like a city plan showing different zones (business domains), major infrastructure (core platforms), key buildings (applications), and roads (integrations).

• Why it's relevant here: While Wardley Maps dive deep into the dynamics of specific value chains and Pace Layering helps categorize systems by change rate, these broader EA maps provide the "big picture" context. They are particularly useful when communicating tech to technophobic leaders who for some reason have a say in tech 😄A Wardley Map might analyze a strategic new "road" or "utility" for this city, and Pace Layering could define the building codes (governance) for different "zones" (SOR, SOD, SOI) within it.

These aren't replacements for Wardley or Pace Layering, but rather complementary lenses. Understanding the business domain deeply (DDD) and organizing your teams effectively for flow (Team Topologies) can amplify the strategic insights you gain from mapping your landscape and layering your systems.

Lightweight First Steps

Feeling inspired but not sure where to begin?

1. Your First Mini-Wardley Map: Pick a single user story or a small feature your team is working on. Try to map out its immediate value chain: What does the user need? What few components directly deliver that? Where do those components sit on the evolution axis? Are there any opportunities to move a component in the x-axis while motivating it by business objectives? Don't aim for perfection; aim for a 15-minute conversation starter with a colleague.

2. Pace Layering Your Next Feature: When discussing a new feature or epic, ask yourself: "What's the expected rate of change for this? Is this an experiment (SOI), a core differentiator (SOD), or something that needs to be super stable (SOR) from day one?" This simple question can spark valuable discussions about expectations and approach. You can then think about NFRs like reliability, security, and compliance.

These frameworks give you a language and a lens to think more strategically about the software you build every day. They help bridge the gap between writing lines of code and delivering real, impactful, and sustainable value.

What do you think? Have you encountered these ideas of tech evolution or different speeds of change in your products? How might you use these tools? Let's discuss in the comments!

My monetization strategy is to give away most content for free. These posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me, you can consider a paid subscription. If you want to save money, you can get 20% off via this link. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

But what tool do I use instead?

Short answer: ephemeral task force.

Ephemeral Task Force (ETF) is a selected group of people with cross-functional knowledge, mandate, and responsibility who are assembled for a specific delivery with a clear end in mind. The group is dismantle after the objective is accomplished.

This article convers:

1. Purpose of ephemeral taskforce and when it’s the right tool for technical leadership.

2. Comparison to technical committee.

3. Examples of how I use it to deliver impact across a large organization.

4. Pro tips: common pitfalls and remedies (the only paywalled section)

Subscribe now

🤖🚫 Note: NO generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs. (why?)

Purpose of ephemeral taskforce

Ephemeral taskforce is a tool in the senior technical leadership toolbox.

It is suitable for initiatives that span across the org and require more bandwidth than a single person can deliver —hence the need to collaborate.

If it one would do, ETF is an overkill. So is the technical committee.

Unlike a typical leader/follower setup (e.g. one Staff Engineer leading a migration across multiple teams), ETF involves multiple leaders, each owning a piece of the execution. In this context, ownership refers to knowledge, mandate, and responsibility.

ETF is a temporary assembly of technical leaders and engineers. For example:

• Implementing new regulations like European Accessibility Act (EAA) or GDPR compliance concern across the product landscape

• Implementing a new business model that requires significant mutation of existing architecture

These cases often require a higher execution bandwidth and broader knowledge than one person can deliver in an acceptable time.

For example, it doesn’t help to have a solid understanding of GDPR, it is also important to know how user data is stored in the database, how it is transferred between services, how the payments are processed, logs are stored, UI is tracked, etc.

That’s too much knowledge.

Same is true about the mandate and responsibility where the required ownership is not consolidated in one person, but a group.

That’s why a selected group of individuals are assembled in a virtual team for a concrete mission.

If the initiative falls within the ownership boundaries of one team in the current org chart, there’s no reason to assemble a taskforce.

Similarly, if the nature of the delivery calls for an ongoing engagement without an end-date, then reorganization is a better tool than ETF. A thoughtful reorg can facilitate the initial delivery and continuous evolution.

To recap, ETF is composed of multiple people with the required ownership to deliver an impact in a set time frame.

Ideally the ephemeral taskforce is composed of:

• A TPM (technical program manager) and/or a Staff or Principal who owns the high level impact

• Individual contributors, SMEs (subject matter experts), and product managers across the org who:

  • Have the required mandate to make changes to systems that are owned by their home team

  • Possess the know-how and technical hard skills required to execute the tasks as the discovery process uncovers new aspects to the mission. Here, a can-do and positive attitude trumps having ninja-level skills in the most obvious technical aspect of the mission.

  • Relate to the problem being solved (maybe it’s a problem they have in their team) and feel the responsibility and motivation to solve that problem for their team and others

Typically a written strategy document can build clarity and alignment across this diverse group.

Comparison to technical committee

Let’s quickly go through the major flaws with technical committees (TC) and examine how ephemeral taskforce (ETF) addresses them:

Knowledge

❌ Technical committees tend to recycle the same brains for all sorts of decisions and initiatives.

✅ ETF bring the best people with the right skillsets and subject matter expertise to execute.

Mandate

❌ TC is typically a symptom of poor engineering management causing engineers to take the matter to their hand. TC acts as a self-appointed parliament. Formally, management has the mandate but due to lack of engineering literacy, they delegate to whoever runs the TC (we have covered this anti-pattern previously).

✅ Ephemeral taskforces are typically composed of a selected group of people by the engineering leadership with a clear mandate to execute a specific mission. The clear mission often justifies collaboration, so the formal mandate is less likely to be [ab]used.

Responsibility

❌ Technical committees typically turn to echo chambers with “us vs them” narratives. The lack of delivery and meaningful impact is often blamed on “others” who aren’t in the room.

✅ Ephemeral taskforces have the owners of the problem and solution with clear expectations and responsibility in the room. There are no others because everyone who is connected to the mission is either part of ETF or directly connected to it.

Productivity

❌ Technical committees tend to be larger and be composed of higher ranking engineers who meet weekly or biweekly indefinitely. This makes them too expensive for what they deliver because the delivery is often limited to manifestos and playbooks that aren’t connected to the reality of the engineers’ day-to-day work.

✅ Ephemeral taskforce is created out of necessity to execute at scale with a clear delivery in mind. ETF is often much smaller than TC due to razor sharp focus. The group dismantles after the objective is reached.

Story

A few years ago, I joined a company that suffered from poor delivery speed and unreliable technical solutions.

Each release was treated like a novel event worthy of rituals, manual testing, heavy process, gate keeping, and strict governance.

Despite that, the incidents were rampant, took too long to spot and resolve, and had an unreasonably high impact on business metrics.

This was a few years after DORA metrics were a hot topic. The leadership asked if I could help the org measure DORA metrics. Because it was considered “best practice”!

DORA stands for DevOps Research and Assessment. At the time, its focus was to measure and improve these metrics:

1. Deployment Frequency (DF): how often a team deploys code to prod.

2. Lead Time for Changes (LTC): how long it takes from commit to prod.

3. Change Failure Rate (CFR): ratio of deployments that break prod.

4. Mean Time to Recovery (MTTR): average time prod is broken.

Fun fact! DORA also stands for the new EU regulation called Digital Operational Resilience Act! 😄That one is concerned with digital resilience of financial entities to withstand, respond to, and recover from disruptions.

It is debatable whether DORA was the right tool, but I was too new to the org to push back with a reasonable alternative. I saw it as a learning opportunity to both get to know the org better and learn from implementing DORA at scale.

Soon after initial self-onboarding I realized that the task is much larger than I can deliver on my own in a timely manner.

It involved:

• Identifying all the moving parts. Not a single person knew the entire tech landscape. Sure, some of the big parts were known but no body knew how many services we have and where do they run. This is the type of problem that an IDP (internal developer portal) like Backstage or Compass solve but we had nothing! So I had to start by crowdsourcing that data in a Notion Database!

• Setting up a destination for the data, ideally with graphing feature. Observability tools are usually an excellent choice for this type of time-series. Unfortunately, the observability tooling landscape was fragmented and many services simply missed it altogether!

• Tweaking CI/CD pipelines and SCM (source code management) tools like Github, Bitbucket, Azure DevOps (yeah we had all of them!) to get the f**ing DORA metrics.

• Introducing the org to DORA metrics, their purpose and tune it to WIIFM (what’s in it for me) while simultaneously talking to leadership to control their urges to weaponize these metrics! I had talks, write-ups, a dedicated Slack channel and even video recordings to spread that knowledge but to this day I think there are some corners of the org that I failed to penetrate.

It was a big task and if I worked on it alone, by the time I was done, the whole thing would be rewritten in Gemxy (a hypothetical programming language from 2052)!

Fortunately, I had a good relationship with the director of platform engineering. He dedicated a few SREs to this task and I found a team who volunteered to be the guinea pig. We also outsourced part of the problem to Sleuth to extract metrics from various data sources like Github and Azure DevOps.

After the initial delivery, the Platform Engineering org took ownership of the DORA metric as one of the services they offer to the rest of the org going forward.

I moved on to something more interesting: reliability metrics. You see, since 2021, DORA has a fifth metric that doesn’t get as much attention as DF, CFR, LCT, and MTTR.

It’s our friend SLI!

If you’ve been following my work, you know that I’ve covered everything from SLI, SLO, and SLA to terms that I coined like SLS! Hell, my book is on that topic! This is how it all started!

But before starting with SLI, I had to fix one problem: software catalog!

Remember that crowdsourced Notion Database? Yeah, turns out crowdsourcing is error prone because:

• The data should be manually updated as the shape of the architecture changes and new services emerge while others are sunset.

• The data was impartial because I had to literally go to every corner of the organization, introduce the need, interrogate them, and document what I learned and ask them to complete it. That is, if they even felt comfortable with the transparency that this kind of initiative demanded.

• The Notion Database was collapsing under this level of load! I had a couple of sessions with their product managers and even met one of their engineers who was working on improving Notion Database performance. But they considered this way of using it as an edge case and wouldn’t optimize for it.

We had to move that data out of Notion.

We needed something better. I needed to systematically go through our components and move each team through a Kanban swim-lane from introduction to SLI/SLO, to measuring them using observability tooling, to tying their commitment (SLO) to alerts and on call.

It was around that time that some of my colleagues introduced Backstage to the mix. Backstage is an open source IDP (internal developer platform) and one of its key features is software catalog.

I started fetching the data programmatically from the Notion API to generate software-catalog.yaml files and push it as PRs to Github repos. Backstage would in turn take those files and show them in a clean directory.

While I’m coding the above, a developer showed up out of nowhere and offered to help. Turns out he saw what I’m up to in my open TODO tasklist (I know it’s almost 2 years I said I’ll write about it, but haven’t published yet 😄).

Subscribe now

Together, we migrated the entire system catalog database from Notion to Backstage. He was a competent coder. After the initial onboarding and giving him access to Notion and Backstage, my involvement was shrunk to writing the specifications for how our custom fields should be mapped to Backstage labels, tags, and annotations.

Our ETF was 2 people, and we migrated 400 components and systems. He also created a Slack bot so we could use automation to harass people at scale to double check the automatically generated software catalog entries.

The rest of my time went to massaging org leaders to adopting service levels while carrying SLO workshops team after team.

Another person showed interest in running those SLO workshops. Initially he took the back seat on a couple of workshops. Then we switched seats, and he ran the workshop while I was observing. After a while he ran workshops on his own.

Although the ETF was 3 people, the other two rarely interacted because they didn’t need to. One was coding the catalog while the other ran the workshops. I was involved in both.

Currently I’m running a larger ETF to roll out a unified observability stack across the company while establishing a unified on-call response process and tooling. I’m taking notes and will for sure publish the learnings.

Subscribe now

Let’s shift gears and look into some of the common pitfalls of ephemeral taskforces and how to tackle them but before that:

Pro tips: common pitfalls and mitigations

We’ll discuss the following pitfalls:

1. Picking the wrong people

2. Unclear delivery, Scope creep or “gold plating”

3. “Home team” sabotage the ETF priorities

4. Disbanding the ETF too late

5. more

The pro tip section is a token of appreciation for those of you who support my work. My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support me is to like, subscribe and share this post. If you really want to support me, you can consider a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. You can get 20% off via this link. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access. And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

Read more

1. Architect

2. Tech Lead

3. Solver

4. Right hand

I used those archetypes a lot when I was a new Staff Engineer. I have used it as a reference hundreds of times and shared it with tens of thousands of people.

These archetypes are great because:

• They frame a simple, memorable shared language for how high impact engineers operate

• They acknowledge that there are different paths to becoming a force multiplier

• They clearly indicate what areas to grow more —as we’ll elaborate soon.

But as I gained more experience over the years and across companies, I realized that the most impactful Staff Engineers I know, aren’t any of those archetypes. They’re all. Plus more!

What makes me respect them isn’t the archetypes but their ability to switch hats as the situation calls for it. They are all rounded great engineers and technical leaders.

This post elaborate why those archetypes are anti-patterns and how the most effective Staff Engineers work.

Subscribe now

🤖🚫 Note: No generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs. (why?)

The problem with archetypes

I have been following Will Larson’s work and highly respect his insight as someone who has catapulted my career into technical leadership. This is not a criticism of Will but rather an observation that was solidified with many examples over time.

First, what is an archetype? Below is an excerpt from the book:

The more folks I spoke with about the role of Staff-plus engineers at their company, the better their experiences began to cluster into four distinct patterns. […] In literature, recurring character patterns are called archetypes […] and the archetype term is helpful for labeling these frequent variants of Staff-plus engineers.

So archetype is roughly same as pattern, variant, or flavor in this context.

My counter-argument boils down to the title vs role distinction which we have covered before.

In a nutshell:

• Title defines responsibilities towards an outcome with narrow ownership. For example:

  • Front-end developer, develops front-end and doesn’t necessary touch backend or do QA or DevOps. Those concerns are handled by other titles.

  • Project Manager, manages the project and leaves the customers, strategy, empowerment and other aspects to other people with appropriate titles.

• Role frames around the impact and broader ownership. For example

  • Software engineer uses engineering disciplines and methods to create technical solutions. That may involve coding, architecture, testing, operation, etc.

  • Product Manager effectively acts as a mini-entrepreneur for a product. That may formulating hypothesis, market research, collaborating with design and engineering, and even marketing and budgeting if the situation calls for it.

So let’s go through the 4 archetypes and diagnose what exactly goes wrong when they are treated as informal titles.

1. The Architect anti-pattern

Architecture is just ONE of the concerns of good engineers and effective technical leaders. Other concerns include QA (quality assurance), operations (e.g. DevOps/SRE), security, database, etc.

We need well rounded engineers (generalists) who own knowledge, mandate, responsibility and that requires:

• Up-to-date and hands-on experience with the very artifacts that’s used to implement an system architecture.

• Full accountability of the decisions made as an architect. And being on-call is the minimum guarantee to manifest that accountability.

I’ve been in the industry more than 2 decades and am yet to meet someone who specializes on architecture but hadn’t turned into the dreaded Ivory Tower Architect (ITA).

In recent years the word architect has got a bad rap in the software industry (examples 1, 2, 3, 4, 5).

What do you think ITA’s did?

Renamed themselves to Staff, Principal and Distinguished engineers!!! 🤦‍♂️

The nature of the task stays the same: draw boxes and arrows, act as a tech-translator prolonging the undeserved career of high level engineering managers, and call the shots without accountability.

They changed the facade without upgrading the interior.

It works in the broken organizations with poor leadership who can’t tell the difference between confidence and competence. That’s also the equivalent of painting yourself to a corner career-wise.

The tech translator is objectively bad for the organization because shields leadership and engineers from each other.

Sometimes even, there’s a tech committee in place which itself acts as a tech translator for the tech translator!

This is not a joke! It’s a patch to a broken system. Engineering leaders need to understand engineering without the need for a tech translator. There’s no way around that. Not even AI! 🤣

For the tech translator, it’s career dead-end. You can only go to so many meetings before your technical axe is blunt and engineers learn to work around you. The practitioners tend to converge to mandate-heavy initiatives like governance and compliance —more on that in a minute— but are light in implementing automation.

If you choose that path, beware that your career choices are limited to equally broken organizations.

2. The Tech Lead anti-pattern

Previously we’ve elaborated that the main value proposition of the Staff+ role is to own a technical solution that spans across multiple teams.

If the scope is one team, Staff Engineer is an overkill. There are different schools of thoughts about the role of Staff Engineer.

• Some see it merely as a career level to justify the pay or years of experience

• Others, see the change in the job title from Senior Engineer to Staff Engineer as a signal for a change in job description and impact.

I’m in the second camp. Staff Engineer is not more engineer than Senior Engineer. It’s a slightly different type of job acting as a force multiplier, hence the title change.

For the tech that is inside one team, the engineering manager is expected to be technical enough to cover that need. If the EM is merely a team lead doing admin stuff and babysitting the engineers, they’re not doing their job.

Tech lead can be used as a patch to cover for incompetent EMs, but as we’ve elaborated previously, this is an anti-pattern that is harmful for the team, tech and product.

3. The Solver anti-pattern

We’ve all seen those super-skilled coders (also known as coding machine) who can solve problems that are too complex for others and deliver solutions in record time.

That technical moat is respectable. In a way the Solver is the opposite of Architect.

However, to go ahead and call them Staff Engineer creates more confusion than clarity.

There is, however, a need to distinguish between a “regular” Senior Engineer and a high-impact Senior Engineer if the title and pay are hardly coupled.

The change in the title should communicate a change in responsibilities which requires different skillset:

• Senior Engineer: hard skills (coding, troubleshooting, etc.)

• Staff Engineer: soft skills (influence, leadership, negotiation, etc.)

• Principal Engineer: business acumen (connecting profit to code, etc.)

There needs to be a way for companies to reward outcome without drastically changing the title. It’s totally fine to have Senior Developer 1, Senior Developer 2, Senior Engineering 3, … to signal the reward.

Coupling the pay bands to title is one of the most common career ladder pitfalls.

Those ladders fail to communicate that the expectations, responsibilities, and the toolbox of Staff Engineer is fundamentally different than that of a senior engineer. Without soft-skills the code machine is just a glorified Senior Engineer not a Staff Engineer.

Does a team need Staff Engineer? Not really.

There are cases when you can temporary have the Staff role at a team level:

• When the Staff Engineer is new to the company, having a smaller scope allows smoother onboarding.

• When there’s a new Engineering Manager in the team, they can use the help of a Staff Engineer to get hold of the tech

• When the company is committed to confuse employees with no clarity in the career ladder.

4. The Right Hand anti-pattern

There are 100 ways to get this wrong and I haven’t seen a proper implementation in my career. The wording ("right hand") implies that the Staff has a borrowed or delegated authority.

But authority over what?

The tech.

The tech doesn't talk or listen. Staff MUST go through EM and PM to deliver an impact. That's what makes it tricky but also valuable.

Lack of mandate is one of the most common frustration among Staff Engineers and for good reason: it’s much easier to pull the mandate card to get sh*t done.

However, I believe that the lack of mandate is actually a perk because it forces you to resort to other tools:

• Follow me leadership: show how it’s done instead of telling people what to do

• Strategy: strategy is a powerful tool to build alignment around a common goal

• Mentorship: good leaders bring the best out of people, great leaders create new leaders.

• Open TODO: communicate your priorities to gain volunteers for your initiatives while demonstrating transparency

• etc.

I call it “learning leadership the hard way” because once you learn how to lead without the mandate, you’re much less likely to abuse the mandate when you have it (e.g., if later in your career you decide to switch lanes and become a engineering manager).

When you're a manager, people respect you knowingly or unknowingly: your jokes are funnier, your concerns are taken seriously, even your f*rt smells good! 😆

As it turns out, people have a soft spot for the person who is in charge of their raise, promotion, vacation planning, etc. Who would have thought?

Staff Engineers don’t have mandate over people.

The lack of mandate unlocks quality feedback from the engineering community that’s much less likely to reach the managers on the reporting lines. As it turns out, people feel more connected with fellow engineers when their words are less likely to be used against them. Is that surprising? 😄

Recap

1. Archetypes should not be treated as job titles.

2. Breaking a role to multiple titles is bad both for the company (more bottlenecks) and employees (turning to one trick ponies).

3. The most effective Staff Engineers can act as any of those archetypes depending on the needs and situation.

4. Architect is not a full time job. Tech translator and tech governor are career dead-ends (unless you’re planning to spend all your career in broken organizations).

5. Avoid fake work like plague. Always trying to put what you do into the context of business value.

6. Change in the title communicates a change in responsibilities which requires a change in the toolbox to do the job.

7. Staff Engineer is responsible for the technology and its main potential is realized when that tech spans across teams.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

The simplest way to support me is to like, subscribe and share this post.

If you really want to support me, you can consider a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. You can get 20% off via this link. Your contribution, also funds my open source products like Service Level Calculator.

You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

Now you need to quickly onboard yourself to a new technology landscape, product area, operation model, and new people (T-POP).

This post discusses:

• What makes self-onboarding special for senior tech lead roles?

• How effective senior technical leaders (Staff, Principal, Distinguished engineers) ask questions to self-onboard?

• What are the common pitfalls and how to avoid them?

• A list of bad questions, what’s wrong, and how to ask better questions.

Challenges of self-onboarding into senior tech lead positions

There are a few aspects that makes self-onboarding particularly tricky for Staff+ roles:

• As a senior IC (individual contributor), you’re given more freedom and trust which implies less land holding. Somehow it’s assumed that you know what you’re doing even if it’s not always the case.

• As a higher level engineer with elevated voice, you also act as a role model for the other engineers. The way you approach your onboarding sets the bar for how others navigate ambiguity.

• As a technical leaders, your deep understanding of the T-POP is key to do your job effectively, but you cannot gain that knowledge without building some credibility and gaining experience. At times this catch-22 can feel like changing tires while the car is driving! 😄

In a previous post we have covered what your manager can do to facilitate your onboarding:

This post is about what YOU can do as a senior IC to self-onboard. More specifically: how to ask questions as a senior technical leader.

Characteristics of good questions

Questions are the primary tool to build a knowledge base. Another tool is to dig in the code but that’s out of the scope of this article.

Subscribe now

There are big differences between poor questions and excellent ones:

1. Collaborative vs. judgmental tone: good questions invite partnership rather than suggesting criticism of existing systems.

2. Forward-looking vs. backward-blaming: good questions focus on understanding for future improvement rather than assigning blame to previous decisions.

3. Curious vs. prescriptive: good questions demonstrate genuine interest in learning rather than suggesting you already know the answer. New technical leaders are especially vulnerable to this pitfall because they may feel the pressure to solutionize the visible problems and throw their baggage from previous jobs at the new one.

4. Specific vs. vague: good questions target precise information rather than general complaints or observations.

5. Open-ended vs. closed: effective questions encourage detailed responses that reveal context and nuance.

6. Me vs We: effective questions frame you as part of the group facing the same problem. They avoid putting the spotlight on you as the hero and savior but rather invite the group to have a constructive dialogue that helps evolve the status quo together.

7. Pulling ranks vs Service Minded: while it’s true that you may be hired at a higher position in the career ladder, pulling ranks can quickly alienate you. Instead adopt a service-minded approach (e.g. how can I help? What can I do with my elevated voice to unblock you?)

8. Positive vs negative: every interaction also makes an impression. These impressions are even more important for leadership roles like Staff+. Positivity is magnetic. Anyone can call out what’s wrong, but effective leaders can reframe a problem to a challenge followed by action using positive language.

9. Noise vs can-do attitude: when you start in a new environment, too many opportunities for improvement meet your fresh eyes. Plus, you may feel the pressure to make a big first impression. Instead of making noise, pick a few high impact initiatives and expand your learning while executing. (see 3L: look, listen, learn)

The primary purpose of the question is to gain knowledge, but for senior IC roles, the secondary objective is to build a connection and make a good impression.

The next section goes through good/bad questions to navigate:

1. T - Technology, architecture, and technical health

2. P - Product and Business

3. O - Operation and Culture

4. P - People and networking

Example Questions

Note: generative AI (Claude) was used to create the initial draft of this section. I have gone through every word and edited the result heavily to make sure it reflects my views and experience. Hope it will be useful for you as well.

The format is intentionally made to be easy to skim through so you don’t have to read the whole thing.

1. Technology, architecture, and health

Current Architecture and Systems

❌ "Why is the architecture so complex here?"

• This question assumes complexity is negative rather than necessary.

• It puts the respondent in a defensive position.

• Lacks specificity about what information you're seeking.

• Can be perceived as criticizing previous decisions.

• Doesn't acknowledge the evolution and constraints that shaped the system.

✅ Effective Questions:

1. Could you walk me through how the main components of our system interact and the reasoning behind these design choices?

2. I'd love to understand the architecture diagram from a historical perspective—what were the key inflection points that shaped our current system?

3. Which parts of our architecture are you most proud of, and which areas do you think might benefit from fresh perspective?

System Evolution

❌ "This looks outdated. When are you planning to modernize it?"

• Makes a judgment about the system being "outdated" without understanding context (e.g. maybe there was a time or budget limit).

• Presumes modernization is needed and already planned.

• Implies criticism of the current maintainers.

• Any fool can criticize a system, but do you have a fit solution and more importantly are you going to be part of the solution beyond talking?

✅ Effective Questions:

1. How has this system evolved over time, and what were the key decision points that shaped its current state?

2. What trade-offs were considered when designing the current architecture, and how have those trade-offs aged as the business has grown?

3. I notice we're using [specific technology]—could you share the context behind that choice and how it's served the team?

4. How can I help improve this system’s most immediate pain points?

Pain Points

❌ "What's broken that I need to fix?"

• Assumes systems are broken rather than evolving.

• Positions yourself as a fixer before understanding the landscape.

• Lacks specificity about problem areas.

• Can come across as arrogant or dismissive of previous work.

• Misses opportunity to understand prioritization and constraints.

✅ Effective Questions:

1. Which areas of our technical stack currently face the most challenges or would benefit most from improvement?

2. What technical areas keep you up at night or consistently require the most maintenance effort?

3. Where do engineers typically encounter friction when developing or deploying changes to the system?

4. [If there’s verified tech debt] Do we need to set a payment plan for the tech debt?

Long-term Direction

❌ "Why doesn't the company have a clearer technical strategy?"

• Assumes a negative (lack of strategy) without evidence.

• Signals judgment before understanding context

• Puts the respondent in a defensive position. Implies criticism of leadership while conveniently ignoring the reason you’re hired as a technical leader.

✅ Effective Questions:

1. How is the long-term technical vision developed and communicated across the organization?

2. What are the biggest technical bets the company is making in the next 2-3 years?

3. How do technical leaders balance tactical delivery with strategic architectural evolution?

Technology Selection

❌ "Why aren't we using [trendy technology]?"

• Suggests you've already decided what's best without context.

• May come across as bringing baggage from previous employers.

• Doesn't acknowledge the costs of technology transitions.

• Can seem dismissive of previous decisions and constraints.

• Presupposes the current technology choices are suboptimal. It may well be the case, but while you think of yourself as the guy with the wheel, the others may see the move as arrogant and simplistic. Read more here.

✅ Effective Questions:

1. What processes do we use for evaluating and adopting new technologies?

2. How do we balance innovation with standardization across the tech stack?

3. What criteria guide our technology adoption decisions, and how do we measure success after implementation?

Technical Debt

❌ "Why is there so much legacy code?"

• Assumes negative judgment about existing codebase.

• Presupposes so much legacy code exists without context. How do you even measure code? Lines of code? Number of invocations? File count? Git pull volume? Number of components in a system? In general stay clear from code size metrics especially in a negative context which motivates the maintainers to put their critical thinking hat on and find flaws in your argument.

• Implies criticism of previous engineering decisions and makes maintainers of that code defensive.

• Frames legacy code as purely negative rather than evolving. Legacy code may be intimidating to you but once you bother to learn the constraints, you get a better picture about the limitations and where you can improve it with surgical precision.

✅ Effective Questions:

1. What's our perspective on our technical debt landscape and how it's currently being addressed?

2. How does the team balance maintaining existing systems with introducing new technologies?

3. Which parts of our codebase have stood the test of time, and which areas might benefit from investment in modernization?

Technical Standards

❌ "Who enforces coding standards here?"

• Focuses on enforcement rather than collaboration.

• Suggests standards need policing rather than facilitating. Policing can also be disguised behind more benign terms like governance. In general, when someone focuses on governance and making the rules without actually being part of executing those roles, you’re dealing with an ITA (Ivory Tower Architect) —which is by the way the biggest risk ahead of Staff+ engineer positions.

• May signal a rule-oriented rather than outcome-oriented mindset.

• Doesn't acknowledge team autonomy and different contexts.

• Misses opportunity to understand how standards evolve.

✅ Effective Questions:

1. How are technical standards or best practices developed and adopted across the organization?

2. What balance have we found between team autonomy and technical consistency?

3. How do we measure and improve the adoption of shared architectural principles?

2. Product and Business

Current Priorities

❌ "What should I work on first?"

• Puts the burden of your direction on someone else.

• Signals passivity rather than strategic thinking.

• Doesn't demonstrate awareness of organizational goals.

• Misses opportunity to show you've done homework on priorities.

• Positions you as waiting for direction rather than driving impact. As senior IC role, you are expected to navigate the ambiguity and be self-motivated to focus on what matters.

✅ Effective Questions:

1. What are the organization’s top priorities this quarter (e.g. OKRs), and how do they align with broader company and business goals?

2. I've reviewed our roadmap and am particularly interested in [specific area]. How do we assess someone in my role contributing most effectively to this priority?

3. What technical capabilities would unlock the most value for the business in the next 6-12 months?

Business-Technology Symbiosis

❌ "Does the business side understand what we do?"

• Creates an us-versus-them division between technical and business teams where in reality, there should be a strong connection and symbiosis.

• Assumes lack of understanding rather than different perspectives.

• Places blame rather than seeking shared understanding.

• Doesn't acknowledge your role in bridging potential gaps. Senior IC positions have a key role to play to bridge that gap. And that question doesn’t give a proper answer about the width of that gap.

• Frames technical work as separate from business outcomes.

✅ Effective Questions:

1. How do we ensure business and technical priorities remain aligned as we scale?

2. What have been effective ways to translate between business needs and technical solutions?

3. How do senior ICs contribute to shaping business strategy, not just implementing it?

Product/Tech Strategic Alignment

❌ "Why are we focusing on this instead of that?"

• Challenges current priorities without understanding context.

• Positions yourself as questioning leadership decisions immediately.

• Creates an adversarial dynamic.

• Suggests you think current priorities are wrong. We’re not aiming to normalize self-censorship, but when you’re new at the company and haven’t built credibility, you may first want to focus on delivering on some of the existing priorities before challenging the system from the get go.

✅ Effective Questions:

1. How do our current technical investments align with the company's strategic direction over the next 1-2 years?

2. I'm curious about how technical strategy gets developed here—how do product roadmaps and technical roadmaps influence each other?

3. What business outcomes are we trying to achieve through our current technical priorities? (note: instead of sharply piercing the prio list, this question is inviting to connect them to what matters to the business model)

Measuring impact

❌ "How do I prove I'm adding value?"

• Focuses on personal validation and seeking approval rather than organizational impact.

• Reveals insecurity rather than confidence. Please note that our goal is not to fool those who confuse confidence for competence. If you’re not confident but honest, that’s OK. But asking so bluntly sends the wrong signal and confuses people about what your role actually means.

• Frames contribution as something to prove rather than deliver. The results should speak for themselves.

• Doesn't acknowledge different types of value senior ICs can provide.

• Misses opportunity to understand success metrics.

✅ Effective Questions:

1. How is success typically measured for initiatives at my level, both in the short-term and long-term? (note the subtle difference: “initiatives at my level” vs “me”)

2. What types of impact have other senior ICs had that were particularly valued by the organization?

3. Beyond direct technical contributions, how do you evaluate the effectiveness of senior technical leadership? (note: technical contribution can be to code, design systems, introduce new tools, and even write/present ideas that lifts the level of the organization)

3. Operation and Culture

Previous Initiatives

❌ "What projects have failed here before?"

• Focuses exclusively on failure rather than learning. It is easy to call out failure in retrospect, but when the decision was made, it was probably not that obvious given the circumstances and people involved.

• Uses charged language ("failed") that may make people defensive. Instead of trying to look smart by calling out other people, focus your attention on the learnings.

• May come across as judgmental of past work.

• Shows no interest or acknowledgement to successful initiatives.

✅ Effective Questions:

1. Could you share examples of previous technical initiatives that either succeeded or faced challenges, and what we learned from them?

2. What patterns have we noticed in projects that gained strong adoption versus those that didn't achieve their intended impact?

3. I'd love to understand our technical journey—what major shifts in direction have we made over the past few years and why?

Working Style

❌ "Do people work late here?"

• Narrowly focuses on hours rather than overall work culture. For knowledge work, productivity (output/input) is notoriously hard to measure. Not all hours are equal.

• Could be interpreted as wanting to do minimal work. Could signal concern about work-life balance before even starting. While this balance is important, my experience as Staff+ across different organizations shows that the extra autonomy and higher expectations complement each other. The balance is more influenced by the internal drive than external expectation.

• Doesn't effectiveness (doing the right thing) and impact on business bottom line through increasing revenue or reducing expenses.

• Misses opportunity to understand different working styles. Many companies for example of Flexible Schedule where the employee can get the job done at their own pace even if it means resting during standard office hours and working outside those hours. As a parent, I appreciate this degree of flexibility, but it also requires discipline to not mix personal and professional life —again, as a senior IC, you’re supposed to know how to manage your time.

✅ Effective Questions:

1. What working styles have we found most effective in this environment, and how do successful senior ICs typically operate?

2. How does the team balance deep focused work with collaboration and meetings?

3. What communication norms and expectations should I be aware of as I join the team?

Organizational Learning

❌ "Does the team repeat the same mistakes?"

• Creates a yes/no question instead of inviting detailed context.

• Implies judgment about team's ability to learn. Assumes negative patterns without evidence

• Positions yourself as an outside critic. If you continue that route, you’ll be cast out and lose the influence that is required to deliver more than one person (my definition of leadership: “Leadership is the ability to deliver more than any single individual is capable of”)

✅ Effective Questions:

1. How does the organization capture and apply lessons from previous technical challenges or incidents?

2. What mechanisms exist for sharing knowledge and learning across teams?

3. How do retrospectives or post-mortems influence future technical decisions?

Knowledge Sharing

❌ "Why isn't there better documentation?"

• Assumes current documentation is inadequate. While that might be the case, as a senior IC role, you’re expected to roll up your sleeves and fix the problem instead of stopping at finger pointing.

• Doesn't acknowledge trade-offs in maintaining documentation. Every word has a cost to write, edit, keep up to date, find, and read. The documentation may be just enough, because the code compliments it for its primary users.

✅ Effective Questions:

1. What knowledge-sharing practices have been most effective across the engineering organization?

2. How do you balance documenting systems versus building personal connections for knowledge transfer?

3. What opportunities exist for me to contribute to our collective technical understanding and documentation?

Executive Communication

❌ "How technical are the executives here?"

• Can be interpreted as judging executives' capabilities.

• May suggest unwillingness to adapt communication style. Although the best engineering leaders are technical (and your role may be an anti-pattern to cover their lack), you should probably stay clear from this kind of question where the answer can be very subjective.

✅ Effective Questions:

1. "ow do senior technical leaders typically engage with executives on technical matters?

2. What formats work best when communicating complex technical concepts to leadership? (for example: some companies run on PowerPoint and discussions behind closed doors while others rely on written text)

3. How involved are executives in technical decision-making, and at what threshold do issues escalate to their attention?

Growth Opportunities

❌ "What do I need to do to get promoted?"

• Focuses on advancement before demonstrating value.

• May appear presumptuous early in your tenure. This is especially tricky when the company is not growing and fails to prioritize employee growth accordingly.

• Suggests primary motivation is personal advancement. While it’s true that no one cares more about your growth than you do, adding a few words, you can make it less personal and gain broader perspective.

• Many companies use a Y-shaped career ladder where above a level, you get to continue as IC or manager. The promotion criteria beyond that level is often vague and subjective. Instead of focusing on the word “promotion”, you can ask about how impact is assessed.

✅ Effective Questions:

1. How do senior ICs typically grow their impact and influence within the organization over time?

2. What opportunities exist for technical leadership outside of people management roles?

3. How does the organization support continued learning and skills development for experienced engineers?

4. What mechanisms does the company use to map individual growth with the business growth?

Driving Change

❌ "How do I get other teams to adopt my ideas?"

• Frames adoption as producer-consumer rather than a dialog and collaboration.

• Positions your ideas as inherently worth adopting.

• Focuses on personal evangelism rather than collective evolution.

• May signal overconfidence before understanding organizational context.

✅ Effective Questions:

1. How do technical initiatives that span multiple business units typically get coordinated?

2. What governance structures exist for platform or foundational technologies?

3. What approaches have been most successful for driving organization-wide technical improvements?

Career Growth

❌ "Why aren't there more senior engineers on the team?"

• Implies judgment about team composition. Promotion and hiring are hard to get right. In many cases, there’s a mismatch between people’s formal level and their impact. It’s not ideal but it’s very common.

• Positions experience level as the primary value metric.

• Could be demoralizing to current team members who are mis-leveled.

• Pulling ranks may also put the other engineers in defense mode and severely reduce your influence and ability to drive change.

• Doesn't acknowledge diverse forms of contribution.

✅ Effective Questions:

1. How are technical mentorship relationships structured here?

2. What opportunities exist to help grow and develop the technical skills of engineers across experience levels?

3. How do senior ICs typically influence technical practices and standards while empowering others?

Decision Making Process

❌ "Who do I need to convince to get things done?"

• Frames relationships as transactional or manipulative.

• Focuses on convincing rather than collaborating.

• Suggests bypassing proper processes.

• Overlooks the importance of building consensus.

• Signals potential disregard for team dynamics.

✅ Effective Questions:

1. How are technical decisions typically made here? Who are the key stakeholders for different types of changes?

2. Could you walk me through the process for proposing and implementing significant architectural changes?

3. What's been your experience with the most successful cross-team technical initiatives? How did they navigate decision-making?

Information sharing

❌ "Do people actually read the documentation here?"

• Contains a negative assumption about current practices.

• Uses actually which signals skepticism.

• Invites a yes/no answer rather than useful context.

• Can be perceived as judgmental about team habits.

• Misses opportunity to learn about effective communication strategies.

✅ Effective Questions:

1. What communication channels work best for different types of information sharing? How do teams stay aligned on technical changes?

2. How is technical knowledge typically documented and shared across the organization?

3. When proposing a significant technical direction, what formats and forums have you found most effective for building alignment?

Resources and Support

❌ "What's the budget situation?"

• Vague and lacks context for why you're asking. It may appear to be fishing for sensitive information.

• Doesn't specify what resources you're interested in.

• Misses opportunity to discuss alignment with business priorities.

• Focuses on constraints rather than possibilities.

✅ Effective Questions:

1. What resources are available for exploring improvements or innovations in our technology stack?

2. How does the team typically approach experimentation and prototyping new solutions? (note: this is clever because it works around the budgeting constraints and focuses on smaller experimentation rather than big bang changes)

3. What's the process for securing resources for technical initiatives that weren't initially in the roadmap?

4. People and networking

Informal Hierarchy

❌ "Who has the real power here?"

• Suggests focus on politics rather than collaboration.

• Implies distrust of formal organizational structure.

• Uses loaded term real power which can sound manipulative.

• May make the respondent uncomfortable discussing informal dynamics.

• Positions relationships as power-based rather than collaborative.

✅ Effective Questions:

1. Beyond the formal org chart, how do ideas typically gain traction and support in the organization?

2. How do successful technical initiatives usually build momentum across teams?

3. What approaches have you seen work well for gathering feedback and building consensus around architectural changes?

Key Relationships

❌ "Who are the most important people I should know?"

• Reduces relationships to hierarchical importance.

• Suggests networking for political purposes.

• Lacks context about why relationships matter.

• Doesn't specify what kind of knowledge you're seeking.

• May come across as calculative rather than collaborative.

✅ Effective Questions:

1. Which individuals or teams would you recommend I connect with to better understand our core systems and processes?

2. Who are the domain experts I should consult when working on [specific area of the system]?

3. Which team members have historical context that would help me understand why certain architectural decisions were made?

Team Collaboration

❌ "Which teams don't work well together?"

• Focuses on negative dynamics rather than constructive patterns.

• Invites gossip rather than understanding.

• Can make the respondent uncomfortable sharing sensitive information.

• Doesn't acknowledge your role in improving collaboration.

• May position you as someone who focuses on problems not solutions.

✅ Effective Questions:

1. How do cross-functional teams typically collaborate on projects, and where do you see opportunities to improve this collaboration?

2. What collaboration patterns have been most successful for complex projects that span multiple teams?

3. As a senior IC, how can I best contribute to strengthening cross-team relationships and technical integration?

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

The simplest way to support me is to like, subscribe and share this post.

If you really want to support me, you can consider a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections and my online book Reliability Engineering Mindset. You can get 20% off via this link.

You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.

At the moment I’m in a couple of processes but I want to build more options to pick the best one. If your business needs a curious observer who is not work-shy or ivory tower, we may become colleagues 🙂

More info on LinkedIn including what my former and current colleagues say about me.

Vertical Organizational Distance (VOD) is measured as the number of people between two individuals in the same reporting lines through the organization tree:

VOD primarily influences the information flow which in turn impacts the quality of decisions and execution:

1. Leaders need quality insight for good decisions. That insight is often at the leaf nodes of the organization tree where people solve concrete problems for the service consumers.

2. Leaf nodes of the org tree need alignment to pull in the same direction effectively (doing the right thing) and execute efficiently (doing the thing right).

This article is part of a series about organizational architecture and covers the impact of Vertical/Horizontal Organizational Distance with illustrations, videos, and examples. We conclude with pro-tips to reduce the negative impact of high VOD.

🤖🚫 This page contains 1 AI generated image which is marked clearly. Otherwise no generative AI was used to create this content. This page is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.

DTT

Distance To Top (DTT) is the Vertical Organizational Distance between an individual and the top person in an organization (typically the CEO of the company or the department lead).

DTT is a special VOD because it impacts the general agility and effectiveness of the organization/company as a whole.

MAX DTT indicates the distance of the individuals farthest away from the leadership which increases the risk of misalignment and poor leadership decisions.

Flat organizations have smaller MAX DTT.

Heavily hierarchical organizations have a higher MAX DTT which means there’s a thicker layer of middle managers.

Silos

If you’ve worked anywhere with more than a thousand people, you’re familiar with the silos: different parts of the organization act as if they are different companies with distinct concerns, cultures, and contexts.

Silos show a bit of “we vs them” attitude and cause friction against cross-organization collaboration.

When I was younger, I tried to break the wall of those silos to bring people together. I still do. But I’ve learned that silos are a side effect of how human hierarchies develop to work around our limited cognitive abilities and bandwidth.

How do they work?

The primary focus of each middle manager is their own subset of the org. They are given objectives (e.g. OKRs) that may not necessarily align with their peers at other parts of the organization.

So they get to work. They may develop strategies and impact the culture in their corner of the “world” where they have mandate.

They may even try to influence and inspire other parts of the org but their impact is limited without the mandate.

Sidenote: it’s totally possible to impact without mandate, which is how great Staff Engineers act as a force multiplier.

HOD

Horizontal Organizational Distance (HOD) is measured as the number of people between two individuals in the organization through the org hierarchy.

For example, there are 3 people between person A and person B in this org tree:

Similarly the HOD between the following A and B is 7:

HOD can be calculated from VOD:

That’s an oversimplification because the line between A and B involves both going bottom-up and top-down, each introducing unique risks —as we’ll discuss.

With that caveat in mind, our main focus in this article will be VOD.

HOD usually doesn’t matter if the organization architecture is reasonable. However, it gets in the way of cross team collaboration especially when the interests of the different part is misaligned. Leadership mandate can act as a tie-breaker as a last resort.

We’ve previously covered how TPM (technical program manager) acts as a virtual kebab stick to align teams across the organization to deliver a specific outcome.

An effective method for reducing HOD is to bring relevant parts of the organization together. We’ve covered this in Kebab vs Cake organization architecture:

You can also read an example of high HOD for platform organizations here:

A problem as old as humanity

The ability to organize in large numbers and collaborate flexibly is why humans dominate the earth while much stronger animals are in cages or struggle to survive on the sidelines. 🐳🦍🐅

If an alien observer zooms in to a hive of hairless monkeys, they may detect hierarchies: formal and informal ones, information hierarchies, power hierarchies, military ranks, governments, and corporate org charts.

We tend to organize in hierarchies because they are an efficient way to manage cognitive load and skill demand while creating a cohesive whole that has higher bandwidth than its parts. The reporting lines primarily exist to support alignment, distribute accountability, and aggregate insights.

Unfortunately hierarchies also introduce new problems like diffusion of accountability, information bottlenecks, and the heroism it takes to keep a corrupt system up and running.

This post uses a simple tree structure to represent the org chart. The reality is more complex and resembles a graph:

A simplified model

Let’s use a simplified model to examine the impact of VOD.

Suppose all the important decisions are made at the top while the bottom layer is only responsible for execution.

Note: this is the kind of environment where the managers at the top call themselves “Head”. To them, the rest of the org is just an execution “body” with no interesting thoughts. All thinking are abstracted to the top. You can easily look up what your managers call themselves on LinkedIn. 😄

The decision has to trickle down through layers of information brokers to reach its target:

Each layer introduces a risk of:

• Delay: it takes time for each layer to understand and communicate to the next layer regardless of using meetings or more clever mediums like write-ups

• Distortion: misunderstanding, misinterpretation, miscommunication whether intentional or malicious

• Reduction: filtering out information whether intentional or unintentional

Good decisions need good insights.

Assuming that in this simplified model, only the leaf nodes of the organization interact with consumers to solve their problem, that’s where the best insight is.

However, an insight has to bubble up through layers of information brokers to reach its target:

You may recognize the similarity between an org tree and the computer data structure of the same name. There’s a significant difference between computers and humans.

Computers are quick, predictable, and logical creatures which store and process a massive amounts of data without distortion.

Humans are the opposite!

In fact, when computers fail to do what they’re supposedly good at, almost always a human was involved! 😁

Humans are flexible and can adopt to novel situations, but every additional layer of humans in the hierarchy increases the risk of delay, distortion, and reduction.

The Chinese Whisper game is a method of perception education that relies on a line of people convey a message by whispering mouth to ear one at a time. The fun of the game comes from the subsequent solution, in which the last person in the row says out loud what they understood as the message.

It demonstrates delay, distortion, and reduction of information.

The idea is not new. Here’s a visual variation with 2 persons:

Reduction

Let’s do an experiment.

Suppose every layer of the hierarchy is a sheet of glass. Don’t we all want glass-level transparency? 😄

By this point, it should be obvious that the glass is not fully transparent: the information is lost at every human layer due to delay, distortion, and reduction.

For the sake of simplicity, let’s just assume that only 20% of the information is lost in each layer.

Putting a few layers of 80% transparent glass on top of each other, we get this:

As you can see, every layer makes the layer below slightly more opaque. We are overly simplifying here. In reality, some human information layers are completely opaque while other layers are more transparent. But we go with 80% transparency in this model.

Bubble up information reduction

Leadership needs insights that exist at the lower layers of the org tree. At the bottom layer, these insights are obvious:

That information is not very legible when looking from the top layer down:

In the hierarchy pyramid, there’s more people at the bottom than the top. This means the leaf nodes of the org tree generate more data than can be consumed at the higher level:

The leader cannot consume all the data or insight. “Executive summaries” are pragmatic but they take away part of the insight, context, and impact of the feedback.

An aspect that’s not shown in the image is information compartmentalization. In reality, the insights of leaf nodes exists in pockets of information that’s localized to the context defined by department, cluster, and team solving a subset of consumer problems.

“The lower the rank of managers, the more they know about fewer things. The higher the rank of managers, the less they know about many things.”

— Russell L. Ackoff

Trickle down information reduction

Let’s look at a strategy that’s clear at the leadership level:

Changing perspective from the bottom layer, this is the amount of information that trickles down:

Not readable right?

That’s how it feels to be at the leaf nodes of the org and look at fancy talks and carefully crafted presentations coming from top levels.

Many people at the leaf nodes of the org tree learn that top-level leadership worries are not their worries. The longer the VOD, the less impact it has on their day-to-day lives in general. They can afford to ignore leadership until they can’t.

The leadership on the other hand is frustrated with misaligned execution and lack of insights which forces them to go with gut feelings, services from management consultancy companies (e.g. McKinsey), or just winging it. Those are symptoms of ivory tower leadership who are detached from the realities of their organization.

Most organizations don’t have a proper bottom-up feedback channel due to power asymmetry between employees and managers. As a result, the insights are typically stuck somewhere in the middle and never reach the top.

The communication network is clogged. Poor communication leads to poor execution.

Distortion

To show the impact of information distortion, we can add noise to one of those layers:

Here’s what we get for top-down decision distortion:

…and bottom-up insight distortion:

And that’s just one layer distorting the information!

Scale it to multiple layers and you get this:

Tall organization charts cultivate a gap between executive and executives that is filled with middle managers whose sole reason to exist is to manage up and down.

This cartoon aptly demonstrates the key skill needed at these levels:

Delay

Our illustration used light as a metaphor for information. Each layer adds delay to understand the information and then communicate it to the next layer.

Here’s how it goes in a meeting-heavy work culture:

1. The CEO goes to a meeting with the other C-Level, to take a decision.

2. They in turn meet their VPs to communicate the decision.

3. The VPs communicate with directors, and so on and so forth all the way to the team level.

The reverse is equally horrible with an extra twist: most bottom-up information flows are broken. For a piece of insight to travel upwards, it has to come from multiple sources to gain enough credibility and weight to bubble up. That accumulation takes time.

Subscribe now

The longer the distance between executives and the people who execute, the longer it takes for the organization to react to change.

Think of it like holding a long and flexible screwdriver while trying to fix an engine. The longer it is, the more inertia it has and the less precision.

Pandemic, war, sanctions, inflation, AI, end of ZIRP, elections, climate change, etc. It’s safe to say that accelerating change is the new norm.

In time of crisis, that lack of agility can cost the collapse of the company.

The quality of information networks (both bottom up and top down) is the key contributor to the agility of an organization in the face of change.

What distinguishes the organizations that thrive, survive, or collapse is their ability to adapt to change.

It’s not about alignment

High VOD manifests itself as:

• Frustrated executives

• Clueless execution

• Slower evolution

• Waste of effort, time, bandwidth, and money

Before going to solution mode and discussing pragmatic ways to reduce VOD, let’s make one point clear: alignment is not the goal.

Perfect alignment can scale failure:

A healthy information network is a 2-way street where leadership and leaf nodes of the organization tree can exchange without the need to go through brokers —AKA middle managers.

I recently came across a comment under this post:

Let me translate this corporate theater into actual numbers:

The real cost of "zero value" management:

• $500K salary × 100 managers = $50M burned

• 50% efficiency loss × 1000 engineers = 500 engineers wasted

• 30% project delays × $10M projects = $3M/project lost

• Countless good engineers leaving

But here's what nobody talks about: This isn't a bug. It's a feature.

Large tech companies NEED:

• Political layers

• Decision diffusion

• Responsibility dispersion

• Accountability shields

Why? Because at scale:

• Clear ownership = clear blame

• Direct communication = direct responsibility

• Efficient processes = fewer managers

• Quick decisions = risky careers

The real game isn't finding “good managers”. It's understanding that:

• Systems create behaviors

• Scale rewards politics

• Size demands inefficiency

• Bureaucracy protects careers

Want actual success? Don't look for “good managers”. Build small, accountable teams. —Pranjal Gupta

I’m well aware that republishing this comment makes very powerful enemies but it resonates with my experience over 2 decades.

Besides truth is bigger than one person. Knowing this enables leaders make more deliberate choices when designing their organizational architecture with VOD, HOD, and MAX DTT in mind.

Politics

Most of what’s labeled “Politics” are just reasonable conversations that were had in a room where you were not in.

That has two side effects:

• Feedback: Your feedback wasn’t heard and didn’t influence the decision

• Context: You weren’t part of the thought process that led to the decision

By the time it reaches you, it just looks like bullsh*t because it doesn’t map to the problems you see or solution you expect.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support this work by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

And to those of you who support me already, thank you for sponsoring this content for the others. 🙌

Pro-tips: How to reduce VOD?

VOD, HOD, DTT, bubble-up, trickle-down, information networks, communication brokers, … What can you do with this information?

Flattening the organization is the obvious reaction to this post but it’s not always an option.

There’s a lot that can be done without firing middle managers. You can experiment with these at a small level and scale up if they works for your unique business model, people, and challenges.

This section discusses actionable advice to improve the quality of information network in both ways:

1. Async decision records (ADRs): pushing decisions closer to where they’re executed

2. Skip-levels

3. Report-down

4. Stay-interviews

Read more

When the tech health is poor, it is naïve to think that the engineers have no clue about what good looks like. It is tempting to define a grading system —a “maturity model” if you will— for the engineers to grow the f*** up and get their s*** together.

The idea is simple:

1️⃣ Define what good looks like
2️⃣ Ask the teams to self-assess based on the criteria
3️⃣ Sit back and relax while the teams find a way to pay the expense and lift their level

Unfortunately, it doesn’t work like that in reality. Poor tech does not mean poor engineering and bad engineers. It is just a symptom. The real root cause is somewhere else.

This article elaborates why engineering grading systems don’t work and what to do instead.

🤖🚫 Note: No AI was used to generate this content (except one image which is clearly marked). This essay is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.

Subscribe now

Why and what?

Let’s get to ❌ WHY engineering grading systems don’t work?
and ✅ WHAT to do instead?

❌ One size fit all: it is very hard to define GOOD in a way that fits a wide range of technical artifacts but is still tangible enough to drive meaningful action. There's often some cargo culting at play where best practices from one company are shoehorned to another.

✅ Instead of emitting manifestos, spend time with each team to understand the nature of their services and challenges. Then define measurable expectations for the service (not the people) and give the people full ownership to improve the service.

SLI and SLO come in handy as a tool to measure and set those expectations while acknowledging team autonomy.

❌ Hands off: these kinds of initiatives often come from hands-off technical leadership (Ivory Tower Architect) and engineering leaders who rely on them to act as a technical translator.

✅ Large systems are complex and change dynamically. Seek to understand before trying to optimize. Otherwise, your efforts will be limited to the symptoms and surface while the root causes fight against any optimization effort.

If you want to change an organization, start small, work closely with a few teams, and take iterative action while monitoring the impact. Approach it as an experiment with a clear hypothesis and validate the solution before rolling it out across a large organization.

❌ Presumptuous: the grading systems often assume that engineers don't know any better because if they knew, their outcome would show.

✅ The majority of engineers are smart and know what good looks like. If the results don't match the expectations, you need to dig deeper to understand all the forces at play. Maybe their good is good enough given all the circumstances.

I’ve been in the industry since 1999 across more than a dozen companies and worked closely with hundreds of people. I haven’t met a single engineer that I can call “immature”. That’s not the right word to use for human beings because we evolve. Tech can be immature, but it’s a symptom. The root causes can be:

• Lack of budget (time, money, and resources)

• Bombarding teams with feature works and turning them to feature factories

• Treating developers as code monkeys detached from the business problems

• Moody leadership & constant reorg

• Starving technical competence (e.g. due to low pay and recruitment bar)

• Non-existent or broken career ladder which fails to map personal growth with organizational growth

• Too much tech debt and no payment plan

• Poor organizational architecture that doesn’t support reliable and seamless customer journeys

• Any of the archetypes of broken ownership

• …

Don’t you think if the solution was as easy as drawing a north-star, somebody would have done it before? These days even LLMs can do a pretty decent job at that. Where AI fails is to understand the human experience of building, owning, and being in charge of technical solutions to product problems that solve human problems.

If you must create a north star, make damn sure to frame it around the tech, not people. Don’t make it personal. People lift the tech only when technical maturity is detached from their own identity. They are more likely to improve the status quo if they don’t feel called out.

❌ Distract accountability: typically, the engineering grading systems are formulated in a way that frame the accountability on the engineering teams. They conveniently ignore the role of leadership to hire, mentor, utilize, and retain talent.

They’re equally mute about the responsibility of the organization to invest in software. If the tech is unhealthy, it's engineers' fault.

✅ This is one of the cases where the distinction between accountability and responsibility creates clarity.

You get what you pay for. Tech health has a cost. Software rots over time, top talent may leave or get blocked. Improving the quality of tech requires investment.

The relationship between software and profit (`revenue - expenses) has a significant impact on how investments are motivated and what's the long-term technical vision:

• You don’t need Google-level tech when your startup hasn’t proven a product-market fit

• Your local retail site cannot justify the investment to beat Amazon in every metric

• You don’t need to pay the upfront cost of breaking your monolith to microservices just to make it fit your org structure

• You probably don’t need to refactor your tech stack to use GraphQL

The software timeline and budget drastically differ from business to business. Understand the limitations before proposing a solution. Leave the ivory tower and work shoulder to shoulder with engineers if you want fresh first-hand insight for meaningful initiatives.

Recap

• Instead of cargo culting and blindly applying best practices, try to understand the problems and apply solutions that fit

• If you must create a maturity model, make sure to frame it as a technical maturity model, not team or people maturity model.

• Put the majority of the responsibility on upper-level leadership, not the teams. Associating the tech quality with engineering skill is naïve. There are often more factors at play and the company/leadership needs to invest in tech instead of being mandate heavy.

• A maturity model is useless without proper support from leadership and the company putting its money where its mouth is and investing in lifting the state of tech. And no, threatening isn't support. Invest in education, tools & automation, documentation, and roll up your sleeves to work shoulder to shoulder with other engineers.

• Beware that there’s such thing as good enough! Not everyone needs Google-level tech because quite frankly not everyone has their recruitment bar, compensation, and revenue model.

• Always experiment at a small scale before rolling it out across a large organization. Measure the right metrics before and after each iteration.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

You cannot be responsible for something you don’t control. You need the mandate.

You cannot use that mandate effectively over something you don’t understand. You need knowledge.

You gain knowledge only if you are fully responsible for the consequences of your mandate.

Anything else, is a broken ownership.

Then I got this question from multiple readers:

What about accountability?

To me responsibility and accountability were synonyms until I did the research.

There is a nuanced but profound difference between the two.

The difference isn’t too important in the context of the ownership trio, but it is important implications in:

• Roles Definitions: e.g., what is an Engineering Manager or Staff Engineer accountable for?

• Legal Consequences: e.g., when a company goes bankrupt who is ultimately accountable?

• Army and military: e.g., if the commander’s mistakes perishes soldiers’ lives, who is accountable?

🤖🚫 Note: No AI was used to generate this content (except one image from another article). This essay is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.

Subscribe now

Dictionary

Cambridge English Dictionary:

• Responsible: to have control and authority over something or someone and the duty of taking care of it, him, or her.

• Accountable: Someone who is accountable is completely responsible for what they do and must be able to give a satisfactory reason for it

There’s a bit of recursion there and the meaning isn’t that distinguishable. 😄

Side note: the two words enjoyed different popularity through modern history:

RACI

Responsibility assignment matrix (RAM) describes 4 roles:

• Responsible: those who do the actual work to complete the task

• Accountable: signs off the work that the responsible provides and can only be 1 person. Accountable is also responsible but in practice that doesn’t mean the same level of engagement.

• Consulted: Those whose opinions are sought, typically subject-matter experts (2-way communication)

• Informed: Those who are kept in the loop on progress (1-way communication)

Putting it all together you get this picture:

Army

Lieutenant Colonel Joe Doty (US Army) and Captain Chuck Doty (US Navy) put it like this:

Accountability means consequences, both positive (awards, promotions, superb ratings, etc.) and negative (letters of reprimand, Article 15s, relief for cause, poor ratings, etc.). For example, the Army’s officer evaluation reporting system holds commanders accountable for what happens in their units. —Command Responsibility and Accountability, Doty & Doty

I can’t claim I’m smarter after reading that essay. The “positive consequences” reminds me of idea theft or credit theft:

Legal

Now we’re getting somewhere:

• Responsibility has a factual cause/effect aspect. For example, a storm can be responsible for causing damage to a home.

• Legal responsibility (AKA liability) has a punishment and penalty aspect attached to it. For example if a child steals something, the parents are liable to pay the damage.

In other domains

In ethics and governance, accountability is answerability, blameworthiness, liability, and the expectation of account-giving. If you are accountable for something, you are in a position that requires you to report to or answer to someone for your actions and decisions.

Accountable gives an account, reason or explanation for the action. For example, a healthcare professional is obliged to offer a rationale for their decisions during the course of treatment (ref).

In corporate world, that is reporting lines!

Why we should not separate Responsibility from Accountability?

The primary reason is culture: a command and control structure may work for the army or when there are legal consequences (jail or punishment) but is that really the kind of team environment we need in our companies?

Most organizations don't need to separate accountability from responsibility. If they do need the separation, most probably there are larger cultural problems to solve than which word to use!

Formally, there’s only 1 accountable party (usually the Engineering Manager or any variation of it: Director, VP, etc.). But does this mean individuals aren’t supposed to give an account or explanation for their actions?

More often than not, the separation is pushed by:

• Managers who want to call all the shots (monkey with a gun archetype of broken ownership)

• Upper management who are desperate to find a neck to choke without bothering to understand the details (babysitter archetype of broken ownership)

• Dodgy team members who don’t want to accept the consequences of their actions (teenager archetype of broken ownership)

If there’s a need for a single neck to choke when things go wrong (or hang the medallion when things work out), we’re fundamentally building the wrong organizational dynamics.

I really enjoyed this TED talk by Gitte Frederiksen who leads Boston Consulting Group's work in over 15 countries, with an emphasis on the US, the Middle East, and Sweden:

What if leadership could be for the many? That's scary. And for all you leaders out there thinking it won't work, maybe in some cases it won't. But when it does, we have better outcomes and happier people, with everyone leaning in, even if just a bit more.

A typical view of leadership is a hierarchical organization chart. Either you're a leader or you're not. Most people are being led, not taking lead. Communication often flows from top to bottom along just one line, which means it doesn't match the complexity of problems which move in several directions at once. Decisions are left to one person, the leader, who, being only human, can become a bottleneck of speed and scale. They can miss new ideas, diverse capabilities and potential that exist all over the chart.

So in a network instead, everyone's in charge and we replace power of the few with influence of the many. Sure it looks more messy, but I'd argue more beautiful, more multi-dimensional, more dynamic, more like nature. I believe this model can help us do more and be less dependent on each individual. Which means it’s resilient, and progress is sustainable.

Subscribe now

When should Accountability and Responsibility be separated?

I reached out to one of the hard core proponents of this separation. The main question is: “where does the buck stop?”

"The buck stops here" is a phrase that was popularized by U.S. President Harry S. Truman, who kept a sign with that phrase on his desk in the Oval Office.

“Buck passing” is said to have originated from poker but it evolved to have a nasty meaning in politics (and by extension corporate politics):

Buck passing, or sometimes (playing) the blame game, is the act of attributing to another person or group one's own responsibility. It is often used to refer to a strategy in power politics whereby a state tries to get another state to deter or fight an aggressor state while it remains on the sidelines —John, Mearsheimer (2001). The Tragedy of Great Power Politics

Basically the army version that we discussed above. Any of the following scenarios can benefit from separating accountable and responsible:

• A finger pointing culture in need of finding someone to blame when things go wrong

• A culture of heroism where a single person takes the credit for the team doing the actual work (credit theft)

• Non-technical leadership who needs to rely on technical translators to make sense of things. We’ve talked about why it’s an anti-pattern for Staff Engineers but Engineering managers aren’t immune to this either:

  

• A hierarchical culture where the information is trapped behind closed doors and the responsible parties aren’t deemed worthy of knowing WHY they are doing WHAT they are doing (information asymmetry).

• A parent/child situation where the child is not legally accountable or deemed mentally mature to be in charge of their actions.

• A product where quality of execution does not matter and the responsible party is not obliged to do their best because they are not ultimately held accountable

• A group of monkeys led by an astronaut! 🐒🐒🐒🐒🐒👨‍🚀

• Organization where there’s a “head” role. This title signals that the rest of the org is merely the execution “body”. The head is accountable for all the thinking while the body is responsible for executing orders (a mini-dictatorship). The trick is to actually hold the head accountable when things don’t work out! In my experience that’s when the “head” blames the “body” for poor execution.

Jokes aside, the only valid scenario is when there are legal consequences for the wrong actions.

In tall organization trees where the distance between tech and leadership is too large, accountability can be implemented as answerability: someone to ask questions about how things work. In that context, the team should ideally know the answers but the EM should definitely know.

Most tasks and decisions aren’t of that nature, even if the “head” thinks otherwise! We’re all in this together.

What about ownership trio?

Back to where we started: where’s the accountability in the full ownership?

Accountability is embedded in responsibility and mandate:

• The team has the mandate to make decisions to improve their service

• The team is also responsible to measure the service level from the consumer’s perspective and if their commitment is under threat, the team is collectively responsible to prevent or mitigate that. Alerting is the absolute minimum mechanism to do that.

The reason I didn’t separate accountability from responsibility is because in a healthy culture, most teams don’t face legal consequences. Instead their on-call hurts. In my experience, incidents are enough motivation to nudge the team to up their knowledge and use their mandate effectively.

The knowledge growth element is important, because without that, you’ll end up with the gambler archetype of broken ownership.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support this free content by sparing a few bucks on a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

• An EM: I don’t want to deal with people anymore. I want to focus on the tech. Is Staff Engineering for me?

• A Senior EM responsible for multiple teams: I have too much admin work. Should I hire a Staff Engineer to take over the technical part of my job?

• A Staff Engineer at a team without an EM: I’m practically acting as the team manager and like it. Should I just switch lanes to become an EM?

• A Senior Engineer frustrated with Ivory Tower Staff Engineers: what is the accountability of the Staff Engineers? They act like retired devs.

• A new Staff Engineer: What are dotted reporting lines? I have my line manager, why should I care about other managers?

So I thought it’s time to settle all these questions once and for all.

🤖🚫 Note: No AI is used to generate this content. This essay is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.

When do you need a Staff Engineer?

Staff an EM have some similarities:

Let’s step back and build a model.

Engineers create and maintain the tech. But the tech doesn’t exist in vacuum. It’s a solution to a problem. The problem is defined by the product.

The product offers a service to the consumers, whether they are end users or internal consumers. It solves a problem for someone. It can be:

• An app that’s used to measure running distance and sharing it with other runners

• A website for booking hotels

• An infrastructure platform that’s used by other teams

• A time-report system

• etc.

The engineers usually work in a team led by an EM.

The Engineering Manager is accountable for the engineers as well as the technical artifacts they produce:

When the team is large or the tech is too complex (or both), the EM may not have enough bandwidth to do justice for both their people and technical accountability:

To increase their bandwidth, EM’s are faced with a choice:

• Delegate admin work: To optimize the HR processes and tooling or hire an admin assistant to lift some of the people management load from their shoulder

• Delegate technical load: To hire a Staff Engineer to lift some of the technical load from their shoulder.

You may argue if having a dedicated admin assistant is justified at a team level, but I have seen one admin assistant shared by multiple teams. AI can cover some of the mechanical activities like scheduling events, answering admin questions, and gathering continuous feedback. Good HR tooling and processes can reduce the admin load as well.

Subscribe now

Nonetheless, the admin assistant or AI is not supposed to take over the EM’s people accountability like building a good team, mentorship, performance review, hiring, etc.

Previously we’ve argued that hiring a Staff Engineer to take over the technical aspects of Engineering Management is an anti-pattern.

Particularly, Staff is not supposed to act as a translation layer between the tech and engineers for the EM:

Engineering manager needs to be technical and we have covered some ways to keep the technical skills sharp here.

Staff Engineer could be useful when:

• The technical load is beyond the capacity of EM. For example:

  • Legacy tech that requires intense babysitting/migration

  • Overly complex solution that spans across teams or requires deep technical knowledge

  • Simply because the EM isn’t technical (it’s an anti-pattern but they do exist)

• A clear accountability can be established for a subset of technical load

• The tech expands the boundaries of individual EMs (more on that later)

Organizations that have Staff, should hold them accountable for the technology, not people:

There’s a small but important difference between responsibility and accountability that we’ve covered in a separate article:

The Staff Engineer should be deeper in the technology and have higher technical literacy tied to an accountability. The lack of people responsibility opens up their schedule to be more invested in tech. But the Engineering Manager is not supposed to be completely off-hand either.

So we went from this:

To this:

Now I have some bad news. Having Staff Engineer at the team level is an overkill. As we’ll discuss in a minute, the real value of the Staff Engineer comes from operating across teams. Besides, having two roles accountable for tech can create confusion and breaking one role into multiple titles:

However, there are cases when a Staff Engineer may operate in the scope of one team. For example:

• Legacy tech stack with a steep learning curve when the EM is new

• When onboarding a new Staff Engineer (starting from a smaller scope)

• Too much tech debt piled up and hurting the tech health metrics

• Complexity getting in the way of maintaining the tech

In all those examples, team-scoped Staff Engineer can be a temporary position.

Where Staff Engineers really shines is:

• Acting as glue between the teams.

• Working shoulder-to-shoulder with other engineers and being their voice in the upper management room. Turns out engineers are much more open with other engineers than when communicating with the person who has mandate over their salary, vacation, performance review, and raise.

• Getting very deep into the technical aspects. Unlike EMs, Staff Engineers don’t spend a lot of time in meetings, 1:1’s, and various administrative activities. That extra time should be spent on honing their engineering skills and technical depth in order to lead the technical decisions effectively.

By far, the biggest pitfall ahead of Staff+ Engineers is to turn into Ivory Tower Architects.

Staff Engineer for a system spread across multiple teams

In his seminal essay Will Larson, identified 4 archetypes for Staff Engineers:

1. Tech lead

2. Architect

3. Solver

4. Right hand

What’s shown in the diagram above is a Tech Lead operating within one team. Calling that role a Staff Engineer is a bit of a stretch.

That’s because most of the value of the Staff Engineer comes from operating across multiple teams:

Often, the tech that is owned by one team is just a piece of the puzzle. The larger system or domain may be spread across multiple teams or at least has tight dependencies across the org.

In this scenario it makes sense to have a dedicated role (Staff Engineer) for the entire system regardless of which team owns which piece of it.

Subscribe now

Here’s the catch: the “tech” doesn’t talk or listen:

The tech doesn’t exist in vacuum!

For Staff engineers to deliver their true value, they have to go through both people (engineers) and product:

And that’s why Staff Engineers have all those “dotted reporting lines” (the unofficial but critical collaborations and commitments across the org):

Those with the kin eyes may ask why the arrow is pointing from the Staff to “lower” positions (in the career ladder). The answer is two-fold:

1. Good leadership is about collaboration not authority. See it as Staff going to a team-level EM or PM and asking for favors in a way that fits the engineers and product roadmap, not demanding to lift the state of tech in isolation

2. Staff Engineer is an IC (individual contributor) and by definition does not have any direct reports. If you’re lucky to have a sync channel with EM/PM, please don’t waste it in regular report, but instead turn it to a 2-way dialogue about the state of tech (status quo) and where the tech needs to be to solve the emerging product problems (vision) as well as cohesive actions to take us there (organizational tech strategy).

All these reporting lines can be overwhelming. One tool to align product teams across the org is a strategy document:

There are other ways to reduce the span of tech to one team through deliberate organizational architecture. We’ve discussed one powerful alternative in Kebab vs Cake model using consumer journeys:

Staff Engineers should have clear accountability over a system. That means they expose 3 elements of ownership:

1. Knowledge: They know the tech stack, the product problem and can speak fluently about it as well as code when needed

2. Mandate: They have a say in how the tech evolves and is maintained.

3. Responsibility: They are held accountable for the health of the tech (e.g. incidents, tech debt, documentation, tech defragmentation, etc.)

As we discussed before, Staff Engineer is not a pure technical role and heavily relies on soft-skills to influence the technical organization.

Funny enough, focusing too much on soft-skills is one of the most dangerous pitfalls ahead of Staff Engineers. It turns them to Ivory Tower Architects:

Conclusion

Not all organizations need Staff Engineers, especially when:

• EMs are technical enough to manage their team’s tech

• Tech stack is healthy enough that doesn’t need a dedicated technical steward

• The tech has high cohesion and does not span across multiple teams (one way to achieve that is the Cake organizational architecture)

• The engineering organization is mature enough to find the best way forward for the system they collectively own without the need for a dedicated owner

For orgs that have Staff Engineers, they should clarify the technical ownership and hold the Staff Engineer accountable for it

• Ivory Tower Architect is bad

• Breaking a role to multiple titles is inefficient

• Non-technical EM is inefficient

And just like any essay by some random person on the internet, please please please use your best judgement and don’t get stuck in cargo culting.

I didn’t write some universal laws of physics. You know what is best for your particular technology, product, operations, and people.

If you have counter-arguments, feedback, corrections, or questions, let me know in the comments down below. 🙏

You can also share your thoughts and comments on HackerNews.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support my work by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

I reached out with this question to one of my friends who is the CTO at a mid-size company (500-1000 employees).

Below is his response:

I believe staying technical has been key to my success as a CTO and also past roles that led to this position. I feel that I have a big role to play in setting engineering culture.

It lets me:

• Grasp what is the level quality

• Champion a strategy to improve quality

• Solve problems without needing a tech translator

• Be the voice of my team in C-suit room

• Understand the nuances of the decisions I take

• Connect to my people—uniting us in our craft

Executives love to simplify everything into bullets. But often the bullets hide the context and complexity.

Nuances are key. Devil is in detail. As a CTO you cannot afford to abstract yourself from the technical aspects.

Without technical depth:

• My understanding will be reduced to bullet points! That is a nightmare!

• I lose the ability to drive practical aspects of innovation.

How do I keep up with technology?

• I learn directly from my team

• I remain relentlessly curious, and I'm constantly reading

• I am a generalist these days and know enough about enough, to know that I should not get into details

• I learn by reading RFCs, ADRs, etc. and participating in architecture reviews and forums

• I sometimes discuss technical challenges in 1:1s and skip-levels (when an executive talks face to face with people who aren’t in their immediate line of report)

• I follow conversations on Slack

• I occasionally drop by engineers’ table and have a casual conversation to understand their challenges and world while updating them

• etc.

To what extent? I know enough about our technology to:

• make or support decisions

• connect technical work to business objectives

• understand my team's challenges and strengths first hand without them having to dumb it down

It might be tempting to hire a Staff Engineer to cover that gap. Please don’t go there for the reasons described here:

Engineer-ication

who has a CTO background and these days coaches other CTOs, has written about “Engineer-ication”:

During a few days you block your calendar and agenda rejecting meeting invites and delegating certain tasks to other people, as if you were to go on vacations.

You join one of your engineering team as Individual Contributor, and work on one or more software engineering tasks. — Sergio Visinoni

He also followed up with some best practices in another post.

Sergio formulated his thoughts about this topic at Should Engineering Managers Code?

1. You should prioritize the well-functioning of your team over any coding work.

2. You should be able to drop any ongoing coding work at the first sign of stress on the system.

3. Your team should not make assumptions on your contributions, and […] capacity

4. Don't [take] tasks or features that are in the critical path for delivery.

Hobby Projects

Another idea is to have hobby projects. Some of the engineering leaders I respect a lot have their own hobby projects to improve their engineering skills and catch up with technology evolution.

It doesn’t matter if it’s tinkering with Raspberry Pi or setting up a new Kubernetes cluster, as long as it allows you to live the life of an engineer and gain fresh insight, it’s good enough.

Subscribe now

Power Asymmetry

How technical is technical enough?

Engineering Managers have an say in the team. After all, they hold the levers on everyone’s salary, performance review, vacations, etc.

It is rare that the engineers challenge an engineering manager without the fear of repercussions.

There’s a risk that a bad idea from engineering manager stays unchallenged and goes all the way to production. And when things break, it’s ultimately the engineering manager’s accountability. The engineers are merely responsible for what they’re tasked to do.

It is best to say within a safe distance and let the engineers run the show as much as possible.

It is also very important to build the psychological safety in the team for the engineers to call out bad ideas even if it comes from the leadership.

An EM friend put it in terms of PR (pull requests):

How does a developer feel when the leaders send in a PR? How about comments on a PR review? What if the feedback is actually not a good solution? Does the developer feel forced to implement the feedback due to the manager telling him/her?

What about a technical EM sending in a PR, will the developer feel forced to approve it to not get into "trouble" on performance ratings or salary reviews?

Those are two aspects i have seen that can create a worse technical solution than not having a technical EM. However, i also see your points as well.

I’ve had my share of “micro-managers” who hurt the team culture (with good intentions). Getting too close to code requires a level of trust, openness, and humility that takes time to establish.

So please don’t jump on code after reading this post (easy). Instead spend your time and energy on building an environment that makes the best ideas win regardless of where they came from (much harder).

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

Although the term originated from historical events, its usage expanded to other areas like software, systems, and organizations.

Here’s the story of how cargo cults came to be, with some examples from software and corporate world. In Pro-Tips we discuss actionable insights to prevent, spot, and dismantle cargo culting in your organization.

🤖🚫 Note: No AI is used to generate this content or images.

The historical phenomenon

During World War II, Japanese forces established bases in the remote islands of Melanesia.

This practice was later followed by the US and allies due to logistic and strategic importance of these islands.

To give you an idea how remote these islands were, here’s Tanna where multiple cargo cults emerged:

In a short period of time, the indigenous population were exposed to the largest war ever fought by technologically advanced nations.

Almost all equipment, supplies, clothing, medicine, and food were brought to the military bases as cargo.

The military personnel occasionally interacted with the indigenous peoples as guides and hosts. This exposed them to WHAT these cargos contained.

However, they were left completely in the dark about WHY and HOW of these cargo drops —possibly due to language barriers, racism, or security reasons at the heat of the war.

By the way, the reason “WHAT”, “HOW”, and “WHY” are all caps is because they’re a reference to Simon Sinek’s seminal TED talk.

After the war, the military bases were evacuated.

With that, the cargo stopped coming!

That didn’t stop indigenous peoples to engage in rituals hoping for the return of the cargos. They believed the cargos were sent from their ancestors, and the reason the “white man” captured them is because they were better at rituals.

So they got to work and imitated what they observed!

They marched…

…built airplanes…

…airport runways…

…complete with control tower.

There’s a short documentary on YouTube if you prefer video.

Beneath the funny imitations (symptom), there is a sad story (root causes and contributing factors).

Many indigenous societies of Melanesia had a big man cultural element. Big men were individuals who gained prestige through gift exchanges. The more wealth a man could distribute, the more people were in his debt, and the greater his renown.

When these cultures were exposed to seemingly unlimited supply of goods for exchange, they experienced "value dominance". That is, they were dominated by others in terms of their own (not the foreign) value system.

The language (and potentially security) barrier prevented their understanding of the truth.

Analysis

Cargo cults are very diverse but they all share these 3 elements:

1. Value dominance: judging others by our own value system. e.g., if someone from Google preaches something, it carries more weight because we all know how hard it is to get a job at Google, and most people dream of solving Google-level problems and earning Google-level salary/profit. This is halo effect bias where one’s association (company, country, race, etc.) gives them an unfair advantage.

2. Shallow understanding: critical thinking is replaced by superstition and magical thinking. e.g., “best practices” are often someone’s interpretation of why something worked at a certain time and environment. This interpretation is often exaggerated to make a point. Naïve folks extrapolate these best practices to their own time and environment and hope to get the same result. Spotify model is one example, but this phenomenon happens for more “logical” ideas like design patterns (that sometimes conflict with each other: DRY, WET, KISS, TDD, …)

3. Imitation: going through superficial motions, processes, and rituals. e.g., SAFe is an elaborated process with many roles and rituals. It promises to make large software teams “agile” but in practice it’s yet another waterfall process packaged for profit.

Subscribe now

There are many factors that accelerate the effect of each element:

1. Value dominance: is boosted by name-dropping, slick presentation, and deliberate DSV (demonstration of high value).

2. Shallow understanding: is boosted by charismatic leaders, lazy practitioners, and a culture that has blocked feedback loops.

3. Imitation: is boosted by creating false hope, lack of education, and collectivism overriding individual identity

Put together, you get VSI:

Examples

Cargo culting can happen in various contexts, such as technology, business, or even education, where people adopt practices without fully comprehending their purpose and rationale.

As a reminder, these are the elements of cargo culting:

1. Value dominance

2. Shallow understanding

3. Imitation and rituals

Try to see if you can identify these elements in the examples below. 🙂

Cargo cult Programming

This was my entry to the term cargo culting, so it has to go first.

Cargo cult programming is a style of computer programming characterized by the ritual inclusion of code or program structures that serve no real purpose. —Wikipedia

Typical examples include:

Someone returned pumped from an inspiring conference about functional programming. They rewrote major parts of their codebase to gain hands-on experience at the company’s expense. In the process they introduce new bugs while making the code base harder to maintain for others who didn’t go to the same conference or weren’t onboard with the idea.

Inspired by an awesome YouTube talk and some hobby project experimentation, the team lead is convinced that Rust is objectively a better programming language than Java/C#. So they convince the PM to give them 3 months to rewrite a “legacy” system in Rust.

The team spends 6 months trying to rewrite everything in Rust only to revert the decision because of the steep learning curve, poor ecosystem, and the tool “not being there yet” for their use case. This is a case of tech bet where the investment wasn’t justified from the start.

Attending a local meetup, a few team members are totally sold on the idea that all their tech debt will be paid if they implement SOLID design patterns in their code.

They get to work and eventually spent more time on SOLID rewriting than actually fixing the original tech debt. The added value of SOLID was miniscule in their CRUD-heavy application. New bugs replaced the old ones.

The original master minds left the company and the recruitment bar was higher for new recruits. Abusing design patterns when it’s not justified is not limited to SOLID. I’ve seen abuse of other concepts like KISS (keep it simple stupid), DRY (don’t repeat yourself), WET (write everything twice), MVC (model view controller), the Unix philosophy (do one thing and do it well), etc. These are not objectively bad ideas but can be abused to feed premature optimization.

Frustrated with Amazon EC2, the DevOpsy people in the team convinced the non-technical leadership to migrate everything to Kubernetes (circa 2018). Things went as you’d imagine:

A software-supported company wanted to up its software-game to cope with competition. Guided by McKinsey, they decided to throw money at the problem and hire from Big Tech.

The company opened an office in London and started gulping in talent from Facebook, Google, and Amazon. The ex-big-tech employees started cargo culting rituals from their ex-employer: tech committees, platform unification, tech radar, tech standards, etc. which rapidly burned the budget without making the company closer to any of its objectives.

Not only the employees and the office were expensive but these things take time and require cultural transformation. And some of these initiatives downright didn’t work because big tech has fundamentally different competition landscape, revenue model, influence, talent bar, and investment model.

In this particular case, the company broke in two halves after the unsuccessful investment, and the ex-big-tech employees quit one after another.

The London office was shut down.

Recap

1. Cargo cults emerge when we overestimate the value of ideas by our own measures. We need to ask about the trade-offs. When hiring from big tech, be mindful about who you’re hiring. Many of these folks didn’t initiate the great work but were merely a bystander or practitioner. Be careful about the name-dropping effect and authority bias.

2. Understanding WHAT (they do) is important but don’t stop there. Dig deeper to understand HOW (it works) and WHY (they do that).

3. As a social species, we’re vulnerable to imitation and rituals. Monkey see, monkey do. But what makes us different from monkeys is critical thinking. As it turns out, individual differences are quite good at flagging pointless exercises. The problem is when leadership overrides these hints and boosts fake alignment.

Many people accept cargo cults as an inevitable part of tech work. If that were true, I wouldn’t write this post! My goal is not to rant and document the status quo!

Pro-tips

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

In the pro-tips, we discuss actionable advice to avoid, identify, and dismantle cargo cults in your organization.

Read more

These are common symptoms:

• The tight dependency between people carrying those titles leads to unnecessary blockage, inefficiency and friction. Any point of handover is fundamentally a risk for misunderstanding, delay, broken accountability, and service degradation. Those dependencies are reflected in the services provided by the organizations due to Conway’s law.

• When issues happen, the accountability is distributed across titles. This leads to finger pointing and blame game.

• Narrow titles typically reward narrow-minded professionals who only care about their perception. Any effort to diverge from this model, leads to stepping on toes and ensuing punishment.

• Many initiatives are multi-dimensional and require coordinating those titles to build a holistic perspective from multiple perceptions. This is much easier said than done, and typically leads to a meeting frenzy, design by communities, long documents, and expensive workshops and offsides which are expensive, time-consuming, and yield subpar quality (if they even reach the finish line).

• The duties that fall under a given job title may not qualify for a permanent full-time job. The ones bearing the title, may end up slacking (working less time than paid), or generating work: putting up unnecessary presentations, workshops, governance, demos, processes, etc. This is a desperate effort to create meaning out of something that’s incomplete by design.

Subscribe now

We’ve previously discussed this phenomenon at an organizational level. TLDR; teams that own end to end customer journey are less likely to suffer from broken ownership and accountability:

Not my job

Let’s just put some brain teasers before we go to the main discussion:

OK! You are ready now! Let’s discuss this…

Role vs title

For the sake of this article, we distinguish between the two:

• Title defines responsibilities towards an outcome with narrow ownership. For example:

  • Front-end developer, develops front-end and doesn’t necessary touch backend or do QA or DevOps. Those concerns are handled by other titles.

  • Project Manager, manages the project and leaves the customers, strategy, empowerment and other aspects to other people with appropriate titles.

• Role frames around the impact and broader ownership. For example

  • Software engineer uses engineering disciplines and methods to create technical solutions. That may involve coding, architecture, testing, operation, etc.

  • Product Manager effectively acts as a mini-entrepreneur for a product. That may formulating hypothesis, market research, collaborating with design and engineering, and even marketing and budgeting if the situation calls for it.

Title has limits while role’s boundaries are more fluid. You can think of the title as a piece of lego while the role can be thought of as a lego house.

Output, outcome, impact

If the point of a language is to communicate, it pays dividends to clarify what those words mean in this context.

At a high level, an organization has people and goals.

• People make an effort. For example, programmer code, or workers on an assembly line put parts together.

• The effort yields some output. The output is usually easy to measure. For example, number wines boxed by a worker

• The output builds towards an outcome that the organization cares about: For example, selling those products or releasing software yields to business profit. The outcome is for the business to make money and pay the individuals.

• Impact, connects the business outcome to a higher goal that the individuals can relate to. This is where mission and vision come to play. Not only it helps align the individuals but it also builds a collective story that connects them to the processes, goals, objectives, and strategy.

Title is typically not concerned with how the outputs add up to create an outcome or impact. This is partly due to the narrow ownership:

• Knowledge: title doesn’t have the big picture

• Mandate: title can only do so much without stepping on other toes

• Responsibility: many titles are responsible for one outcome

The outcome typically has a dependency on multiple titles.

Role on the other hand is more outcome and impact-driven. It has more mandate, requires less gatekeeping, and has clearer accountability.

Example 1: Product

According to Marty Cagan Product Management is about ensuring that what gets built is both valuable and viable. It requires deep knowledge of the data, business, industry, users and customers.

Some organization instead have these titles:

• Product owner

• Business developer

• Scrum master

• Release manager

• Product operations

• etc.

These job titles held by respectable professionals each devoted to do a great job within their tiny box in the overall value stream.

Call it collaboration or synergy.

But are each of those titles a full-time job? Because if they’re not, you’re sure going to see some slacking or generated work.

Take release manager for example. On paper, it’s about coordinating the effort required to release a new version of the software. But how often do you release? And what happens when the release manager gets bored and wants to add processes, friction, and get in the way to be noticed?

Stepping back a little: why would a release need a manager? What makes regular software engineers incapable of releasing their own code? Isn’t it a case of broken ownership?

There might be lots of reasons, but having a dedicated babysitter is not the best solution.

Another example: business developer! Isn’t understanding the business, customers, value stream, and connecting them to product a key task for the product manager? Why have a separate role for that? What should the product manager (PM) do then? Oh right! Manage the backlog! Say hello to the dreaded product owner (PO) or Project Manager!

How about product operations? If the goal is for the PM to have a product-informed assistant, just call it PMPA (product manager personal assistant).

The list goes on. Each title claims territory and protects it. It feels good to be SME (subject matter expert), a go-to person, a cog in a system, and “add value”.

Subscribe now

Example 2: Engineering

Software engineers are responsible for:

• Architecture

• Coding

• Testing

• Deploying

• Troubleshooting

• etc.

One would argue that these are different concerns and instead we should have:

• Architect

• QA for quality assurance

• Operations (usually called DevOps but practitioners miss the basic DevOps principles to break the wall between developers and operations)

• Infrastructure for integration, deploy, release, monitoring, and on-call

• Coder for typing the code, unit tests and debugging (also known as “code monkey” due to limited concerns)

• etc.

The problem is when these roles block each other and produce results that don’t fit.

For example:

• QA acts as a gatekeeper that cause unnecessary friction in the flow.

• Operations enjoys the learnings from the incidents without proper mechanisms to share that knowledge back to the development teams.

• Infra treats the developers as code monkeys and controls access to the infrastructure resources.

• Architects draw fancy boxes and arrows while adding friction with [premature-]“standards” and tech radars. We have a separate article for that one.

Developers are categorically smart and can pick up new skills given the chance. In fact that’s exactly what happened at one of my jobs when the company decided to deprecate the QA and DevOps titles in favor of handing those responsibilities to generalist software engineers.

The catch is of course, the cognitive load. It requires more time and energy to master all those areas and truly own the product.

With the advent of AI however, it is possible to quickly learn new skills and get copilot support.

As a company, you get what you pay for. Having all around senior engineers can improve the product quality and lifecycle, but those people aren’t cheap. You can either fill that role with multiple headcounts each with a dedicated title and concern, or have fewer generalists who are curious and aren’t afraid to pick up new skills.

Personally, I lean towards a 2-role model for engineering:

• Creative thinking: concerned with making it work (functional and non-functional requirements, testing, deploying, and being on-call)

• Critical thinking: concerned with what breaks the functionality (e.g. QA) and what to do in case of software failure (e.g. operations)

I don’t necessarily see them as two different types of engineers. Good engineers can do both types of thinking when needed but an individual typically is more effective in one of these modes.

Conclusion

The larger the organization, the smaller your seat. Larger companies (especially those which enjoy a high profit margin) are more likely to break a role into multiple titles. That’s one reason that enterprises are slower than startups.

When it comes to execution, people with diverse skillsets are typically more productive. One who can do design, code full stack, and deploy the code is generally more productive than a setup that requires 3 different titles to do those parts.

The feedback loop is often more rapid when there are fewer handovers. The ownership is clearer and there are less cracks for things to fall into. Moreover, there is less misunderstanding.

On the other hand, having dedicated specialists allows deeper specialization and typically improves quality at a higher price point.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

In 1992 Dewayne Perry and Alexander Wolf drew parallels between software architecture and building architecture.

They’re not the same!

Building architects are professionals trained in the art and science of building design. An architect is more concerned about how a building looks and feels than the engineering aspect of it —that job is left to the construction engineer.

Subscribe now

Example: Lotus Temple

Take the Lotus Temple in Delhi, India as an example:

The architect was Fariborz Sahba. This is an example of the model that architects build:

The purpose of the model is to communicate and visualize the design, study proportions, analyze light, and acoustics among other things.

Architecture is an art.

Construction/structural engineers translate that design into something that will be built:

Can an architect do construction/structural engineering? Yes, and the combination of two skills (art and engineering) would be great but that’s not how it is done typically. From Sydney Opera House to Burj Khalifa there’s an architect and a structural engineer involved.

UX

The closest match to the role of the building architect in the software world is the UX/UI designer.

UX as an art and discipline, is concerned with the look and feel and behavior of software.

Engineers bring the design to life. UX and engineers work together and it is possible for UX to have programming skills and vice versa (I call it UX Engineer back in 2015 and the term has caught on in recent years).

Software Architecture

In the world of software, there are many titles that carry “architect” in them:

• System Architect

• Domain Architect

• Solution Architect

• Application Architect

• Security Architect

• Data Architect

• Platform Architect

• Integration Architect

• Digital Architect

• API Architect

• Cloud Architect

• Network Architect

• Enterprise Architect

• etc.

Most of these roles are concerned with composition and topologies. In a different article, I argued that this concern alone is not a full time job, and practitioners tend to devolve into Ivory Tower Architect (ITA):

Alternatives

If architecture is only one part of what you do, there are better titles that communicate the diverse skills and value proposition.

Personally, I like the Staff+ titles (Staff/Principal/Distinguished/Fellow). Although Will Larson counts architect as an archetype for Staff+ roles in his book. I believe it’s an anti-pattern for the reasons that we touched here.

If you do only “architecture”, maybe you can consider “topologist” instead?

• System Topologist

• Network Topologist

• etc.

I know it doesn’t sound too common but it communicates what you do. If architecture is the only thing you do, it might be tempting to go with the Staff+ titles but please don’t. It just confuses people. Read more here:

• Introduction to Staff Engineer

• Senior vs Staff Engineer

• Principal Engineer

• Distinguished Engineer (upcoming)

The lack of the word “architect” in those titles is a signal that they do far more than that.

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

🤬 Call me names! Any curse would work. But not architect! 😱

That one hurts!

After 20+ years of software development, I have come to grow a specific despise for the title architect. Becoming an architect is one of my biggest professional nightmares.

That’s because people carrying this title typically:

1. Claim expertise over what should be in every engineer’s skillset.

2. Abuse power and cause friction.

3. Are avoided by skilled engineers.

In short: they are irrelevant!

The best phrase to describe them is ivory tower architects or ITA for short.

Ivory Tower (noun): a state of privileged seclusion or separation from the facts and practicalities of the real world.

This is a classic “monkey with a gun” scenario: mandate without knowledge or responsibility:

Architect is a misnomer in the software world. It's a common fallacy to compare it to a building architect. The closest role to the building architect is UX:

Do all architects eventually devolve to become ivory tower? Why having a dedicated architect is bad for the software organizations? What can you do to not become an ivory tower architect?

I’ll answer those questions but be warned that it’s not easy and involves actual work. Done right, your skillset will be so diversified that you may find it insulting to be reduced to an architect.

Another article elaborates why breaking one role into multiple titles is bad for quality, speed, and cost efficiency.

TLDR; architecture is one concern —one of many— for software engineers.

Symptoms

Ivory tower architects typically:

❌ Have beautiful charts and diagrams. Those boxes and arrows don’t map to the reality of how the system actually works. Some argue that this level of abstraction is what distinguishes a map from territory. But a map that is outdated and wrong, serves only one purpose: to confuse and eventually lead wrong decisions.

❌ Claim to represent the average engineer on the leadership table where in reality a tiny fraction of their time (if at all) is spent living the life of the average developer. Many developers rarely encounter them and when they do, they try to work around them.

❌ When you ask them a simple question, you get a complex answer back which doesn’t really solve the problem at hand. You ask them 2 + 2, they give you a lecture in algebra. They take the problem to their comfort zone instead of solving it. Sounds smart but not really the answer.

❌ They resort to rituals like tech committees which are objectively too expensive for what they deliver.

❌ They act like retired developers: heavy on opinions and “best practices” (which is someone’s interpretation of how things worked at a certain point in time, often glorified to make a point) while shying away from doing the actual work.

❌ When there’s success, they claim the credit but when there’s failure, they put the blame on engineers who executed their ideas.

❌ They’re not on-call for the systems they claim to own (this is an archetype of broken ownership).

❌ Instead of tackling the problem at hand, resort to name-dropping: “This is how we did it at [big tech]” (ex-Google? get over it). They have a tendency to cargo cult the rituals from a previous employer to the new ones.

❌ They demand that all decisions go through them and then delay the decision and objectively take wrong decisions due to lack of context and ignoring feedback loops.

❌ They engage in high visibility low-impact work like tech radars, engineering handbooks, and [premature] standardization without lending a hand to do the actual migration, operation, or refactoring.

❌ They have too much focus on best practices without context and a thorough understanding of what the tech they claim to own.

❌ Calling their engineer colleagues immature while conveniently ignoring their role to motivate the engineering leadership to invest in upskilling and optimizing organizational architecture.

❌ Design systems that look great on paper but are too flaky when implemented. That’s because they miss hands-on experience with the very artifact they’re designing for.

❌ See their engineering peers who are below them in the career ladder as disposable rows of an spreadsheet because “architect is only concerned with tech not the flesh”.

The benign type

While most architects do more harm by existing, there is a benign type that I just want to single out: the “wise” one who asks smart questions, answers simple questions, points people to search results, talks about the history, writes documentations, copy/pastes LLM results, holds presentations, organizes meetups, and generally tries to not get in the way.

I respect that.

It is still worth to evaluate their role in relation to their salary level but at least they don’t generate negative value by getting in the way of other engineers.

Staff, Principal, Distinguished

In recent years, Staff, Principal, and Distinguished Engineer gained popularity. Aware of the negative connotations of their role many architects disguise as Staff+. In fact, Will Larson counts architect as an archetype for Staff+ roles in his book. I believe it’s an anti-pattern for the reasons we get to.

This is not to say that all Staff+ engineers are ivory towers. In fact one of the beauties of not having the world “architect” in the title unboxes the Staff/Principal/Distinguished engineers to diversify their contribution. Architecture may be one thing they do, but they’re expected to do much more as we’ll discuss.

Becoming ivory tower is one of the most common pitfalls ahead of Staff+ roles. I’ve seen several senior engineers being hired/promoted prematurely to the Staff/Principal level where they don’t know how to handle the increased autonomy and mandate. As a result, with the best of intentions, they engage in practices that cause harm.

Unfortunately, they’re less likely to get feedback (or even care about it) due to the power imbalance that’s embedded in the career ladders.

Should engineers code?

Coding is the best way to earn street credit with other engineers.

Again, the higher autonomy of these upper IC roles is a slippery slope. It's easy to go to executive room and ask smart questions or even untie a technical problem and feel useful but I'm firm in my stance: many of these "tasks" are engineering leaders' responsibility.

IC's should be deeply technical and up to date.

Pairing a technical Staff with a non technical engineering manager is an anti-pattern. Staff+ should increase the execution bandwidth of technical engineering leaders, not to replace the need to have engineering knowledge.

3L

I got a good piece of advice a few years ago that was summed up as 3L: look, listen, and learn.

One can learn a lot by observing architects:

• Look: Ivory tower architects put up a show and engage in high visibility low value work (also known as preening). They're often insecure in their technical abilities and towards the end of their tech skill shelf life so they resort to politics to stay in power, instead of upskilling to stay relevant

• Listen: Ivory tower architects tend to use "engineers" to describe a group that does not involves them (e.g., "engineers implement my designs"). They're typically more fluent when talking about tech and that skill alone prolongs their career. That works because incompetent leaders easily confuse confidence with competence and eloquence with substance. ITA's employment opportunities however is typically limited to those kinds of environments (e.g. old school enterprise going through "digital transformation" hiring from big tech to save the day). If you listen to them, they're delivering value. If you listen to the "engineers" they haven’t heard of them or actively work around them.

• Learn: not much here. If you drag them to actual technical conversations you see their lack of competence in full glory. Skilled engineers avoid them. Usually ITAs have good social skills, so their presence acts as a classifier: they attract the junior type who likes to outsource thinking while simultaneously repelling the experienced ones who assess competence by outcome.

FAQ

Is architecture a full-time job?

In every organization, there are SMEs (Subject Matter Experts) who can be pulled to architectural decisions. Are there enough work to fill a full-time bandwidth? Even if there is, shouldn’t this work be distributed to a large group of people?

Does having a dedicated architect hinder the growth of the other engineers into this important matter?

When the architects turn what should be one-off consultations into a monopoly and demand that all decisions run through them, they create a bottleneck. Essentially, they cannot know the context and technical nuances of all problems to take objectively good and timely decisions. Centralizing the architecture concern takes away the growth opportunity from the rest of the organization to own their architecture.

Why do skilled engineers avoid Ivory Tower Architects (ITA)?

There are multiple reasons, which can collectively be summed up to: counter-productivity.

ITAs turn themselves into gate keepers for decisions, developers have to go through them. But the developers soon learn that they needed to educate the ITA due to lack of context, outdated technical skills, and world views that requires some unlearning.

This adds to the cost of the decisions. The pragmatic engineers learn to work around ITAs:

• Downplay the importance of a decision so it doesn’t end up on ITA’s table

• Implement the idea before it gets the chance to be decided

• Skipping the improvement altogether

• …

Do all architects converge to become ivory tower?

Doing architecture full-time starves the architect’s ability to update their technical skills.

It is hard to learn a skill without any requirement to use it hands-on and implement a solution with it.

One who voluntarily chooses to spend their time only on architecture gradually becomes blunt in implementation skills.

Excuses like “coding isn’t the best use of my time”, just communicate elitism and poor understanding of technical evolution.

…with all apologies to my many wonderful, highly skilled "Architect" friends, I tend to think it's a bullshit role. 🙃

I believe that only the people building software systems get to have opinions on how those systems get built. —Charity Majors

Read more:

• Charity Majors on Architects, Anti-Patterns, and Organizational Fuckery (blog post)

• Michael Nygard on architecture without an end state (podcast)

Personal Story

This essay may look like a personal rant about a specific person. It kinda is: I listed 12 actual examples of ivory tower architects that I've encountered in my own career over the years.

Coincidentally many of them are my own negative role models (the opposite of a role model, people you aspire NOT to become).

On a given day I may design systems, code, review PRs, measure system performance, use data to improve security, reliability, and scalability. I may spend time reading, doing POC also write or contribute to ADRs (architecture design records), tech strategy, hold knowledge sharing workshops for my engineer peers, and presentations for leadership. Interviewing, and improving our recruitment pipeline is also something I contribute to. These days I act part-time as a technical product manager for an internal developer platform (IDP) and am on-call for the tech that I’m accountable for. Sometimes I venture to areas that are formally outside my tech domain like organizational architecture because it impacts the health of tech which is my responsibility.

But it wasn’t always like that.

I was a simple front-end developer working with jQuery. I couldn’t deploy code. That was too advanced for me. I relied on testers to find my bugs. I was happy, they were happy, we all shared responsibilities.

That all changed when back in 2018 I was working at a company which due to lack of money decided to get rid of many roles like QA and DevOps (full story here).

I was horrified. Over night I got admin access to our cloud infrastructure and was put on-call.

Little did I know this inconvenience is going to catapult my understanding of software systems, SRE, quality, security, and reliability.

Me and my team learned by making mistakes until the company got us multiple formal training.

Subscribe now

Gradually we got hold of it. As it turns out, you don’t need all those roles to build software. All you need is generalists who don’t suffer from technophobia and are willing to learn. And the combination of having all those skills paid dividends: on a given day I would code, test, deploy, troubleshoot, and even manage product to some extent.

Once I learned how to learn instead of being scared, that trajectory continued to a Staff SRE role and later Senior Staff role (I wrote about my journey to Sr Staff here).

☝️Above, I wrote what I do these days. None of that spells architecture yet all of that impacts architecture.

The combination of a high level title, escalated autonomy, years of experience, and having the leadership’s ear is a huge abuse pitfall.

On multiple occasions I’ve had the chance to bloat my results and sell it as transformational initiatives, represent engineering voice to non-technical leaders, or shine a light at the end of a tunnel.

It is much easier to wing it up there, especially when the org is composed of tall silos with thick walls.

I resist the urge. Not because I’m some wise a** with a benevolent intention. But because the moment I take advantage of a broken system, I’ll be part of that system.

That’s why you see many architects stay at the company for too long! They don’t have anywhere better to be. Over the years, their technical knowledge level becomes too shallow that they’re practically unrecruitable to any company with a sane hiring process. Their career choice is limited to staying within one broken system or switching to another broken system.

I have a theory that everyone is in the best place they can possibly be. If you come across someone who complains and nags too much, it is often a symptom that they are trapped and got nowhere better to be.

Ivory tower may sound like an amazing place to be, but it’s a trap. Keep your axe sharp and build options. Options put YOU in power.

Pro tips

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

Let’s dig a bit deeper into:

• How to identify ivory tower architects and neutralize their negative impact?

• What is that negative impact of ivory tower architects and how to reduce that?

• How not to become an ivory tower architect?

• How to hire/promote senior IC (individual contributor) roles like Staff, Principal, and Distinguished Engineers while minimizing the risk of creating an ivory tower architect?

How to identify ivory tower architects?

We touched on some symptoms above, here are more with reasoning and examples:

• Spending more time in meetings than coding. Senior IC roles require up to date, hands-on experience in order to effectively deliver impact. For example, when LLMs were new, I invested from my own pocket to try Github code pilot to use it on my open source projects to learn about its limitations and strengths (before it was even allowed at my current company).

• Interacting more with leadership than the engineers. When interacting with engineers, mostly doing so with a handful of their favorites (e.g. technical committee) than all the engineers in their org. For example, when rolling out SLOs across my org (100+ teams), I could ask leadership to demand SLOs from all teams. Instead, I systematically took the time to meet every team at their own pace, learned about their system architecture, and together with their consumers set their service levels. I learned a ton along the way which I write about here.

• Relying on mandate and policing than empowering others. Engineers are smart and respond to logic. Mandate is not a replacement for that. A good example happened at one job where there were multiple processes for incident management. This apparent inconsistency, hurt the ability to general central incident reports. An ivory tower architect borrowed the mandate of upper management to kill all those processes but one. There was resistance, politics, and essentially the engineers didn’t buy into that unification effort. I saw those extra incident processes as a symptom. The real root cause was that the official incident handling process was cumbersome and had poor tooling. By removing those pain points with the feedback from teams, I created a process and tooling that was more reasonable to its users: engineers.

• Not being on-call either because the system is too complex or it’s beneath them. If EM is responsible for engineers and PM is responsible for product, Staff+ is best described as “Tech Manager”. The health of the tech is primarily the accountability of the Staff+ role (for organizations that have this role). Being on-call helps gain first hand experience with how things work and break. Plus, making system reliability a Staff+ problem motivates them to use their elevated mandate to actually do something about it.

Overall, turning ivory tower is the biggest risk ahead of senior IC roles and it takes deliberate effort to not let the higher position in the career ladder detach you from the other engineers.

What is that negative impact of ivory tower architects and how to reduce that?

• By assuming mandate over the architectural decisions, they break ownership. Just by mere existing and assuming architectural responsibilities, you limit the ability of other engineers to grow their system design skills. A better alternative is to diversify your role by contributing to code (e.g. refactoring) while participating in architecture discussion on a case-by-case basis. Rest assured that the organization doesn’t burn into flames if you don’t babysit every single decision. Mistakes aren’t bad. That’s where most of the learning comes from. It is better to stay within a safe distance and offer a helping hand when needed instead of proactively storming into every room and demand that decisions go through you.

• Having a high level abstract view of the systems and organization often leads to bad decisions that look nice in theory but create a mess when they go through. Modern software is complex. Human systems are even more complex. It takes time and effort to get a good understanding of both to an extent that allows informed decisions. A trick I use is to limit my vertical scope. For example as I said I am responsible for over 100 teams. There is no way in hell I can know all the systems made by such a large organization. Instead, I focus primarily on reliability engineering aspect and offload the other aspect to others. Even within reliability engineering context, I use SLIs and SLOs in a specific way to let the teams and their consumers decide what to measure and how good is good enough. Following my own definition of ownership, I don’t claim mandate where I don’t have responsibility.

• Illusion of knowledge is worse than no knowledge. Architects often rely on out of date system design diagrams and technical know how. This outdated knowledge —illusion of knowledge— can lead to bad decisions that sometimes work against the very purpose of having an architect. The truth of the matter is that modern systems evolve rapidly and new technologies solve old problems. It is better to honestly say “I don’t know” and use that knowledge gap to learn, than just wing it with pretty diagrams and beautiful charts.

How not to become an ivory tower architect?

• Be on-call: many job ads for Staff/Principal/Distinguished engineers require on-call. But many companies implement these roles without any on-call responsibilities. It is important to be in the loop when an incident happens even if you’re not the primary responder. It helps gaining experience about a system’s weak points and be more deliberate when it comes to improve reliability, security, and general health of the tech —which is your responsibility.

• Be available: it feels good to go to a meeting room with high level leaders and talk about important stuff. But don’t mistake that for impact. Most probably customers don’t pay the company for those meetings but the product. Stay close to the product and product developers. Once you try that, you may realize that your title and reporting line may get in the way of feedback loop. Do everything necessary for engineers to know that you’re one of them. Any minute spent working with other software engineers is a minute that makes you more efficient when working with leadership to drive high impact decisions.

• Stay up to date: I can’t stress this enough. The primary value of IC roles is their technical depth. You need to actively spend time learning about new technologies to be able to assess which ones can solve the old problems more effectively. But I hear you, there’s not enough time to learn everything. ThoughtWorks tech radar is a good example of how to allocate your energy. No need to learn everything with equal depth. Only the parts that relate to your area is enough. One way to learn deeper is to do some hobby projects or proof of concepts, but be careful not to fall in love with your hobby project and push it to production by force. A tip is to try to teach what you learn. That way, not only you spread the knowledge and demonstrate good technical leadership but you get to learn from all the questions that the “students” may ask.

How to hire/promote senior IC (individual contributor) roles like Staff, Principal, and Distinguished Engineers while minimizing the risk of creating an ivory tower architect?

When hiring for or promoting people to senior IC roles like Staff/Principal/Distinguished engineers, look for these signals:

• Only architectural work: Architecture is not a full time job. It’s part of the duty of software engineers. Give equal weight to actual contributions to the source code.

• Try to understand their motivation: you can ask it straightforward, but it’s better to ask indirectly and read between the lines. If their motivation with senior IC role is to avoid work, it’s a red flag. If they are motivated by leading others and multiply their impact through follow-me leadership, coaching/mentorship, while working shoulder to shoulder with other engineers, you got the right person. A good signal is to ask about the build and deploy process for one of the products or details about one incident. This is where you want to ask a lot of follow up questions instead of jumping to different questions. The deeper you go, the better signals you get about one’s technical literacy, contribution, and motivations.

• Sound opinions: there’s no perfect tech. All technical decisions are trade-offs. Try to understand how they made high level technical decisions? What feedback loops they used? Did they go big bang or gradually rolled out the new decision? How did they measure success and why?

• Up to date knowledge: over the past years, system design interview became more established and preparation material became more available. As a hiring manager or member of the promotion committee, you cannot assess the qualities you don’t understand. So take some time to familiarize yourself with modern system design. You can subscribe to or for a weekly dose of system design newsletter for example. This is very important because we as humans have a cognitive bias to confuse confidence with competence. Without having a good knowledge about architecture, you’re more likely to hire/promote incompetent but confident candidates to senior IC level with elevated mandate. They can cause serious harm before you know it. Do your homework first.

• Beware of the Staff anti-pattern where a skilled engineer assumes the knowledge while an incompetent leader has the mandate. This is the sign of a broken ownership. I have not seen a single such leader voluntarily do what’s best for the org and get the hell out. So I’ll just put it there to raise awareness. If you’re not technical enough to lead an engineering organization, actively coach your senior ICs to create the next generation of leaders. Or you can honestly ask their help, do the work, and gain the technical skills. It’s not rocket science. Many software engineers don’t even have an engineering degree. If you really want to learn, you can do it.

If you find this post insightful, please share it in your circles to inspire others.

Share

Share or discuss on Hacker News

1. Service reliability is directly impacted by the shape of the organization.

2. Consumer journey is the key to designing better organization architecture.

If you are curious or have any doubt about those points, read that article first:

This essay goes through the 2 most common organization architectures, their key characteristics, pros, and cons:

1. Kebab: The teams are structured based on the capabilities they provide. The consumer journeys go through the artifacts of multiple teams like a kebab stick. This model favors autonomy and loose coupling. The risk is handover.

2. Cake: The teams are structured based on the consumer journey. They manage cognitive load through layers of abstractions. This model favors end to end ownership and reducing handovers. The risk is cognitive load.

We evaluate both approaches through the lens of consumer journeys using the same example: a news web site.

The goal is to help engineering leaders make deliberate decisions when designing the shape of their organizational architecture.

Note: Team topology is concerned with how the teams interact with each other and what is their area of ownership. Org structure is defined by reporting lines. The two influence each other to the extent that bad org structure hurts team topologies and vice versa.

About the example

To make this post more approachable, we use an example organization that is behind a news site with the following capabilities:

• Front page: the landing page of the site which lists the latest articles

• Article page: renders the text, images, ads, and other media

• Search: allows the readers to search for articles and sort or filter the list of results

• Payment: allows the readers to pay for a subscription

• Identity: allows the readers to log in

• CMS (content management system): used by the journalists to create the content

Notes about the news site

1. This is a simplified example. In practice, there may be other features like CDN (content delivery network), SEO (search engine optimization), tracking, data analytics (for various recirculation techniques like “most read”, “top articles”, etc.), DAM (Digital Asset Management), Android and iOS apps as well as IoT (internet of things. e.g., digital billboards).

2. We also leave the editorial, marketing, legal, sales, accounting, customer support and other functions out of context. Our focus in this article is software organization although much of the learnings can be applied to other types of service providers. That is because regardless of the implementation details, most businesses offer a service that has a provider and a consumer. And that service provider/consumer model works surprisingly well when applied internally.

   

3. There are SaaS solutions that provide all these “digital capabilities” to a team of journalists who want to monetize their content. This works well for smaller news sites, but the larger ones prefer to build their build their own news platform for 2 main reasons:

   1. Flexibility (tell compelling digital stories without the limitations of any generic platform)

   2. Independence (from other tech companies to be able to criticize them when needed).

4. I picked this example because I had spent 5,5 years working for 25 brands at one of the largest media companies in Scandinavia at various functions. I can speak fluently about both software architecture and organization architecture. In short: this is my comfort zone! 😁

Kebab org

In the kebab organization, teams are organized based on the capabilities that build the consumer-facing service.

We can start by mapping the capabilities above to an org structure like this:

If you squint, you can already see some silos and that’s not by accident. Whether intentional or not, “divide and conquer” is a strong theme in kebab organizations.

Kebab teams are sometimes called functional teams. I believe that’s an unfortunate naming because the organization as a whole is often dysfunctional as we’ll see in the cons of this model! 🙃

This type of organization is very common in large enterprises which are going through the so-called “digital transformation” (which is fancy language for catching up with innovations to survive) for a few reasons:

1. This aligns well with the traditional grouping of people by function: e.g., legal team, accounting team, sales team, etc. It is common to shape the teams after the technologies they work with front-end team, android team, etc. This is due to the fallacy that software developers are unable to learn multiple technologies good enough. It also has some base in truth because the more junior talent defines themselves by their tools (e.g., “I’m a java developer”) rather than their profession (e.g., “I’m a problem solver, I just happen to use code”).

2. This model may feel familiar for companies that primarily produce physical goods. For example, a truck manufacturing company may treat the user like a truck that goes through a production line. In this analogy, each team is adding some value to the user journey the same way workers assemble a door or add a coat of paint to the truck. This model fundamentally shatters when you consider the fact that humans and cars are fundamentally different: humans are unpredictable, have free will, and in this case are the ones who buy the product.

3. Management consultancy companies (like McKinsey) make good money selling slides to these kinds of businesses due to their fat pockets, desperation, and clueless leadership

4. It makes sense to call themselves “head of XXX” because they assume the rest of the organization is just body-parts to them. For what it’s worth, they might not be wrong: someone high up has them at choke hold for any risk taken and any mistakes made, so they have to be on top of everything and kebab is the best way to “divide and conquer” this problem.

The services offered by the kebab teams are narrow by definition and rely on handovers to complete a consumer journey.

Essentially the consumer journey has multiple dependencies.

In an ideal world, handovers wouldn’t be a big deal. In practice it creates a fragile consumer journey because:

• Kebab teams are more focused on their features than the holistic consumer journey. These are best described as product parts at best. It is common for members of kebab teams to have an identity crisis: “I don’t know where my contributions fit in the grand scheme of things”.

• Humans are prone to miscommunication, bias, delay, and territorialism which inherently reflects in their creations. Kebab organizations ship their org structures to the consumer.

• The accountability of the end-to-end consumer journeys are distributed over a large organization surface. This makes failures both hard to track and tricky to resolve. Hanovers create ownership cracks in consumer journeys. Incidents love to fall between those cracks and play hide-and-seek. This feeds a culture of finger pointing and blame.

• Implementing new consumer journeys requires orchestrating multiple feature teams and often ends up taking an unreasonably long time. The quality is often sub-par as the teams try to shoehorn the new feature requests into their existing understanding of their feature and how it fits the holistic consumer journeys.

In one sentence: kebab could work if it wasn’t made of human flesh! 😖

Take the Article team for example:

The Article team is only concerned with rendering the article based on business logic. It is deliberately agnostic to how the user ends up in the article page.

As far as the Article team is concerned, their job is to render an article —which can be quite a complex task, given all variations of screens, media content, internet connections, they have to support.

The article team is primarily focused on output rather than outcome. The output is the rendered article. The outcome is for the consumer to be informed, feel engaged, and hopefully spare a few bucks for a subscription.

Feature factory is a term popularized by . In a feature factory, the emphasis is on continuously delivering features without a deep understanding of their impact or value to users. This approach often leads to a relentless cycle of producing and shipping features based on a backlog, with little consideration for whether these features actually solve user problems or contribute to the overall cohesive product's success. John’s articles are linked at the end of this one.

Sure, they’re aware of other feature teams but for all practical purposes, they may as well not exist. Members of the Article team may even think of themselves as the core of the business: there’s no news site without well-rendered articles after all!

If something breaks, it is usually not the Article team’s fault (of course! 🙃).

• If the front-page or search teams end up sending the wrong query parameters to the article page, leading to a broken render it’s their problem.

• If the article URL changes as a result of updating the content in the CMS, leading to 404 error for links from social media, it’ll probably go unnoticed for too long because well, whose problem is that even? Who is even monitoring that? We should be lucky if an employee (e.g., an editor) notices that. In some cases, the users (usually paid users) may contact customer support to inform them. That’s because the responsibility of the Article team in this example stops below the URL level.

• If the identity or payment team changed their interface as part of an internal refactoring or bug fix, you guessed it! It’s their fault. 👌

You get the idea. The consumer ends up suffering, but no single team formally cares. At best, leadership is aware of it and maybe even throws in some process and extra meat to the machinery hoping to make it work.

The meat here refers to additional headcount for central babysitting teams:

• UX

• QA

• Incident Operations

Keb-UX!

The easiest way to hide away the visible side effects of shipping the org structure to the consumer is to hire a central UX team.

In reality the UX team is more of a UI team due to being detached from any feature team.

This is sad because the UX researchers potentially have the ultimate wisdom to define the shape of the org (reminder: consumer journey comes from user journey which is a UX concept).

QA-bab!

Integration is the Achillis heel of the kebab teams. Those capabilities developed in isolation don’t suddenly self-align in production.

It takes a lot of hard work to make sure that every release fits what’s in production and doesn’t break any consumer journey.

That’s where the QA (quality assurance) team comes in. This team usually ends up acting as black-box testers with various ratio of automation and manual tests.

Note: black-box testing is intentionally blind to the implementation details whereas white-box testing is aware of them. White-box testing is usually done by the developers (shift-left model) whereas black-box testing is usually cheaper because it doesn’t need a hand in the code and can even be outsourced.

The QA team often gets in the way by acting as a gate keeper. Teams are not allowed to ship production until they get the QA-blessing. 🎖️

Due to the limited bandwidth, the QA team may need to prioritize other work. This creates bottlenecks, blockage, and further frustration.

Unfortunately throwing in more money and increasing the headcount of product teams doesn’t improve the velocity.

KebOps!

Handover, integration, and misaligned autonomy. What could possibly go wrong?

Interestingly, as mentioned in the previous article, the incident metrics are early warnings for poor organization architecture: TBF, LT, CFR, TTR, incident impact radius, incident cost, manual/automatic ratio, false pages, etc.

Unfortunately, incompetent leaders focus on fixing the symptoms instead of the root cause.

Hiring a central operation team to babysit the teams relieves the leadership from some of their “duties” and streamlines what’s worse about the broken ownership.

• Just like the QA team, the Operations team becomes a bottleneck

• Just like the QA team, the Operations treats the artifacts of the feature teams as black-box. This negatively increases the number of false alerts and the time it takes to identify the right team in case of failure.

• Even when they find the right team, incident resolution is often not the top priority and may drag weeks or months. That is because kebab teams don’t truly own the consumer journey, every incident is a new reminder that their priorities doesn’t align with the consumers.

• The Operations team can potentially get access to good quality metrics (e.g. CFR and manual/automatic ratio) but typically end up measuring vanity metrics like incident count and MTTR.

Incidents are more common at the point of handover and are harder to resolve due to complexity, lack of accountability, and higher rate of overall change.

Putting it all together, we get something like this:

That is running business as usual.

Things get even more hairy when you’re trying to add a new feature to the site!

Kebab “PM”

Adding new consumer journeys requires coordinating a bunch of kebab teams and the project (not product) takes much longer than anticipated:

Since project management got a bad rap, some companies hire program managers, but that doesn’t take away the fact that it’s a waterfall process with a lot of team-chasing, rank-pulling, fake deadlines, late night crunches, surprise blockages, and unnecessary frustration for all parties involved.

All of those are symptoms. The root cause is the fact that the consumer journey has multiple dependencies to different teams:

• There are hidden team dependencies (which stems from the fact that architecture diagram is a graph, while the org structure is a tree, as pointed out in the previous article)

• It’s hard to grab the attention of all teams at once (they each have their own backlog, priorities, understanding of the domain, and tech stack)

• The initial shape of the problem evolves —as it is common with SDLC (software development lifecycle). What makes it particularly hard is the balkanized nature of the kebab teams which makes them slow to adapt to new consumer journeys or drastically changing the existing ones.

Let’s recap.

Pros of kebab org

1. Allows specialization to address complex requirements

2. Specialization also enables more junior and less all-around talent to deliver some output regardless of the outcome. Kebab orgs can take advantage of temporary (but expensive) workforce like consultants.

3. Kebab’s autonomy allows the teams to pick the right tool for the job (as long as it fulfills the requirements, it doesn’t matter how it works).

4. Kebab’s loose coupling allows managing multiple backlogs independently. Although in practice, release management is a dedicated role in these organizations due to the risk of these individual pieces breaking the whole. CD (Continuous Delivery) becomes CO (Continuous Orchestration).

5. Separation of concerns promotes creating clear interfaces between deliveries of different teams (e.g., API, SDK, queue, etc.).

6. The cost of each feature is easy to calculate: Add the head count and cost of operation to get a good estimate of how much a feature costs. Unfortunately, this data is less actionable. The cost of a capability in isolation is as good as knowing the price of RAM in a PC build. Buying cheaper or more expensive RAM doesn’t impact the performance of the PC without understanding and considering how the RAM fits the rest of the components.

7. Kebab puts more power in the hands of the engineering leaders. If you enjoy micromanagement and firefighting, this is the ideal situation. You will act as the focal point for the end-to-end consumer journeys despite having no hand in the code. This is a classic archetype of broken ownership of course.

8. The individual capabilities often enjoy higher quality due to dedicated team and smaller scope which translates to more focus.

Cons of kebab org

1. Handovers make the consumer journey fragile. Incidents are more common, and they often have more severe consequences because they take longer to resolve.

2. Each kebab team has a piece of the consumer journey puzzle. A simple change from the business perspective may require coordination across multiple teams with subpar velocity, reliability, and cost.

3. Integration of all sorts (UI, API, SDK, etc.) becomes a challenge and may incur additional cost both in terms of money, but also quality, and velocity (due to gatekeeping and mandates that are spread across different roles).

4. The headcount of the kebab teams is usually fixed. When the work needed for a capability is less than the available work bandwidth, there’s a risk of fake work or slacking which typically goes unnoticed. For example, kebab team may add frameworks, tech bet, premature optimization, and architectural “best practices” where none is needed.

5. Due to the smaller team size, management is less demanding, but the org chart is tall. The combination of more “free” time and longer chain of commands can create a vicious environment where politics is rampant and clarity scarce.

6. There’s a risk of fragmented tech landscape which results in the business paying for multiple solutions to the same problem. The fragmentation also reduces resource mobility (e.g., a Java developer is less likely to join a team that uses Go). The teams may think they’re “picking the best tool for the job” where in reality, standardization could reduce the cost of operation and increase the opportunities for collaboration.

   Standardization isn’t a silver bullet. I have a dedicated upcoming article about the risks of premature standardization.

OK, enough bashing kebab. It’s actually one of my favorite foods! 😋

Cake org

In the cake organization, teams are organized after consumer journeys. The goal is to minimize handovers and maximize end to end ownership of those journeys.

Each team caters to the needs of a class of users. For our example news site, it may look like this:

• Reader experience team is responsible for everything on the site from the front-page to the login screen to the payment flow

• CMS team is responsible for all the editorial needs including authentication and “admin” site usage

What’s the catch? Cognitive load.

Cognitive load

Cognitive load was coined in the 80’s by John Sweller as the total amount of mental effort being used in the working memory.

More recently it became popularized by Matthew Skelton and Manuel Pais in Team Topologies.

In its modern usage, cognitive load roughly refers to the amount of expectations a team or individual can handle without burning out!

Obviously, cognitive load varies between individuals and teams based on their skillset, technical/domain experience, tools (AI anyone?), and collaboration efficiency, among other things.

To keep the cognitive load at a manageable level, cake organizations use layers. Each layer has clear consumers above them.

• The web platform provides basic constructs to build a news website. They provide a well-tested flexible platform in close collaboration with the reader experience team.

• The payment and identity hide away the complexity of securely logging in/out the end user and handling payment as well as query the payment status.

• The content API team abstracts away the content databases for various types of media which is used by both the CMS team and the reader experience team.

If there’s enough work and opportunity to unify (e.g., kubernetes), there may even be an infrastructure team:

Again, this team has clear consumer journeys:

• Consumer: Developers

• Consumer journeys: provisioning resources, handling credentials, accessing documentation, setting up observability, etc.

• How? It depends. Some teams use golden path, some use automation, while others use serverless. The exact implementation is beyond the scope of this article, but the main point here is that there’s no infrastructure team until it is really needed.

Note: a common “platform” team frustration is not being able to define their product and missing out on all the goodness of being product-minded, ideally with a dedicated product manager. Starting with the consumer (in this case developers) is a key to define the identity of the platform team.

The top-most layer has the end user as their consumer while themselves acting as a consumer to the services provided by the lower layers.

Just like kebab, there is a focus but this time it’s horizontally across consumer journeys. For example, this is how the world looks like from the reader experience team’s perspective:

And this is what the CMS team is concerned about:

This may look a lot like a kebab organization. There are interfaces, different teams, and dependencies. There’s even a handover between the reader experience and the CMS team!

So, what changed apart from rearranging some rectangles on the screen? 😂

Let’s take a short fika at a Swedish cafe!

The primary point of interaction with customers is the server. Server finds a table, takes the order, spits in the food, serves it, handles the payment, and takes out the dishes, …

Servers don’t do all of that on their own. There’s an army behind them:

• Internal: chef, cleaning staff, the person buying the ingredients, accountant, etc.

• External: the payment provider, farmers growing the ingredients, transport, etc.

The servers act as a layer of abstraction above all of those lower layers of services.

Even inside those lower layers, there are other layers:

• The farmer growing ingredients may rely on retailers to sell them.

• The cleaning staff may not actually be hired by the restaurant but rather work for a cleaning company that provides cleaning services across multiple sites

The point is: we use these abstractions in real life all the time where one party is accountable for an end-to-end experience acting as an aggregator of lower-level services.

Note: the law of leaky abstraction states that all non-trivial abstractions are leaky to some extent (they cannot hide what’s behind them entirely and all the time). In this example, if you find a hair in your food, you may learn that the restaurant uses a bunch of suppliers, but the ultimate accountability is with the restaurant and that’s an important detail.

When the complexity grows, instead of breaking the consumer journey into multiple product surfaces, new layers are added beneath the consumer journey team to provide services to them.

Adding more layers to handle cognitive load is just one solution. There are other ways:

• Investing in education to improve the team skill level (e.g., training, online courses, newsletters like this one, hackathons, etc.)

• Investing in tooling (e.g., managed observability instead of building your own, Copilot for coding, etc.)

• Hiring more senior generalists (increases the headcount cost but the cake machinery more efficiently takes advantage of their skillset)

Pros of cake org

1. Consumer journeys are less likely to break because there is one (or as few as possible) accountable party. It's much easier to identify the responsible party when an incident happens. The cake organization often doesn’t need a central operation organization because the teams are in the best position to spot and address any issues along their consumer journey.

2. Fewer dependencies and handovers along the consumer journey prevent ownership cracks where incidents fall into. Less “not my problem” and finger pointing, more end to end ownership.

3. Clearer service provider/consumer relationship (both internally and externally).

4. The teams are often cross-functional containing all the required roles (e.g., UX, frontend, backend, DevOps, etc.).

5. There is less (or no) need for a central QA team as the developers take ownership of the end-to-end consumer journey.

6. The teams focus on outcome instead of output, product instead of project, and systems instead of components.

Cons of cake org

1. Higher cognitive load. In my experience it leads to larger teams. While it translates to more work for the EM (engineering manager), it also means less people in the org are managers and more are creating value for the customers.

2. Layers create dependency. I haven’t seen more than 2 layers in my career, but I believe anything more than 3 internal layers is absurd because the lower layer teams are too far from the end consumer journeys (AKA customer journeys). In general, you should avoid dependencies between teams as much as possible for reasons that should be clear in this article.

3. Layers of abstraction are prone to the law of leaky abstractions.

4. Cake teams rely on lower layers to provide a good service (that’s where Service Levels come in handy). If a lower layer provides poor service, it often negatively impacts all the layers above them. As a rule of thumb put your best people in the key teams that:

   1. Have more business impact (usually closer to the consumer)

   2. Are the internal dependency for more impactful teams

5. There’s a risk of tight coupling across a consumer journey. For example, the Reader Experience team may use the same frontend and backend for serving the news and handling payment and identity. The latter two are much more security sensitive and require much stricter technical choices. One could argue that the rest of the news site could enjoy the same level of security but take it from someone who’s spent 2-3 years on each side of the wall: they’re essentially different types of systems with completely different NFRs (non-functional requirements).

Analysis: which one is best?

Cake and kebab represent extreme ends of a spectrum. In reality, most organizations contain a bit of both —a hybrid if you will.

As elaborated previously, ownership has 3 elements:

1. You cannot be responsible for something you don’t control. You need the mandate.

2. You cannot use that mandate effectively over something you don’t understand. You need knowledge.

3. You gain knowledge only if you are fully responsible for the consequences of your mandate.

Both models define ownership but with drastically different concerns and outcomes:

• Cake teams own the consumer journeys which are tied to business objectives

• Kebab teams own the features, capabilities, components, product parts, or sub-services that tied to the system architecture

Both models optimize for something:

• Cake optimizes for end-to-end consumer journey ownership ensuring that there’s one accountable party for each type of consumer using clearly defined services.

• Kebab optimizes for autonomy and specialization ensuring that each component has a high quality in isolation.

Both models try to manage the cognitive load of complex systems:

• Cake orgs use abstraction layers to scale vertically and may end up with large teams and shallow hierarchies.

• Kebab orgs use compartmentalization to scale horizontally and may end up with smaller teams and deeper hierarchies.

Both models require talent that fits the expectations:

• Generalists thrive better in cake: due to the ability to deliver a wide range of consumer journeys. Cake teams are often cross-functional with T or Π or even M shaped talent (people skilled in a broad range but deep in 1, 2, or more areas).

• Specialists thrive better in kebab: due to expertise in one domain or technology that’s required by the capability. Kebab can accommodate one-trick ponies better which reduces the average cost of talent. SMEs (subject matter experts) may miss the broader picture limiting their contribution.

In my experience, two organizations that did roughly the same thing (a content-heavy website with identity, payment, CMS, CDN, apps, etc.):

• Cake org was smaller (around 300 engineers) and had a higher engineer/manager ratio (around 12)

• Kebab org was larger (around 1000 engineer) and had a lower engineer/manager ratio (around 4.5)

The biggest difference is the heuristics that guide the organizational architecture:

• Cake is organized outward: optimize towards the needs of the consumer at risk of higher cognitive load for internal teams.

• Kebab is organized inward: optimize towards the specialization of internal teams at the risk of handovers.

Before you go and turn your organization upside down, I have one last thing:

Every reorg is a trade-off. It makes some things easier and some things harder. It’s important to be aware of what needs to be easier and what needs to get harder in your own business model, which is influenced by talent pool, budget, competition, etc.

I often explain to company leaders that it’s tempting to think that a reorganization will fix their company’s issues, but in most cases, it’s simply rearranging the deck chairs on the Titanic. —Marty Cagan

Further reading

• General management, functional, and hybrid models: Which org design works best for top companies?

• Service-Based Architecture as an Alternative to Microservice Architecture. Neal Ford via Matt Fletcher, InfoQ

• Functional Organization Structure: Pros and Cons, Indeed

• What is a matrix organization and how does it work?, Asana

• The Product Model and Org Design, Marty Cagan, SVPG

• From Projects to Products, Marty Cagan, SVPG

• Product vs. Project Teams (2021) and Product vs Feature Teams (2019), Marty Cagan, SVPG

• Team Topologies, Martin Fowler

• Service Design 101, Sarah Gibbons, NN/g, 2017

• 12 Signs You’re Working in a Feature Factory and a revision 3 Years Later,

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

Recently I went down in working hours and salary by 10% to be able to spend more time learning and sharing my experience with the public. You can support this cause by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.

You can also refer your contacts and gain free subscriptions. Thanks in advance for helping these words reach further and impact the software engineering community.

Last week the recording was finally published on YouTube.

You can watch the whole thing or skip to the parts where I talk:

• Introduction

• Advice for interviewing for Staff Engineering

• Importance of diversity for leadership positions

• My open TODO list to crowdsource accountably (I’ll write about it in another post)

• Balancing soft skills and hard skills (referring to T-POP)

As always, I’d be curious to hear your thoughts and reflections and if you find it inspiring, please share it in your circles.

Subscribe now

I’m also active on LinkedIn where I share original daily tips about technical leadership, reliability engineering, and growth. Follow me here.

In beyond Staff Engineer, we went through the differences between Staff Engineer and Senior Staff Engineer. The key takeaway from that article is that Senior Staff Engineer is like Staff but at scale.

What about the level that comes after it? Different companies have different names, but in this article, we refer to it as Principal Engineer.

The word "principal" in English means chief or foremost. It is derived from the Latin "principalis," which means first in importance or chief. This term is used in various professions to denote someone who holds a primary position or an elevated level of authority and responsibility.

Right off the bat, let’s clear out what principal engineer is NOT:

• PE is not to act like a school principal and make rules or scold other engineers.

• PE is not above other engineers. Using the paygrade card is a symptom of weakness and shows that all other influence techniques have failed.

• PE is not a soft role where the majority of time is spent in meetings and talking to people. As engineers, PE’s have a duty to keep their technical “axe” sharp by being hands on.

Just as going from Senior Engineer to Staff Engineer required a new skill (soft skills), the Principal Engineer requires a new skill: business skills.

But what exactly is business skill? How can you master it? And why is it important for an engineer to think about business?

I’m going to use this career ladder which I helped develop:

First, a few caveats:

• Theoretically the career ladder maps exactly to the skill level. In practice, however, there are many flaws:

  • The recruitment process is broken

  • The applicant's negotiation skill or the urge to recruit them (e.g. when the company is trying to steal talent from rivals) can put someone in a higher level than their skill

  • There's no promotion process

  • The promotion process is broken

  • The promotion committee is biased

  • The perceived impact is different than the actual impact

  • There may be personal favors or grudges at play

  • You may need to have for “years” even though year is not a unit of measurement for experience

• The number of levels in a given ladder depends on the size of the company. While startups suffer from the title inflation phenomenon, large enterprises may not have enough titles to provide adequate resolution to signify different expectation levels.

• Some companies (e.g. AWS) don’t have Staff Engineer in their career ladder and instead call it Principal Engineer. Regardless of the name, we’re talking about a level that’s 2-3 levels after Senior Engineer.

• Hard, soft and business skills are not exclusive. A junior engineer may have a great sense of the business, it just may not be a key part of their job (i.e., it usually doesn’t impact their hiring). Moreover, a Principal Engineer without up-to-date hard skills (e.g., programming) is just as irrelevant.

• Promotions often lag behind your demonstrated skills.

• Some companies treat the career level as pay-bands not a way to set examples, expectations and responsibilities. For example, if someone’s salary expectations are higher than the Senior Staff level, they may get the Principal Engineer role even though their skill level and eventual impact is at the lower level. This happens for example when the company badly wants someone (e.g., to steal key people from competitors and temporarily cripple their ability to ship products).

• Being a level above Staff Engineer, the Principal Engineer has:

  • Even longer strategic time horizon

  • Even larger organizational scope and areas of responsibility

  • Even higher risk of becoming ivory tower and irrelevant! 😉Preening is high visibility low-impact work. Due to the visibility of the role Principal Engineers are even more susceptible to this kind of task than Staff Engineers.

  • Even more need to communicate complex technical topics to non-technical audience and vice versa (communicate complex business topics to technical audience)

  • Even more reliance on alternative leadership methods without formal mandate

  • Even more T-POP 🍭

What are business skills?

In simple terms, if the code is a technical solution to a product problem, business is how that product makes money. It involves:

• The customers, their needs, alternative choices, and key metrics

• Market trends, competition landscape, marketing, and the impact of external factors like economy, politics, etc.

• The revenue stream, payment/subscription/sale flows

• Strategic investments or decommissioning of technical solutions regardless of their engineering ingenuity 😢

• The legal and compliance aspects as well as how to enforce them in technical solutions

More specifically, we’re talking about how the company makes a profit: the difference between what the customers pay and what the business has to pay to run (e.g. cost of personnel, cloud, manufacturing, equipment, tax, office space, coffee machines☕).

Most engineers aren’t concerned with either revenue or profit. Some are concerned with revenue, but only a few understand and influence the profit.

There’s nothing unique about the business skill per se, but what really gives the Principal Engineer a superpower is to connect the business to the technical solutions:

How does Principal Engineer connect business to engineering?

• Being the voice of the customer in the engineering room and the voice of engineering in the business room

• Aligning business strategy, product strategy, engineering strategy, and technical strategy

• The ability to create new technical products that address novel customer needs

• Defining the technical optimization priorities (e.g. Service Level Indicators)

• Cost optimization to map the business investment into different areas to the return of investment (ROI)

• Defining and refining the non-functional requirements (NFR) of different parts of an architecture (e.g. amount of load expected on an online toilet paper shop right before the pandemic)

You may argue that every engineer has these responsibilities to some extent and you are correct. Just as soft skills are not exclusive to Staff Engineer, business skills are not exclusive to Principal Engineer either.

Expanding the diagram that was used in a previous article, all career levels require that skill to some degree:

However, just as soft skills are key to being an impactful Staff Engineer, business skills are key to being an impactful Principal Engineer.

How to speak business?

Just like any skill, business is learnable.

Some would argue that soft skill is something you’re born with, but as an introvert who works as a Senior Staff Engineer, I’d argue otherwise.

It’s about finding the right balance. You certainly don’t need to be as business heavy as someone who does it as a full-time job without having to bother with technical and engineering aspects (e.g. CEO), but you need to be much better than the average Senior or Staff Engineer.

1. Develop Business Acumen

• Learn the Business Goals: Understand the company's overall strategy, target market, and financial objectives. This will help translate technical decisions into business impact. This part takes time, especially if you’re recruited as Principal Engineers instead of internal promotion. 3L is by far the best advice I got for this phase.

• Business Skills Education: Take courses or attend workshops on business fundamentals like finance, marketing, and product management.

• Business Reading: Stay updated on industry trends and read business publications relevant to your company's domain and industry. You may find it daunting to go too deep into this area while keeping your main advantage (technical literacy). Again, it’s about finding a balance, not being perfect in an area you’re not formally trained for.

• Strategic Initiatives: Volunteer for cross-functional projects that involve both technical and business teams. Many years ago, I volunteered to build a tool for the legal team at a company, which catapulted my understanding of the legislative aspects of that business. I have to admit the initial learning curve was steep because we didn’t even have a common language at the start. 😄I was talking Markdown and Mustache, they were talking PDF and price interpolation!

2. Communication and Collaboration

• Learn to Speak the Business Language: Learn to communicate technical concepts in clear, concise, and business-oriented terms. Every company has its own jargon and acronyms when talking business. Literacy in this language distinguishes someone who is struggling to make sense of how money is made vs. someone who is fluent and confident with business.

• Regular Interaction: Schedule meetings with business stakeholders like high level product managers, sales, marketing, and customer support teams to understand their objectives, challenges and opportunities.

• Shadowing: Shadow business leaders to observe their decision-making processes and gain insights into business priorities. As a principal engineer, you may get easier access to the C-level room or organizational leadership (depending on the size of the company). You may be invited to represent the engineers in the leadership room, but it’s an equally important opportunity to learn how “businesspeople” talk and what are the key criteria when taking decisions.

3. Technical Expertise in a Business Context

• Focus on "Why" Before "How": When proposing technical solutions, explain how they address specific business problems and contribute to the overall goals. Arguably this is a skill that comes handy at much lower levels in the career ladder, but for Principal Engineer, it’s crucial to have a solid understanding of the business and use that understanding to influence technical decisions. For example, the NFR (non-functional requirements) of a given architectural problem is notoriously hard to figure out without knowing the size of the market, and external factors.

• Cost-Benefit Analysis: Evaluate technical decisions considering not just technical feasibility but also factors like development time, maintenance costs, and return on investment (ROI). Again, this is something you need to think about at lower levels, but for companies that have the role of Principal Engineer, you’d expect to find solid answers from them.

• Metrics and Measurement: Translate Technical Metrics into Business Impact. Align technical efforts with measurable business outcomes like customer satisfaction, revenue growth, or cost reduction. My favorite tool is Service Levels because it aligns autonomous teams in a measurable way.

• Regular Reporting: Create reports that highlight how technical projects, initiatives, strategies and improvements impact business metrics. Present these findings in terms understandable to non-technical stakeholders. Personally, I find this aspect the most interesting part of working with metrics. Sometimes, the act of trying to generate a metric reveals that I was measuring the wrong thing. A good rule of thumb is to ask is it an output metric or outcome metric? Output is all about WHAT we do.
  Outcomes align with WHY we do what we do. It's a layer deeper. Unfortunately, the output is often abused as a proxy metric to measure outcome because impact is hard to measure. Especially when

  • Impact isn't clearly defined

  • Impact is a delayed metric

  • Impact isn’t quantifiable

  • Leadership would rather chase a vanity metrics and have the illusion of control

4. Focus on Customer-Centric Development

• Engage with End Users: Understand the customer's perspective by participating in user research (UX), customer feedback sessions, or even direct interactions like customer support calls. Regardless of if your company has a B2B or B2C model (or both), there’s someone who pays the money (stakeholder) and someone who uses the technical solution (service consumer). Being the customer’s voice is a superpower in the engineering and high-level leadership rooms. Personally, I was fortunate enough to work as a UXer early in my career which later led me to work as a front-end developer. I may not do user studies every day, but that period (and self-learning) helps me ask the right questions and confidently speak on behalf of the consumers or stakeholders when they’re not in the room. You don’t have to be a UXer, but having a basic understanding of the concepts goes a long way. My favorite book on the topic: don’t make me think

  

• Adopt a Problem-Solving Mindset: Approach technology from the standpoint of solving customer problems, which often leads to innovations that drive business growth.

5. Operational Efficiency

• Understand Cost center vs profit center: Knowledge of how different business operations work and how technology can improve efficiency and effectiveness. touched on that back in 2022 (paywalled).

• Cost optimization: optimizing workflows, reducing costs of operations (cloud anyone?), and improving service delivery through technical solutions. Like any optimization effort, cost optimization requires good data. If you don’t have data, start there. See if you can gather accurate data about the cost of running the technical solutions but also how that cost maps to the revenue generated. Once I was doing this exercise and found an entire team of consultants who’ve been paid for 10 months to maintain a system that got a grant total of ZERO request per month! The service was sunsetted, so either they didn’t get a memo or decided to keep their mouth shut and wallets full. 💸

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to days to draft, edit, rethink, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.

You can support me by sparing a few bucks for a paid subscription. As a bonus you get access to my WIP book Reliability Engineering Mindset. Right now, you can get 20% off via this link.

I had the chance to talk to Tobias about:

• Rolling out Service Levels across a large organization: the WHY, WHAT, and HOW

• Platform Engineering and SRE

• Reliability Engineering Mindset

🔉Listen the podcast on your favorite platform:

Episode #98: Service Level 101 feat. Alex Ewerlöf - Sr Staff Engineer @ Volvo Cars

Here’s a little snipped with video if you want to see how my beard moves when I talk! 😄

In other news, I went down in salary and working hours from first of April.

Since December 2023 I've been using my vacation days to be able to work on Service Level Calculator, my book, newsletter and a hands-on learning plan to catch up with the biggest change in our industry: AI.

I have also been working almost every evening and weekend which has been stressful and unsustainable because my extracurricular activities collide with my duties as a dad.

The downside of using vacation days is that the expectations don't change even if you're working less hours.

So, I decided to go down in salary (which also impacts my pension, bonus, and available PTO) to buy back some time. The idea is to combine this time with my vacation days to effectively gain 20% for extracurricular activities without having to sacrifice the time I should be resting and being with family.

I may have gone down in salary and working hours, but my expenses are higher thanks to inflation. Regardless, I believe this is the right call because self-learning, writing software, and books leads to long term growth.

If you believe that my open-source software and/or writings are good for the community, please consider a paid subscription to my newsletter.

And a huge thanks to those of you who support me already 🙌

There are also other ways to support me spending time on this:

• Sharing the posts that inspire you within your circles to spread the message and inspire others.

• Giving feedback to help me grow from your experience. My perspective is limited by my experience which is shaped by my professional journey. That's why this newsletter is named after me and not some generic phrase that promises to be an authority for a topic. Let me know your reflections in the comments (or if you got it by email, reply it).

Thank you.

• You may have a hard time making sense of technology and solution landscape

• Once you do make sense of it, you may have a hard time driving change

• Once you do manage to change it, you may have a hard time operating it

• Once you do succeed to operate it, it may not align with the evolving product problem definition

Over the past few years, I have changed jobs multiple times and faced one or more of those challenges. Reflecting on those experiences I created a mental model to systematically approach self-onboarding to technical positions. This model surprisingly works well after the onboarding as well to decide how to balance spreading energy and time. I call it T-POP!

In this post we’ll go through:

• Why do we need a mental model like T-POP?

• What is T-POP?

• How to use T-POP?

• What will happen when this model is broken?

Why do we need a model?

Technical leadership roles (i.e., Staff Engineer, Principal Engineer, Enterprise Architect) on the IC (individual contributor) track are unique in multiple ways:

• Lack of mandate: unlike EM and PM who have a mandate over engineers and product, technical leaders are supposed to “lead tech”. But tech doesn’t talk or listen. To do their job effectively, tech leads must go through people and products.

• Wider scope: covering a wider tech landscape, product area, and organizational surface calls for a different toolset than preceding IC roles (e.g., junior/senior engineer) that are usually attached to one team, product, or technology (e.g., frontend, Android, etc.). You must accept that you can’t master some technologies and instead you need to prioritize other aspects to deliver value.

• Longer horizon: Technical leaders need to think about the longer-term impact and technical strategy while simultaneously connecting it to the tactical.

Without a systematic model, the following are the top risks that threatens technical leaders:

• Ivory tower: going back to the origin of the term “staff”, your primary masters are the engineers at the leaf nodes of the organizational tree. Those are the people you’re supposed to empower, and you can’t do that without understanding their struggles and living their lives. Ivory tower is a state of privileged seclusion or separation from the facts and practicalities of the real world which is a real risk if you spend most of your time in the leadership room and neglect the engineers you’re supposed to serve.

• Snacking and preening: The leadership roles usually have a lot of autonomy. If not managed properly, the autonomy can quickly lead to wasting time on the wrong thing (snacking). Even worse, it can lead to working on things that have high visibility but no value for the business (preening). Will Larson did a great job explaining this phenomenon here.

• Premature solutions: due to higher visibility and scope, many decisions may come your way. Sometimes organizational or product problems disguise themselves as technical problems. Without a solid understanding of the tech landscape and your role, it is tempting to respond with premature solution. A good example is premature standardization where the tech lead demands tech defragmentation without understanding the root cause of fragmentation in the first place or offerring a hand in doing the actual migration. I have a dedicated post for premature standardization in the future.

Subscribe now

What is T-POP?

Broadly speaking, there are four categories that define an environment:

1. Tech: the actual technical solution in place and its maturity in terms of applications, DevOps, security, reliability, architecture, 3rd party tooling, etc.

2. People: the org structure and the type of talent making up the organizational DNA: the behaviors the company tolerates, the style of leadership and engineering, the talent pool, the talent intake & recruitment filters, mentorship, etc.

3. Operation: the way of working (WoW), processes, history, facts, incentive models, vision & mission, culture (how people act when they’re not watched), assumptions, and beliefs (or FAB as John Cutler puts it). We’re not only talking about DevOps, SecOps, MLOps, ProductOps, OpsOps (!) and what not, but any way of working that is unique to the company. Processes often are light on WHY and heavy on WHAT. They naively assume that you can “program” people to follow “best practices” (which is a generalized term for something that appeared to work at a given circumstance and point in time). Path dependence theory states that the history of the organization puts constraints in its future trajectory.

4. Product: the context of the problem the business set out to solve and its business model and customers. The tech you’ll be leading tightly couples to the problem it’s trying to solve. That problem is defined by the product and business model.

Put together, you get T-POP: 🤤

You can see those four dimensions clearer when starting a new technical leadership position, but even if you’re promoted internally, these areas become more visible with the elevated level.

As a technical leader, tech is your primary focus but make no mistake: it is far from being the only one. Tech doesn’t exist in a vacuum. It is heavily influenced by people, operations, and product.

How to use T-POP?

Self-onboarding

When you’re new, you often know one of the 4 dimensions. Usually, it’s familiarity with the tech that got you through the interviews.

• People: Sometimes you know the people or have good domain knowledge.

• Operation: Sometimes you’re hired for your skills to help improve how the business operates.

• Product: Some other times, you may be hired for your prior experience with the product (e.g., you’re joining a competitor).

Regardless, use your initial edge to expand into the other dimensions. For example, if you know the tech very well, try to meet people who use that tech, have meaningful conversations about tech, and expand your network (People dimension). Learn how that tech is used, deployed, monitored, and operated. While doing that, try to understand the problem as defined by product because tech is only valuable when it’s the solution to the product problem.

Distributing time and energy

It is easy to hide in the tech cubicle and geek out but that’s not the best use of the tech lead bandwidth, nor does it cover the entire possibilities and expectations.

• People: In Senior to Staff+ we established that soft skills are key to your success. From networking to dotted reporting lines to internal training, you need to spend the time to build a solid people foundation to deliver technical impact. The networking aspect alone lubricates everything else because it makes it easier to navigate the org chart.

• Operation: How the organization operates plays a key role in the health of the technology. Understanding how things operate prevents unnecessary friction (some fights can wait for another day).

• Product: The evolution of the technical solution is tightly coupled to the product. Like all things in life, it’s about balance. Balance is hard when those dimensions are not clear.

Maintain Focus

At a given time, you’re probably working on 1-2 specific initiatives. You can use T-POP to focus on relevant tech, people, operations, and products.

• Tech: No need to spend time trying the latest ECMAScript state 1 proposals if you’re working on a C# code base.

• People: No need to keep a monthly 1:1 with a dotted reporting line that doesn’t have anything to do with your current project.

• Operations: No need to focus on optimizing CI/CD pipelines or picking up a discussion on data governance if you’re optimizing the incident handling process.

• Product: No need to spend time improving the user flows in the app if you’re focusing on improving the backend payment API.

The point is: although T-POP calls for attention in multiple dimensions but the focal point for all your investments at a given time should is to execute and deliver your current initiative.

Measuring impact

As a leader, your impact will be measured by the outcome of the initiatives you drove or contributed to across your organization and company.

It’s too simplistic and naïve to think that it’ll boil down to technical initiatives just because your title has “tech” in it.

Your brag document (a log of your impact over time) should contain all aspects of the POP, not just the T:

• People: You help your organization resolve a people-issue (politics) disguised as a technical issue (e.g., data ownership).

• Operations: You help your organization to adopt and use ADR to push decisions closer to where they are executed

• Product: You build a POC (proof of concept) to demonstrate how a new feature in the mobile operating system can open a new revenue stream for your company.

Recruitment

I have interviewed many Staff Engineering candidates and can confidently say the number 1 reason for rejection was lack of a well-rounded skill in leadership.

The Staff+ interviews are focused on hiring a well-rounded technical leader, not just someone who codes better and faster. As I argued in a previous article, a Staff Engineer is not necessarily more technical than a Senior Engineer.

A good interview process is mindful of all these aspects and assesses them adequately:

• Tech: The candidate can demonstrate depth of understanding technical landscape and navigate complexity at scale.

• People: The candidate can identify and work with key stakeholders and other engineers to deliver impact.

• Operation: The candidate is literate with how modern software is operated and how to scale a piece of software when needed. Moreover, the candidate has a track history of improving operational efficiency and effectiveness.

• Product: The candidate can see the big picture and connect the result of their efforts to the impact on the business model and vice versa: use business data and product insight to calibrate the technical solution.

Promotion

I have mentored a few Senior Engineers and those few who made it to the Staff Engineering level were well-rounded technical leaders, not just hard-core tech savvy programmers.

If you want to grow in your career, whether you're a Senior Engineer or Staff, make sure that you gain experience in all those dimensions not just tech.

• People: network internally and build a personal brand so when it’s time for a promotion, there are enough data points to build a positive history for you. It can be as simple as technical demos, blogging, organizing hackathons, or lunch & learn to inspire others.

• Operation: try to leave a mark on how your organization operates day-to-day. It can be as simple as rolling out async work culture (e.g. RFC, ADR, TDD, etc.), implementing an A/B testing tool to promote testing live, or internal developer portal (IPD) to improve efficiency. Look at the operational problems, develop a hypothesis, validate it with key people, and deliver incrementally in a measurable manner.

• Product: understand how product measures success. has listed tons of metrics here. Once you identify the key metrics, see how you can use your technical edge to deliver a measurable impact.

I’m a big fan of brag docs (a record of your accomplishment) not only for promotion and job interviews but also to feel good about what you have achieved. It also gives a trajectory to your growth.

Broken T-POP

It wouldn’t be a good model if we couldn’t remove something from it, without breaking it!

Let’s see what happens if one element is removed from T-POP:

• Tech: I haven’t seen a technical leader who abandons the tech altogether, but a common pitfall is to prioritize other dimensions and have a shallow understanding of tech which in turn hurts your impact. Let’s be clear: as a technical leader, tech is your home and that’s the primary area you should influence (hence the dash in T-POP).

• People: Computers are logical, predictable, and fast creatures compared to people. It is easy to abandon the people dimensions as an IC (individual contributor). As we argued before the people factor is single handedly the most key factor that blocks the promotion of great Senior Engineers to the Staff level. Leadership by definition is about delivering an impact that is larger than one person can deliver. Metcalfe's Law, states that the influence of a network is related to the square root of the network's total nodes. Any tech lead who abandons networking, avoids dotted reporting lines, skip levels, mentorship, sponsorship, and other means to scale their leadership severely limits their impact.

• Operation: Ignoring how the company operates and the shared stories and rituals that shape the interactions and connection between people and systems is a huge mistake. Too often I’ve seen great initiatives being blocked because “people don’t get it”, “this company sucks”, “upper management is stupid”, “we’re not mature”, etc. That’s usually the sign of something trying to improve the tech without understanding how the company operates. “Best practice” from one company may not work in another. You have to understand the system before trying to change it and even then, you should do it gradually and systematically. You’ve probably heard “culture eats strategy for breakfast”. It certainly applies to tech strategy too. The ways of working at a company can create a tremendous friction that fights anything you’re trying to change. Seek to understand before trying to change and even then, do it in iterations.

• Product: Many software engineers see their job as builders. But builders of what exactly? Tech is a solution to a problem and without understanding the problem, the solution will be inefficient if not flat out inappropriate. You cannot blame them though. When learning software, many of us only had to deal with well-defined abstract problems. We’re even tested for it when applying for jobs. In the real world, problems are hairy, obscure, and act like a moving target. It is easy to dismiss product as “not my job”, but sadly this is the most important aspect (after the tech) to do your job as a tech lead. That’s because leadership needs vision and you cannot have a vision that’s fake and disconnected from product (i.e. how money is made).

Conclusion

I developed this mental model a few years ago when I changed jobs, but it matured over the last 2 years when I worked as a senior staff engineer. Regardless of how much I’ve thought about it, it’s still limited by one person’s limited experience and professional life trajectory.

I’d love to hear your thoughts and reflections. With your feedback I can enrich it even further and make it useful to a wider range of technical leaders. 🙌

A follow up article digs into how T-POP can form the questions that you can use to self-onboard to a new Staff+ position:

My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to days to draft, edit, rethink, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. You can support me by sparing a few bucks for a paid subscription. As a bonus you get access to my WIP book Reliability Engineering Mindset. Right now, you can get 20% off via this link.

If you can’t spare the money for whatever reason, sharing this article within your circles would also help. Thanks in advance.

This one is going to be an entry level post with lots of pointers to other resources. Think of it as a landing page for your curious neighbor! 😊

More realistically, however, you may find clarity here because ambiguity is part of the definition:

“Staff is more of an attitude than a role” — as a former Google Staff Engineer puts it

There even are books written on the topic:

2 years ago, I was the first Staff Engineer to start at a company going through digital transformation. The folks who hired me came from Google and Spotify where this title is well-established, and I had worked at Staff capacity for years prior.

That didn’t stop funny encounters like this:

— “Where can I find the other Staff Engineers?”, I asked during my onboarding.

— “Oh, we have tens of thousands of staff, and many of them are engineers”, someone answered without missing a beat

As the Senior flavor of the Staff I also had a hand in shaping our Staff requirement process. I also mentored a few great senior engineers on their way to becoming Staff. I felt obliged to clarify this title, especially the expectations. Internally I published multiple write ups and created a dedicated Slack channel for it. Externally, I have published multiple articles about technical leadership.

That right there is the first cue!

Staff Engineer is a technical leadership position.

But why do we need a new title then? What does it mean in the context of an engineering organization? For that we have to look at where the word “Staff” comes from.

Where does the word “Staff” come from?

The term “staff” is used in the military to promote great soldiers as role models without giving them extra command and control responsibilities.

Staff officers assist the commanders in planning, analysis, and information gathering. The commanders have authority and accountability over their subordinates. Staff officers don’t. —Beyond Staff Engineer

Let’s use a simple model for a typical engineering organization:

• Engineers build a technical solution (e.g. a microsoervice, API, frontend, app)

• The tech solves a product problem (e.g. an App for booking a cab)

The Product Manager (PM) impacts the product. The Engineering Manager (EM) impacts the engineers. Staff Engineers impact tech, but tech doesn’t talk or listen. They have to go through both people and products to deliver their impact.

Why isn’t the Staff Engineer in the engineering team? That’s because the EM is supposed to be technical enough to cover both the engineers and tech in the team. Having a Staff Engineer in the team is an anti-pattern even though the Tech Lead is one archetype of Staff that operates in one team. By the way I’ve shared why these archetypes can also be anti-patterns when literally treated as job titles.

Inside autonomous teams, the engineers are expected to fully own the product and its technical aspects. Anything else is the sign of a broken ownership.

The real value of Staff Engineer comes from operating across the organization:

When do autonomous teams need alignment? Let’s look at some examples:

• The app team wants a certain computation to be done server side whereas the API team believes it should be done client side

• A new service is being created but there’s dispute about which technology to use, how it should interact with existing systems, what should be the functional and nonfunctional requirements (NFR), and which team should own it.

• Two product teams want to implement the same feature but slightly differently in a way that doesn’t make sense for the company

• Too much autonomy creates a fragmented tech landscape which makes it extremely hard for the platform team to support a “golden path”

• In the absence of a tech strategy, the teams fall back to “keeping the lights on” without any clear path to significantly improve the product or technology

• …

These are all problems that could use technical leadership. By leadership, I have a very specific meaning in mind.

Good leaders:

1. Are hand-on and practice follow me leadership instead of ordering people.

2. Have a vision that is larger than one person can deliver (due to time, depth, or scope).

3. Can unite people towards that vision and can bring the best out of the people (engineers in particular respond logic and depth of argument more than any title or rank)

Great leaders:

1. Can effectively work with people more skilled than them. They actively seek to be the dumbest person in the room and include a healthy dose of opposition to see their blind spots

2. Create new leaders (e.g. through delegation, coaching, mentorship, sponsorship)

3. Can efficiently scale their impact radius across a larger group over time (when the scope grows to hundreds of people, it calls for different toolset e.g. write ups, videos, etc.)

This description fits both Engineering Leaders and Staff Engineers on two different branches of modern engineering job ladders:

How can the impact of Staff Engineers compare with the Engineering Manager when the latter has a mandate over engineers?

It really shouldn’t come to rank and mandate.

Weak leaders pull their rank to the conversation. They may not even talk to people "below them".

Unfortunately, those are the most dangerous leaders because they'd do anything to protect their rank. Strip them from their rank and they're average (or below). They know that and that’s why they treat their rank as their life depends on it. The rank serves as a wall or facade between them and others. Plus, the pay is good! 🤑

Great leaders don’t pull ranks. They talk to everyone humbly because they know that for their vision to be realized, they depend on serving every single individual. They're the ones that if you don't know their rank, you'd still follow them because they are competent and genuine human beings.

So, if it’s not the rank, what is the difference between Staff Engineer and EM?

• Depth: EMs spend a large part of their time in 1:1’s, coaching, planning, and administrative activities. Staff Engineers don’t have any of that. What are they expected to do with the extra time in their calendar? Get really deep in the technology to be able to lead the engineering organization to a better place

• Scope: the true value of Staff Engineers comes from tackling large technical problems across a large organization. There are tricky discussions that need to be had that a higher rank and deeper understanding of the tech can unblock. However, the company can go really far by hiring technically competent engineering leaders, especially when the technology isn’t that sophisticated to call for a specialized technical leader.

Staff Engineer is an Engineer! Duh!

Staff Engineers are primarily engineers serving as a force multiplier for other engineers. Their true masters are the engineers on the leaf nodes of the organizational tree.

Ivory tower Staff Engineers are detached from the realities of day-to-day life of the very engineers they are supposed to empower. They spend most of their time in meetings with other leaders. Their perception of the tech comes from EM, PM, and other technical leads.

Who is closer to tech than engineers? And who can have an informed opinion about the tech without touching it?

Don't let the title build a wall against who you can meet. It's not "Staff vs Engineer"!

I operate across a large organization. Does that make me a Staff Engineer?

The scope of operation is one part of the equation. But that's not the only thing that defines Staff Engineer. Some roles by nature require an IC (individual contributor) to interact with many teams. For example:

• Application Security

• Site Reliability Engineers

• Platform Engineers

• Integrations engineers (CDN, GraphQL, API gateway)

• Tools engineers (eg. 3rd party tooling governance

• Common components teams (identity, data, UI libs, etc.)

If everyone working at those roles was automatically considered a Staff Engineer:

1. The recruitment bar for those positions would be unnecessarily too high (Junior SRE would not exist!)

2. The pay would follow the title and make some value propositions unnecessarily expensive.

3. The Staff Engineer would lose its meaning (if everyone is a Staff, no one is Staff)

While the engineers operating at the team level are primarily concerned with tactical and short-term initiatives, Staff Engineers should connect tactical to strategic initiatives across a larger organizational scope.

To lead the organization, the Staff Engineers need to have a clear picture of where it needs to get to. Their concern is beyond one team (e.g. the platform team) and benefits the whole organization. Their definition of success is measure at the org level not at the team level.

Sometimes, Staff Engineer is confused with architect. While architect is an archetype of Staff, being hands on is a requirement:

The Architect role tends to evolve in relatively large companies, companies with exceptionally complex or coupled codebases, and companies that are struggling to repay the technical debt they created in their initial sprint to product-market fit.

There is a toxic preconception that Architects design systems in isolation and then pass their designs to others to implement.

Influential architects dedicate their energy to maintaining an intimate understanding of the business' needs, their users' goals, and the relevant technical constraints.

They use that insight to identify and advocate for effective approaches within their area of focus and do it with organizational authority that they've earned by demonstrating consistently good judgment. —Will Larson,

Staff Engineers don’t have any mandate over people or products, yet their entire impact depends on both. In a future article I’ll discuss some of the tools I’ve picked up along the way to lead without a mandate.

Subscribe now

Recap

Staff Engineer is a senior-level technical leadership focused on high-impact engineering projects and technical strategy over a longer time horizon.

Staff engineers possess deep expertise in their domain and are responsible for leading complex engineering initiatives, mentoring other engineers, setting technical standards, and ensuring the quality and scalability of the software architecture.

They often serve as a bridge between the engineering team and management, translating business objectives into technical solutions. Their responsibilities extend beyond coding to include architectural design, technology evaluation, and contributing to the broader technical vision of the company.

You can get far without a dedicated technical leader role, and that’s why many companies don’t need Staff Engineers at all, if they have done their recruitment right and don’t waste their engineering leader bandwidth with administrative work.

Further reading on the topic of Staff Engineering:

My monetization strategy is to give away most of content for free. However, these posts take anywhere from a few hours to days to draft, edit, illustrate, and publish. I pull these hours from my private time and weekends. You can support me by sparing a few bucks for a paid subscription. Right now, you can get 20% off via this link. If you don’t want to spend money, sharing it with a wider audience also helps. Thanks in advance.

The candidate was confidently unprepared! Honestly, the thought occurred to me that the TA (talent acquisition) was trolling us! 😄

The interview was scheduled for an hour. After the first couple of questions, it was evident that we were not going anywhere.

On paper, the candidate had a MSc degree in computer science (earned 2 decades ago) and had spent a substantial chunk of their career as a software team leader, scrum master, and “engineering manager”.

We followed a script to keep the candidate's experience consistent. The interview specifically started stating that we are interested to hear about the candidate’s practical experience with technical leadership, not how it should be done theoretically.

For almost all technical leadership questions, the candidate’s answer was a variation of delegating to the tech lead, “architect”, or staff engineer.

Bottom line, the candidate was a people manager or team admin, not an engineer who leads other engineers.

As part of a larger organizational shift, we were very strict about hiring EMs who are technical, hence this technical leadership interview.

We didn’t have a dedicated Staff Engineer for each team. Most of the value of Staff Engineering comes from operating across teams and filling the gaps in between as I have elaborated in a recent article:

Solely relying on Staff Engineer for technical matters is an EM anti-pattern as we’ll get to. It’s an example of breaking one role into multiple titles.

Subscribe now

I couldn’t just interrupt the interview abruptly —I was representing my employer after all and had every intention to make it a positive experience regardless of the outcome.

So, I did what I had never done before:

I really appreciate giving us your time. We follow a script with a set of questions to keep the experience consistent for all candidates. So, I’ll go through a few more questions, and you can ask us to skip them if you don’t have personal experience in that area. —paraphrasing what I said

And so, we spent the next 10 minutes with “skip”, “skip”, and inadequate answers.

At that point, it was obvious that we could not continue this role-playing game. 20 minutes in, we went directly to the part where the candidate asks their questions:

I don’t have any questions as it’s obvious that I’m not the type of EM you are looking for, but I have an observation to share. You guys reached out to another EM-colleague of mine to recruit her, but as soon as she heard that the recruitment process has a technical leadership interview, she declined. I was the brave one and out of curiosity decided to go through the process.

Why is it so important for you to hire technical EMs?

My response on behalf of the company was of course more brief and mellow but I took note of sharing it publicly because it’s an important issue.

Subscribe now

Non-technical EM

Not all EMs are created equal. Every person has a different background, interests, and professional life trajectory which exposes them to different environments and shapes their growth and belief system.

In his seminal article, Pat Kua identifies 5 archetypes for engineering managers:

1. Tech Lead EM excels at technical leadership guiding the team in making sound technical decisions, mentoring engineers, and ensuring the quality of code and architecture.

2. Team Lead EM focuses on team dynamics and collaboration fostering a healthy team culture, managing conflicts, and facilitating effective communication.

3. Delivery EM is all about execution and delivery, ensuring projects are completed on time, managing dependencies, and optimizing processes.

4. Product EM bridges the gap between engineering and product, collaborating with PMs, defining roadmaps, and aligning technical efforts with business goals.

5. Lead of Leads EM oversees multiple teams and EMs, setting the vision, aligning teams, and driving organizational change.

Tip: Regarding archetypes of other roles, I’ve written about SRE and Will Larsson about Staff Engineer.

The problem with non-technical EM

Personally, I have had my share of non-technical managers and the frustration as an engineer:

• They don’t understand engineers’ world. They act as a shepherd where no shepherd is needed! They are babysitters where there are no babies. Most engineers are clever and logical. They prefer an engineering leader who can meet them where they are and understand their technical challenges.

• They default to using their “team building” and “conflict resolution” tools even when the disagreement between engineers is purely logical and can be resolved for example through creating a POC (proof of concept), writing down an RFC (request for comment), or implementing ways to gather data.

• They have no way of understanding code first-hand and solely rely on their favorite developer (who is usually promoted to tech lead or unofficially acts like one). This alienates the rest of the team.

• When an incident happens, they get in the way instead of meaningfully contributing to identification of the root cause or implementing a resolution.

• When they get access to “the room” (where the decisions are made), they cannot do a good job defending their team or acting as their ambassador. As a result, unrealistic expectations will be set for the team, or their great work will go unnoticed. That’s because they heavily rely on someone else to act as a translator between what’s being done and why it is impactful. They cannot acquire this knowledge on their own.

• When it’s time for promotion, you cannot count on them due to lack of understanding your contribution. A Life Engineered, did a video about this being one blocker for promotion to staff.

• When communicating technical topics, you have to dumb down the issue often to a level that it loses context (admittedly communicating technical topics to non-technical people is a useful skill, but you can’t expect every team member to have that skill just to support an incompetent person in the position of power).

• When it comes to recruiting engineers, a non-technical hiring manager has a limited toolset to assess the candidate’s hard skills. They are more likely to make a decision based on soft skills (AKA chemistry). This can lead to discrimination (e.g. against neurodivergent people) and homogeneity (e.g. hiring a team fit instead of team add).

Does this type of EM do anything useful? Of course! But in my experience the admin work and people management activities don’t qualify as a full-time jobs for the teams I have been part of.

I confidently say that my best managers were all first engineers, then leaders. Not the other way around.

Our follow-up

My interview peer was a seasoned Engineering Manager. He asked if we could talk about this interview after we’ve submitted our candidate assessments.

When we met, he mentioned that he had met similar candidates in the past and whether there is a way to prevent wasting the applicants’ time and giving them false hope by getting them in the interview process?

Our action point was to:

• Add technical literacy and technical leadership as requirements in the EM JD (job ad)

• Add a few examples of the things we’ll be asking about in the technical leadership interview (e.g. tech strategy, hands-on experience, etc.)

• Continue being explicit on the technical profile of the EM positions

• Continue being transparent about having the tech leadership interview in the process

In this particular case, I checked with TA (talent acquisition), and it turns out the candidate’s university degree in computer science was assumed to qualify going through the technical leadership interview process.

Without sidetracking this post to be about degrees, I just want to highlight that due to fast change in technology, a degree (or certificate) quickly gets out of date. It requires deliberated effort and discipline to stay relevant and recruitable.

How technical should the EM be?

It depends. I’d turn that question to be about what the team needs. Here are some quick tests:

• Do you understand what the engineers are doing on a daily basis? It’s not about micro-managing, but rather having a solid understanding that is not too high level and abstract.

• If the engineers get stuck in technical tasks, how likely is it for them to come to you for ideation? Do they respect your experience as an engineer or are you considered out of date?

• When there’s a technical disagreement, how early do you learn about it? Do you primarily rely on one or more favorite team members to chew technical matters and feed you or can you digest this firsthand from browsing the repo?

• And when conflict comes your way, do you quickly understand the trade-offs to be made?

• Where do you spend most of your time? Admin work or engineering? If it’s the former, you need an administrative assistant to help. Don’t use Staff Engineer to act as your engineering leadership assistant. That’s the core of your job.

• Do you fill most of your hours with admin work (planning team activities, hiring, time report, firefighting, etc.) and spend the rest in pointless meetings that have nothing to do with empowering your team? Or do you spend some time coding or at least browsing the code? Can you draw the architecture diagram for everything your team owns? Can you explain how the testing and CI/CD are set up for your team?

• Do you sponsor your team members, or coach? Mentors are generally detached from the team the engineer is working in. Between coaching and sponsorship, the latter has a higher value because sponsors open doors and are more invested in growing the engineer.

• When meeting with higher level leaders, how likely is it for you to fight for your team against unrealistic technical expectations? A small technical “feature” can seriously complicate the code and add invisible costs. Can you assess those situations without having to drag the team into every meeting?

• If the EM position vanishes from the planet tomorrow, would you be able to work as a developer? Or would you be jobless?

As a manager, you have more power over engineers: vacations, parental leave, performance evaluation, promotion, salary raise, etc. You may notice that your jokes are funnier, your errors are more forgivable, your asks are more respected…

That’s due to the power asymmetry that comes with your position. With great power comes great responsibility. Engineers are less likely to call out your lack of technical competence, but your power will not be used efficiently without it.

When is Staff an Anti-Pattern?

A while back an engineering manager reached out for advice on hiring Staff Engineers. His org didn’t have any Staff Engineer and as someone who carries the Senior version of the title and actively writes about it, he thought it’d be a good idea to ping me on the matter.

After hearing about the problem, he was trying to solve and assessing his level of technical literacy, I advised against it. ❌

It was yet another case of an incompetent person being promoted to a position out of their league and trying to hire someone to cover their lack of technical competence.

It is an EM anti-pattern.

It’s the sign of a broken ownership where knowledge, mandate and responsibility are in different parties.

With all due respect for the team management skills and admin work, technical competence is part of the requirement for engineering management.

Without having a good grasp on the technical aspect, the engineering manager is limited by the technical perception of the Staff Engineers.

There’s nothing wrong with that —Staff Engineers are supposed to be more technical— but Staffs are primarily concerned with technology and don’t have the formal responsibility over people or product.

Engineering managers often find themselves making a trade-off decision considering not only technology but also product, personnel, budget, timelines, etc.

You may think that a healthy separation of responsible and accountable is at play. But this is exactly one of those cases where separation is counterproductive.

Even worse is when you have only one Staff Engineer. If you are going to hire Staff Engineers for this purpose, you need at least a couple of them to balance each other’s views. That’s an expensive solution, but far better than having one Staff Engineer gaining too much power.

Staff Engineers need to be capable of communicating complex technical concepts to non-technical audiences. But that shouldn’t be necessary when communicating with people who lead engineers. One can only dumb down a complex technical topic so much before it loses context, relevance, and nuances that are needed to make informed decisions.

Relying heavily on Staff Engineers to call all the shots on technical decisions gives them too much power while simultaneously making their cutting edge blunt! The time the Staff Engineer has to attend engineering manager meetings or read through decision docs is the time not invested in sharpening their technical skills.

How do I know this? Because I had seen it firsthand before!

A very competent senior manager got replaced with an inexperienced one with a completely different background. After a few months of struggle, the new manager hire-promoted a Staff Engineer to act as their missing technical arm.

Poor IC had to go to all meetings as the engineering manager he was supposed to support. But before you feel sorry for the Staff Engineer, I should mention that he took full advantage of his surrogate powers.

That Staff Engineer basically acted as a techno-king demanding other engineers to do stuff that sounded great on paper from the technical perspective but didn’t make too much sense when the non-technical aspects (e.g. budget, personnel, product, timelines) were added to the picture. The end result was that even a technically competent person got in the way of the engineers doing their job!

A word with engineering leads who have Staff Engineer

Staff Engineer is a leadership position, but the emphasis is on the “engineer” part. They are primarily the leaders of the tech but:

• Unlike engineering leaders, Staffs don’t have mandate over people

• Unlike product leaders, Staffs don’t have mandate over the product

Yet their entire impact depends on both people and product. Tech doesn’t talk or listen! Staff Engineers ought to work closely with engineering and product leadership to deliver an impact larger than 1 person.

In a previous article I pointed where the word “staff” comes from:

The term “staff” is used in the military to promote great soldiers as role models without giving them extra command and control responsibilities.

Staff officers assist the commanders in planning, analysis, and information gathering. The commanders have authority and accountability over their subordinates. Staff officers don’t.

The proper way to use Staff Engineers is to improve your:

• Execution bandwidth though delegation of technical initiatives

• Cognitive bandwidth through discussing technical nuances, because Staff Engineers are supposed to be on top of the latest technical development both internally and externally

They do have other duties as well. For example, bridging the gap between organizational silos by operating horizontally across the org and breaking free from the org chart.

Whatever you do, never ever replace yourself with a Staff Engineer. Staff Engineers, no matter how competent, are primarily concerned with technology. Putting them in a position where they have an oversized say in people/product matters is risky.

Let me give an example. A few years ago, we hired a Staff Engineer from a reputable company. He was sharp, smart, and a well-rounded technical leader. Unfortunately, he was put in a people management position: in charge of tens of teams. He didn’t have any formal experience as engineering manager, even at the team level.

He had a handful of Staff Engineers reporting to him. The Staff Engineers were frustrated by the poor state of DX (developer experience). So, this cluster lead (formerly Staff Engineer) decided to do something about it. He managed to convince the leadership to create a dedicated DX department which he would lead. That means the original cluster was left headless.

He spent the next few months talking to people and developing a tech strategy only to throw the towel and leave the company. Before leaving the door, he left a long wish list that he gathered from across the org. Reading through that list, they were all great ideas that would improve our lives as engineers. But it was evident that it required many hands to execute and that was a challenge with someone who has recently been promoted to such a high level of engineering leadership —despite being a super technical person.

Another example is from my own career. Many years ago, I got a job as a TPM (technical product manager). I had read multiple books about product management prior to that job and even launched my mini-products. Unfortunately, the job proved harder than I was prepared for. More specifically, I’m a techie and loved to dabble in the engineering aspects way more than the business aspects. Ironically, that product background gives me extra lens operating as Staff Engineer, but at the time, it hurt my ability to lead the product team to a better place. My takeaway from that exercise is to have immense respect for PMs —it’s a hard job. But I also learned something about myself: for now, I want to continue operating as IC (individual contributor) and be responsible for tech.

Both stories are meant to clarify one point: Staff Engineers are primarily engineers, so don’t expect too much when it comes to product or people management despite the fact that their impact depends on both.

Does this mean that Staff’s mandate over tech relieves the EM from that responsibility? Absolutely not. The main value of Staff comes from operating across teams:

As an engineering leader you should not need a translator to understand engineers. Don’t [ab]use Staff Engineers for that purpose. Going back to the military origin of the word “staff”, the accountability of decision is with the commander not the Staff.

The Staff Engineer represents the engineers. They are not substitutes for engineering leaders.

One could argue that technical literacy is only necessary for the lowest levels of management (e.g. the EM of a team). The argument is that the higher-level leaders (e.g. Senior EM, Director, Vice President, etc.) have too much on their plate and it’s ok to rely on Staff Engineers to be their eyes and translators.

Absolutely the opposite: The higher-level leaders are responsible for even more important decisions where the stakes are even higher. It is true that they are a few layers further away from the code, but that's exactly what makes those positions hard to get right. That’s the reason many of these higher-level leaders are incompetent, having no clue about what they're talking about. They are confidently wrong. That’s why you get cartoons like this circulating around:

Unfortunately, human beings are hard-wired to confuse confidence with competence! It takes deliberate effort and critical thinking to see the bullsh*t.

If people are promoted through a meritocratic process, the higher-level leaders are equally technical (if not more). It is a challenge of course to be both very technical and at the same time go to all those meetings about strategy, reorg, product planning, etc.

But that's exactly why it is so hard to hire competent engineering leaders and it takes a lot of time and energy to grow into those roles.

Just because someone was an EM for a few years doesn't mean that they should become a Senior EM and so on and so forth go up all the way to the CTO level just because they have the years.

A word with Staff Engineers

I once heard a Staff Engineer proudly saying:

As a Staff Engineer, part of my job is to help engineering leaders with technical questions they are too afraid to ask others 😎

If that’s how you see it, you have completely misunderstood the dynamics of that relationship!

Great engineering leaders often gather a wide range of insights from across the org to make informed decisions.

You should feel honored that they're consulting you. But make no mistake: your mastery in one area doesn't mean you are overall superior to the leader.

Besides, assuming that asking question is a sign of ignorance or weakness is such a rookie mistake. That engineering lead is probably juggling tons of responsibilities (some of which you know nothing about).

The moment you think you're better than someone else, a door will shut. You stop learning from that person. Everyone has something to share because we all have different life paths.

So instead of arrogance, just focus on giving them the colors to paint a picture —their picture. If you play your cards right, maybe you can learn a thing or two in return.

Conclusion

What is your experience of non-technical engineering leaders? What do you think about the combo setup where the Staff Engineer is covering the lack of engineering skills for an engineering leader?

My monetization strategy is to give away most of content for free. However, these posts take anywhere from a few hours to days to draft, edit, illustrate, and publish. I pull these hours from my private time and weekends. You can support me by sparing a few bucks for a paid subscription. Right now, you can get 20% off via this link. If you don’t want to spend money, sharing it with a wider audience also helps. Thanks in advance.