Engineering grading systems
Why engineering maturity models are counter-productive, what are the real questions we should be asking, and how to lead the tech to a better place?
High MTTR (mean time to resolve incidents)
High CFR (change fail ratio)
High MADR (manual to automatic incident detection ratio)
Large number of high impact incidents
Low team velocity due to piled up tech debt (e.g. high Lead Time)
Breached SLOs and burned-out error budgets
…
When the tech health is poor, it is naïve to think that the engineers have no clue about what good looks like. It is tempting to define a grading system —a “maturity model” if you will— for the engineers to grow the f*** up and get their s*** together.
The idea is simple:
1️⃣ Define what good looks like
2️⃣ Ask the teams to self-assess based on the criteria
3️⃣ Sit back and relax while the teams find a way to pay the expense and lift their level
Unfortunately, it doesn’t work like that in reality. Poor tech does not mean poor engineering and bad engineers. It is just a symptom. The real root cause is somewhere else.
This article elaborates why engineering grading systems don’t work and what to do instead.
🤖🚫 Note: No AI was used to generate this content (except one image which is clearly marked). This essay is only intended for human consumption and is NOT allowed to be used for machine training including but not limited to LLMs.
Why and what?
Let’s get to ❌ WHY engineering grading systems don’t work?
and ✅ WHAT to do instead?
❌ One size fit all: it is very hard to define GOOD in a way that fits a wide range of technical artifacts but is still tangible enough to drive meaningful action. There's often some cargo culting at play where best practices from one company are shoehorned to another.
✅ Instead of emitting manifestos, spend time with each team to understand the nature of their services and challenges. Then define measurable expectations for the service (not the people) and give the people full ownership to improve the service.
SLI and SLO come in handy as a tool to measure and set those expectations while acknowledging team autonomy.
❌ Hands off: these kinds of initiatives often come from hands-off technical leadership (Ivory Tower Architect) and engineering leaders who rely on them to act as a technical translator.
✅ Large systems are complex and change dynamically. Seek to understand before trying to optimize. Otherwise, your efforts will be limited to the symptoms and surface while the root causes fight against any optimization effort.
If you want to change an organization, start small, work closely with a few teams, and take iterative action while monitoring the impact. Approach it as an experiment with a clear hypothesis and validate the solution before rolling it out across a large organization.
❌ Presumptuous: the grading systems often assume that engineers don't know any better because if they knew, their outcome would show.
✅ The majority of engineers are smart and know what good looks like. If the results don't match the expectations, you need to dig deeper to understand all the forces at play. Maybe their good is good enough given all the circumstances.
I’ve been in the industry since 1999 across more than a dozen companies and worked closely with hundreds of people. I haven’t met a single engineer that I can call “immature”. That’s not the right word to use for human beings because we evolve. Tech can be immature, but it’s a symptom. The root causes can be:
Lack of budget (time, money, and resources)
Bombarding teams with feature works and turning them to feature factories
Treating developers as code monkeys detached from the business problems
Moody leadership & constant reorg
Starving technical competence (e.g. due to low pay and recruitment bar)
Non-existent or broken career ladder which fails to map personal growth with organizational growth
Too much tech debt and no payment plan
Poor organizational architecture that doesn’t support reliable and seamless customer journeys
Any of the archetypes of broken ownership
…
Don’t you think if the solution was as easy as drawing a north-star, somebody would have done it before? These days even LLMs can do a pretty decent job at that. Where AI fails is to understand the human experience of building, owning, and being in charge of technical solutions to product problems that solve human problems.
If you must create a north star, make damn sure to frame it around the tech, not people. Don’t make it personal. People lift the tech only when technical maturity is detached from their own identity. They are more likely to improve the status quo if they don’t feel called out.
❌ Distract accountability: typically, the engineering grading systems are formulated in a way that frame the accountability on the engineering teams. They conveniently ignore the role of leadership to hire, mentor, utilize, and retain talent.
They’re equally mute about the responsibility of the organization to invest in software. If the tech is unhealthy, it's engineers' fault.
✅ This is one of the cases where the distinction between accountability and responsibility creates clarity.
You get what you pay for. Tech health has a cost. Software rots over time, top talent may leave or get blocked. Improving the quality of tech requires investment.
The relationship between software and profit (`revenue - expenses) has a significant impact on how investments are motivated and what's the long-term technical vision:
You don’t need Google-level tech when your startup hasn’t proven a product-market fit
Your local retail site cannot justify the investment to beat Amazon in every metric
You don’t need to pay the upfront cost of breaking your monolith to microservices just to make it fit your org structure
You probably don’t need to refactor your tech stack to use GraphQL
The software timeline and budget drastically differ from business to business. Understand the limitations before proposing a solution. Leave the ivory tower and work shoulder to shoulder with engineers if you want fresh first-hand insight for meaningful initiatives.
Recap
Instead of cargo culting and blindly applying best practices, try to understand the problems and apply solutions that fit
If you must create a maturity model, make sure to frame it as a technical maturity model, not team or people maturity model.
Put the majority of the responsibility on upper-level leadership, not the teams. Associating the tech quality with engineering skill is naïve. There are often more factors at play and the company/leadership needs to invest in tech instead of being mandate heavy.
A maturity model is useless without proper support from leadership and the company putting its money where its mouth is and investing in lifting the state of tech. And no, threatening isn't support. Invest in education, tools & automation, documentation, and roll up your sleeves to work shoulder to shoulder with other engineers.
Beware that there’s such thing as good enough! Not everyone needs Google-level tech because quite frankly not everyone has their recruitment bar, compensation, and revenue model.
Always experiment at a small scale before rolling it out across a large organization. Measure the right metrics before and after each iteration.
My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.
You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.
Love this article.
But this part i dont get it
“People lift the tech only when technical maturity is detached from their own identity.”
Great article.
Your definition of good reminds me of the definition of ready and definition of done we use in software QA: a model everyone can agree on whatever their seniority.