A few years ago I came across this old video from Russ Ackhoff talking about the importance fitting parts to the overall system performance:
If we have a system improvement [problem] that’s directed at improving the parts taken separately, you can be absolutely sure that the performance of the whole will not be improved.
But most applications of improvement programs are directed at improving the parts taken separately not the whole.
I read in the New York Times recently that 457 different automobiles are available in the united states.
Let’s buy one of each and bring them into a large garage.
Let’s then hire 200 of the best automotive engineers in the world and ask them to determine which car has the best engine.
Suppose they come back and say the Rolls-Royce has the best engine. Make a note of it.
Which one has the best transmission we? They go run tests and come back and say the Mercedes does.
Which one has the best battery? They come back and say the Buick does.
And one by one for every part required for an automobile they tell us which is the best one available.
Now we take that list and give it back to them and say remove those parts from those cars, put them together into the best possible automobile because now we’ll have an automobile consisting of all the best parts.
What do we get?
You don’t even get an automobile for the obvious reason the parts don’t fit. The performance of a system depends on how the parts fit not how they act taken separately.
The system is not the sum of the behaviors of it’s parts, it is a product of their interactions.
However, the word “fit” is sometimes misunderstood:
When hiring, many companies look for a “culture fit” but in reality, they’re just filtering for more “people like us”.
Instead, we should be hiring for “culture add”: individuals that enrich the culture by challenging it and normalizing new behavior.
When designing complex distributed systems, too much time is spent on unifying the tech stack. The idea sounds reasonable: one tech stack is easier to monitor, update, and hire for. There are 2 cases:
The single stack was enforced from the start: it can make hiring expensive because professionals can be quite opinionated about their weapon of choice. Different technologies thrive at solving different problems. Praising a golden hammer (tech bet) also requires shoehorning all problems to a single solution. If the unified stack is platformized, we have a single point of failure (imagine your platform was 100% Java when Log4Shell came out).
The single stack is enforced after the fact: this requires many migrations and refactoring. They in turn introduce cost and risk because there’s no guarantee that the system behaves the same once the dust is settled.
If the system is in bad shape, it is very tempting to go to a refactoring and migration spiral instead of gradually improving it. And you might be right, a big bang change might be what the system needs, but the competent leaders should ask the right questions to understand the ROI:
What’s the cost of refactoring? The answer is almost always a range. Don’t fall for the optimistic minimum because it’s impossible to estimate such efforts. There are always complications that shows up after the work has begun.
What’s the motivation? Clever engineers may connect your frustration (e.g. the system is too slow) to their CV growth (e.g. let’s rewrite everything in React). If the team suggesting the refactoring can’t stand logical scrutiny, dodge the bullet and ask for gradual improvement instead.
What’s the benefit? Sure, refactoring and migration will improve some metric, but are they the right metrics and is the diff worth the cost? If there’s no metric or no data, it’s very hard to judge the necessity of a big change. Instead, ask the team to do a timeboxed POC, prove technical feasibility, and compare.
Regardless, fitting parts doesn’t mean similarity or homogenuity. A different way to think about it, is to ask:
Do the parts add to each other and complete each other
or are they more of the same?
One is about exponential yield while the other is just linear.
My monetization strategy is to give away most content for free but hese posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends. The simplest way to support this work is to like, subscribe and share it. If you really want to support me lifting our community, you can consider a paid subscription. If you want to save, you can get 20% off via this link. As a token of appreciation, subscribers get full access to the Pro-Tips sections and my online book Reliability Engineering Mindset. Your contribution also funds my open-source products like Service Level Calculator. You can also invite your friends to gain free access.
And to those of you who support me already, thank you for sponsoring this content for the others. 🙌 If you have questions or feedback, or you want me to dig deeper into something, please let me know in the comments.