Ivory Tower Architect
The insult to any self-respecting software engineer
Note: I’m fully aware that this essay may rub some people the wrong way especially those who make a living by taking advantage of the fact that in many organizations, the ability to pretend to work is as payable as doing the actual work. Nevertheless, if it wasn't important, I wouldn't write it.
🤬 Call me names! Any curse would work. But not architect! 😱
That one hurts!
After 20+ years of software development, I have come to grow a specific despise for the title architect. Becoming an architect is one of my biggest professional nightmares.
That’s because people carrying this title typically:
Claim expertise over what should be in every engineer’s skillset.
Abuse power and cause friction.
Are avoided by skilled engineers.
In short: they are irrelevant!
The best phrase to describe them is ivory tower architects or ITA for short.
Ivory Tower (noun): a state of privileged seclusion or separation from the facts and practicalities of the real world.
This is a classic “monkey with a gun” scenario: mandate without knowledge or responsibility:
Architect is a misnomer in the software world. It's a common fallacy to compare it to a building architect. The closest role to the building architect is UX:
Do all architects eventually devolve to become ivory tower? Why having a dedicated architect is bad for the software organizations? What can you do to not become an ivory tower architect?
I’ll answer those questions but be warned that it’s not easy and involves actual work. Done right, your skillset will be so diversified that you may find it insulting to be reduced to an architect.
Another article elaborates why breaking one role into multiple titles is bad for quality, speed, and cost efficiency.
TLDR; architecture is one concern —one of many— for software engineers.
Symptoms
Ivory tower architects typically:
❌ Have beautiful charts and diagrams. Those boxes and arrows don’t map to the reality of how the system actually works. Some argue that this level of abstraction is what distinguishes a map from territory. But a map that is outdated and wrong, serves only one purpose: to confuse and eventually lead wrong decisions.
❌ Claim to represent the average engineer on the leadership table where in reality a tiny fraction of their time (if at all) is spent living the life of the average developer. Many developers rarely encounter them and when they do, they try to work around them.
❌ When you ask them a simple question, you get a complex answer back which doesn’t really solve the problem at hand. You ask them 2 + 2, they give you a lecture in algebra. They take the problem to their comfort zone instead of solving it. Sounds smart but not really the answer.
❌ They resort to rituals like tech committees which are objectively too expensive for what they deliver.
❌ They act like retired developers: heavy on opinions and “best practices” (which is someone’s interpretation of how things worked at a certain point in time, often glorified to make a point) while shying away from doing the actual work.
❌ When there’s success, they claim the credit but when there’s failure, they put the blame on engineers who executed their ideas.
❌ They’re not on-call for the systems they claim to own (this is an archetype of broken ownership).
❌ Instead of tackling the problem at hand, resort to name-dropping: “This is how we did it at [big tech]” (ex-Google? get over it). They have a tendency to cargo cult the rituals from a previous employer to the new ones.
❌ They demand that all decisions go through them and then delay the decision and objectively take wrong decisions due to lack of context and ignoring feedback loops.
❌ They engage in high visibility low-impact work like tech radars, engineering handbooks, and [premature] standardization without lending a hand to do the actual migration, operation, or refactoring.
❌ They have too much focus on best practices without context and a thorough understanding of what the tech they claim to own.
❌ Calling their engineer colleagues immature while conveniently ignoring their role to motivate the engineering leadership to invest in upskilling and optimizing organizational architecture.
❌ Design systems that look great on paper but are too flaky when implemented. That’s because they miss hands-on experience with the very artifact they’re designing for.
❌ See their engineering peers who are below them in the career ladder as disposable rows of an spreadsheet because “architect is only concerned with tech not the flesh”.
The benign type
While most architects do more harm by existing, there is a benign type that I just want to single out: the “wise” one who asks smart questions, answers simple questions, points people to search results, talks about the history, writes documentations, copy/pastes LLM results, holds presentations, organizes meetups, and generally tries to not get in the way.
I respect that.
It is still worth to evaluate their role in relation to their salary level but at least they don’t generate negative value by getting in the way of other engineers.
Staff, Principal, Distinguished
In recent years, Staff, Principal, and Distinguished Engineer gained popularity. Aware of the negative connotations of their role many architects disguise as Staff+. In fact, Will Larson counts architect as an archetype for Staff+ roles in his book. I believe it’s an anti-pattern for the reasons we get to.
This is not to say that all Staff+ engineers are ivory towers. In fact one of the beauties of not having the world “architect” in the title unboxes the Staff/Principal/Distinguished engineers to diversify their contribution. Architecture may be one thing they do, but they’re expected to do much more as we’ll discuss.
Becoming ivory tower is one of the most common pitfalls ahead of Staff+ roles. I’ve seen several senior engineers being hired/promoted prematurely to the Staff/Principal level where they don’t know how to handle the increased autonomy and mandate. As a result, with the best of intentions, they engage in practices that cause harm.
Unfortunately, they’re less likely to get feedback (or even care about it) due to the power imbalance that’s embedded in the career ladders.
Should engineers code?
Coding is the best way to earn street credit with other engineers.
Again, the higher autonomy of these upper IC roles is a slippery slope. It's easy to go to executive room and ask smart questions or even untie a technical problem and feel useful but I'm firm in my stance: many of these "tasks" are engineering leaders' responsibility.
IC's should be deeply technical and up to date.
Pairing a technical Staff with a non technical engineering manager is an anti-pattern. Staff+ should increase the execution bandwidth of technical engineering leaders, not to replace the need to have engineering knowledge.
3L
I got a good piece of advice a few years ago that was summed up as 3L: look, listen, and learn.
One can learn a lot by observing architects:
Look: Ivory tower architects put up a show and engage in high visibility low value work (also known as preening). They're often insecure in their technical abilities and towards the end of their tech skill shelf life so they resort to politics to stay in power, instead of upskilling to stay relevant
Listen: Ivory tower architects tend to use "engineers" to describe a group that does not involves them (e.g., "engineers implement my designs"). They're typically more fluent when talking about tech and that skill alone prolongs their career. That works because incompetent leaders easily confuse confidence with competence and eloquence with substance. ITA's employment opportunities however is typically limited to those kinds of environments (e.g. old school enterprise going through "digital transformation" hiring from big tech to save the day). If you listen to them, they're delivering value. If you listen to the "engineers" they haven’t heard of them or actively work around them.
Learn: not much here. If you drag them to actual technical conversations you see their lack of competence in full glory. Skilled engineers avoid them. Usually ITAs have good social skills, so their presence acts as a classifier: they attract the junior type who likes to outsource thinking while simultaneously repelling the experienced ones who assess competence by outcome.
FAQ
Is architecture a full-time job?
In every organization, there are SMEs (Subject Matter Experts) who can be pulled to architectural decisions. Are there enough work to fill a full-time bandwidth? Even if there is, shouldn’t this work be distributed to a large group of people?
Does having a dedicated architect hinder the growth of the other engineers into this important matter?
When the architects turn what should be one-off consultations into a monopoly and demand that all decisions run through them, they create a bottleneck. Essentially, they cannot know the context and technical nuances of all problems to take objectively good and timely decisions. Centralizing the architecture concern takes away the growth opportunity from the rest of the organization to own their architecture.
Why do skilled engineers avoid Ivory Tower Architects (ITA)?
There are multiple reasons, which can collectively be summed up to: counter-productivity.
ITAs turn themselves into gate keepers for decisions, developers have to go through them. But the developers soon learn that they needed to educate the ITA due to lack of context, outdated technical skills, and world views that requires some unlearning.
This adds to the cost of the decisions. The pragmatic engineers learn to work around ITAs:
Downplay the importance of a decision so it doesn’t end up on ITA’s table
Implement the idea before it gets the chance to be decided
Skipping the improvement altogether
…
Do all architects converge to become ivory tower?
Doing architecture full-time starves the architect’s ability to update their technical skills.
It is hard to learn a skill without any requirement to use it hands-on and implement a solution with it.
One who voluntarily chooses to spend their time only on architecture gradually becomes blunt in implementation skills.
Excuses like “coding isn’t the best use of my time”, just communicate elitism and poor understanding of technical evolution.
…with all apologies to my many wonderful, highly skilled "Architect" friends, I tend to think it's a bullshit role. 🙃
I believe that only the people building software systems get to have opinions on how those systems get built. —Charity Majors
Read more:
Charity Majors on Architects, Anti-Patterns, and Organizational Fuckery (blog post)
Michael Nygard on architecture without an end state (podcast)
Personal Story
This essay may look like a personal rant about a specific person. It kinda is: I listed 12 actual examples of ivory tower architects that I've encountered in my own career over the years.
Coincidentally many of them are my own negative role models (the opposite of a role model, people you aspire NOT to become).
On a given day I may design systems, code, review PRs, measure system performance, use data to improve security, reliability, and scalability. I may spend time reading, doing POC also write or contribute to ADRs (architecture design records), tech strategy, hold knowledge sharing workshops for my engineer peers, and presentations for leadership. Interviewing, and improving our recruitment pipeline is also something I contribute to. These days I act part-time as a technical product manager for an internal developer platform (IDP) and am on-call for the tech that I’m accountable for. Sometimes I venture to areas that are formally outside my tech domain like organizational architecture because it impacts the health of tech which is my responsibility.
But it wasn’t always like that.
I was a simple front-end developer working with jQuery. I couldn’t deploy code. That was too advanced for me. I relied on testers to find my bugs. I was happy, they were happy, we all shared responsibilities.
That all changed when back in 2018 I was working at a company which due to lack of money decided to get rid of many roles like QA and DevOps (full story here).
I was horrified. Over night I got admin access to our cloud infrastructure and was put on-call.
Little did I know this inconvenience is going to catapult my understanding of software systems, SRE, quality, security, and reliability.
Me and my team learned by making mistakes until the company got us multiple formal training.
Gradually we got hold of it. As it turns out, you don’t need all those roles to build software. All you need is generalists who don’t suffer from technophobia and are willing to learn. And the combination of having all those skills paid dividends: on a given day I would code, test, deploy, troubleshoot, and even manage product to some extent.
Once I learned how to learn instead of being scared, that trajectory continued to a Staff SRE role and later Senior Staff role (I wrote about my journey to Sr Staff here).
☝️Above, I wrote what I do these days. None of that spells architecture yet all of that impacts architecture.
The combination of a high level title, escalated autonomy, years of experience, and having the leadership’s ear is a huge abuse pitfall.
On multiple occasions I’ve had the chance to bloat my results and sell it as transformational initiatives, represent engineering voice to non-technical leaders, or shine a light at the end of a tunnel.
It is much easier to wing it up there, especially when the org is composed of tall silos with thick walls.
I resist the urge. Not because I’m some wise a** with a benevolent intention. But because the moment I take advantage of a broken system, I’ll be part of that system.
That’s why you see many architects stay at the company for too long! They don’t have anywhere better to be. Over the years, their technical knowledge level becomes too shallow that they’re practically unrecruitable to any company with a sane hiring process. Their career choice is limited to staying within one broken system or switching to another broken system.
I have a theory that everyone is in the best place they can possibly be. If you come across someone who complains and nags too much, it is often a symptom that they are trapped and got nowhere better to be.
Ivory tower may sound like an amazing place to be, but it’s a trap. Keep your axe sharp and build options. Options put YOU in power.
Pro tips
My monetization strategy is to give away most content for free. However, these posts take anywhere from a few hours to a few days to draft, edit, research, illustrate, and publish. I pull these hours from my private time, vacation days and weekends.
You can support me by sparing a few bucks for a paid subscription. As a token of appreciation, you get access to the Pro-Tips sections as well as my online book Reliability Engineering Mindset. Right now, you can get 20% off via this link. You can also invite your friends to gain free access.
Let’s dig a bit deeper into:
How to identify ivory tower architects and neutralize their negative impact?
What is that negative impact of ivory tower architects and how to reduce that?
How not to become an ivory tower architect?
How to hire/promote senior IC (individual contributor) roles like Staff, Principal, and Distinguished Engineers while minimizing the risk of creating an ivory tower architect?
How to identify ivory tower architects?
We touched on some symptoms above, here are more with reasoning and examples:
Spending more time in meetings than coding. Senior IC roles require up to date, hands-on experience in order to effectively deliver impact. For example, when LLMs were new, I invested from my own pocket to try Github code pilot to use it on my open source projects to learn about its limitations and strengths (before it was even allowed at my current company).
Interacting more with leadership than the engineers. When interacting with engineers, mostly doing so with a handful of their favorites (e.g. technical committee) than all the engineers in their org. For example, when rolling out SLOs across my org (100+ teams), I could ask leadership to demand SLOs from all teams. Instead, I systematically took the time to meet every team at their own pace, learned about their system architecture, and together with their consumers set their service levels. I learned a ton along the way which I write about here.
Relying on mandate and policing than empowering others. Engineers are smart and respond to logic. Mandate is not a replacement for that. A good example happened at one job where there were multiple processes for incident management. This apparent inconsistency, hurt the ability to general central incident reports. An ivory tower architect borrowed the mandate of upper management to kill all those processes but one. There was resistance, politics, and essentially the engineers didn’t buy into that unification effort. I saw those extra incident processes as a symptom. The real root cause was that the official incident handling process was cumbersome and had poor tooling. By removing those pain points with the feedback from teams, I created a process and tooling that was more reasonable to its users: engineers.
Not being on-call either because the system is too complex or it’s beneath them. If EM is responsible for engineers and PM is responsible for product, Staff+ is best described as “Tech Manager”. The health of the tech is primarily the accountability of the Staff+ role (for organizations that have this role). Being on-call helps gain first hand experience with how things work and break. Plus, making system reliability a Staff+ problem motivates them to use their elevated mandate to actually do something about it.
Overall, turning ivory tower is the biggest risk ahead of senior IC roles and it takes deliberate effort to not let the higher position in the career ladder detach you from the other engineers.
What is that negative impact of ivory tower architects and how to reduce that?
By assuming mandate over the architectural decisions, they break ownership. Just by mere existing and assuming architectural responsibilities, you limit the ability of other engineers to grow their system design skills. A better alternative is to diversify your role by contributing to code (e.g. refactoring) while participating in architecture discussion on a case-by-case basis. Rest assured that the organization doesn’t burn into flames if you don’t babysit every single decision. Mistakes aren’t bad. That’s where most of the learning comes from. It is better to stay within a safe distance and offer a helping hand when needed instead of proactively storming into every room and demand that decisions go through you.
Having a high level abstract view of the systems and organization often leads to bad decisions that look nice in theory but create a mess when they go through. Modern software is complex. Human systems are even more complex. It takes time and effort to get a good understanding of both to an extent that allows informed decisions. A trick I use is to limit my vertical scope. For example as I said I am responsible for over 100 teams. There is no way in hell I can know all the systems made by such a large organization. Instead, I focus primarily on reliability engineering aspect and offload the other aspect to others. Even within reliability engineering context, I use SLIs and SLOs in a specific way to let the teams and their consumers decide what to measure and how good is good enough. Following my own definition of ownership, I don’t claim mandate where I don’t have responsibility.
Illusion of knowledge is worse than no knowledge. Architects often rely on out of date system design diagrams and technical know how. This outdated knowledge —illusion of knowledge— can lead to bad decisions that sometimes work against the very purpose of having an architect. The truth of the matter is that modern systems evolve rapidly and new technologies solve old problems. It is better to honestly say “I don’t know” and use that knowledge gap to learn, than just wing it with pretty diagrams and beautiful charts.
How not to become an ivory tower architect?
Be on-call: many job ads for Staff/Principal/Distinguished engineers require on-call. But many companies implement these roles without any on-call responsibilities. It is important to be in the loop when an incident happens even if you’re not the primary responder. It helps gaining experience about a system’s weak points and be more deliberate when it comes to improve reliability, security, and general health of the tech —which is your responsibility.
Be available: it feels good to go to a meeting room with high level leaders and talk about important stuff. But don’t mistake that for impact. Most probably customers don’t pay the company for those meetings but the product. Stay close to the product and product developers. Once you try that, you may realize that your title and reporting line may get in the way of feedback loop. Do everything necessary for engineers to know that you’re one of them. Any minute spent working with other software engineers is a minute that makes you more efficient when working with leadership to drive high impact decisions.
Stay up to date: I can’t stress this enough. The primary value of IC roles is their technical depth. You need to actively spend time learning about new technologies to be able to assess which ones can solve the old problems more effectively. But I hear you, there’s not enough time to learn everything. ThoughtWorks tech radar is a good example of how to allocate your energy. No need to learn everything with equal depth. Only the parts that relate to your area is enough. One way to learn deeper is to do some hobby projects or proof of concepts, but be careful not to fall in love with your hobby project and push it to production by force. A tip is to try to teach what you learn. That way, not only you spread the knowledge and demonstrate good technical leadership but you get to learn from all the questions that the “students” may ask.
How to hire/promote senior IC (individual contributor) roles like Staff, Principal, and Distinguished Engineers while minimizing the risk of creating an ivory tower architect?
When hiring for or promoting people to senior IC roles like Staff/Principal/Distinguished engineers, look for these signals:
Only architectural work: Architecture is not a full time job. It’s part of the duty of software engineers. Give equal weight to actual contributions to the source code.
Try to understand their motivation: you can ask it straightforward, but it’s better to ask indirectly and read between the lines. If their motivation with senior IC role is to avoid work, it’s a red flag. If they are motivated by leading others and multiply their impact through follow-me leadership, coaching/mentorship, while working shoulder to shoulder with other engineers, you got the right person. A good signal is to ask about the build and deploy process for one of the products or details about one incident. This is where you want to ask a lot of follow up questions instead of jumping to different questions. The deeper you go, the better signals you get about one’s technical literacy, contribution, and motivations.
Sound opinions: there’s no perfect tech. All technical decisions are trade-offs. Try to understand how they made high level technical decisions? What feedback loops they used? Did they go big bang or gradually rolled out the new decision? How did they measure success and why?
Up to date knowledge: over the past years, system design interview became more established and preparation material became more available. As a hiring manager or member of the promotion committee, you cannot assess the qualities you don’t understand. So take some time to familiarize yourself with modern system design. You can subscribe to
or for a weekly dose of system design newsletter for example. This is very important because we as humans have a cognitive bias to confuse confidence with competence. Without having a good knowledge about architecture, you’re more likely to hire/promote incompetent but confident candidates to senior IC level with elevated mandate. They can cause serious harm before you know it. Do your homework first.Beware of the Staff anti-pattern where a skilled engineer assumes the knowledge while an incompetent leader has the mandate. This is the sign of a broken ownership. I have not seen a single such leader voluntarily do what’s best for the org and get the hell out. So I’ll just put it there to raise awareness. If you’re not technical enough to lead an engineering organization, actively coach your senior ICs to create the next generation of leaders. Or you can honestly ask their help, do the work, and gain the technical skills. It’s not rocket science. Many software engineers don’t even have an engineering degree. If you really want to learn, you can do it.
If you find this post insightful, please share it in your circles to inspire others.