An observation I’ve made while working with various teams is that there are a relatively small number of things that separate the men from the boys (so to speak). My experience tells me that knowledge of technologies, frameworks, languages and so on is not a big discriminator between the best software engineers and the rest; these things can be taught and learned pretty easily. Rather the key discriminator is how people think about the world around them, and a few areas in particular make all the difference. Perhaps the main one is the “bigger is better” trap.
This one is particularly prominent among newly-promoted engineers; perhaps they’ve just been given a “senior” or “lead” badge for the first time. People in this position often feel a need to prove themselves, and one way engineers at this phase in their personal development often measure themselves is by their ability to handle complex problems. The logic goes something along the lines of “if I work with more complex things, it proves how smart I am, therefore I’m higher in the ranking”. People in this trap, even if they are aware of the absolute necessity to keep things as simple as possible, are blinded by these instincts. They will seek out the most complex things they can, so they can prove that they can do them. If there’s nothing complex close to hand, they’ll create something!
Another form of this is to create a “territory”, that is big and complex enough to hide in. This becomes “their area”, where all others must seek guidance and/or permission to work on it. Again, the main reasons for doing this are prestige or defensiveness – there’s sometimes a belief that creating complexity can be a good form of job security! Needless to say this kind of behavior is severely limiting for a team’s ability to be effective over any kind of timeframe.
A third way that “bigger is better” can be seen is again in a desire to gain prestige, from over-estimating the size of the problem ahead, and over-engineering a solution to this imagined need. People try to gain prestige from believing they are in the realms of things like “big data”, “web-scale” or “enterprise”. The reality is often far simpler and less glamorous. The problem with this is that if a team convince themselves they have a big problem, they’ll simply “need” to reach for the big solutions. The team stop looking for simple and elegant solutions, because they start from a belief that they need far more than they do. Teams at Google – who legitimately are dealing with big scale – follow a simple guideline: they design and build for 10x the current need. To put that in context, rather than being an over-estimate, this is extremely pessimistic. Instead of designing for the load a full scale product will have, they design for 10x what a prototype would need. Not 100x, 1000x, etc. When the product starts to take off, they redesign for the new level; they explicitly design low-cost disposable solutions for now and replace them only when there is a proven need to do so.
In my experience, when a team has over-engineered something and are struggling to proceed because of the complexity they now find themselves maintaining, the problem is almost never from a technical origin. It comes from the (lack of) maturity of the decision making, and the desire to build a grandiose monument to the engineer’s prowess trumping the business need. The root problems are often cultural – if we emphasize a “ranking” through job titles, through “promotions”, individual goals and objectives, and so on, we are encouraging individuals to put themselves first over the team. The individuals are acting in their own interest, because that is what is rewarded and encouraged. Worst case, they can boast about the complex beasts they conquered on their CV, and move on to another team with these flawed values. We need to instead emphasize the merits of healthy behaviors: value simplicity and maintainability, look for elegant and clean solutions that will not hinder the team, look for ways to build the team and grow together rather than climbing over each other in some dysfunctional pecking order, and have mature and balanced assessments of what is needed, with the discipline to reign in the urge to “play with shiny toys” and serve the business need first and foremost.
Remember: “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius – and a lot of courage – to move in the opposite direction.” (Ernst Schumacher)