In software development, it's extremely common for us to choose solutions that aren't optimal - meaning that they weren't the most effective, safe, or efficient solutions available to solve the problem at hand. Many would attribute these decisions to a simple lack of knowledge, awareness, or competence. However the reasons behind these choices are often more complex and nuanced than that, and stem from a variety of deeply human factors which are worth exploring - I'll describe a few.

Anxiety-Based Decisions

• Anticipating What Doesn't Exist: Often, we imagine problems that never come to pass. This leads to solutions that are too complex. An example is designing for scalability that we don't need, or setting up rigorous processes to mitigate problems that don't happen.

Bob, a seasoned developer, once spent weeks building a caching mechanism for a small application to handle an expected traffic surge that never materialized. The time could have been better spent on other high-priority features.

• Choosing Safety Over Efficiency: Choosing the safe route is often a fallback, even if it's not the most efficient path. It's rooted in a deeper concern - the fear of blame if things go wrong. Standing against safety can seem reckless, even if the risks are extremely unlikely. As a result, advocating for efficiency over safety is rare, despite the potential benefits in performance and productivity.

Emily, a product manager, always insisted on an exhaustive risk assessment for every software update, even those with minor and clearly understood changes. While this process was unnecessary and considerably slowed down the deployment cycle, her team never disputed it as it was presented under the guise of ensuring safety and stability. However, in reality, Emily's overly cautious approach was driven by her fear of being blamed for any potential issues that might occur post-deployment.

• Job Security Over Efficiency: Sometimes, we sidestep the best solution because it could threaten our job. This can mean choosing a solution that needs our ongoing attention. It can also be setting up a process that can't run without us.

Jim, a project manager, had a daily task of manually collating and emailing status reports from various team members to all stakeholders. Despite the availability of project management tools that could automate this task, Jim held back from using them, fearing that it could make his role seem less essential.

Growth and Professional Development

• Learning Through Complexity: At times, we opt for a suboptimal solution because we want to learn. This can be choosing a new tool or language over an old but efficient one. It can also mean taking on a new process simply to expand our experience.

Sarah decided to build a project in Rust, a language she was just learning, despite being proficient in Python. She knew it would slow down her progress but saw it as a worthwhile trade-off for learning a new language.

• Building Our Portfolio: Similar to the last item, we may go for a non-optimal solution to enhance our portfolio. It's common to choose a new tool or language to add to our resume. We may also choose a more complex solution to show off our skills.

James decided to set up Kubernetes for hosting a small informational website. Despite the overhead and complexity associated with Kubernetes being unnecessary for a project of this scale, he thought it would make a valuable addition to his resume.

Desire for Novelty

• Fun Over Efficiency: We may overlook the best solutions for more exciting ones. This happens when we choose an interesting but less efficient approach. It also happens when we introduce a fun but time-consuming activity that cuts into productivity.

Lily decided to build an internal tool for her team using a game engine. It was not the most efficient way to build such a tool, but she found the process engaging and fun.

• Complexity for Impressions: The need to impress can add unnecessary complexity. We may use flashy but needless tools to show off our skills. We could also use complex metrics that don't add value but look impressive.

David trained and used a custom machine learning classification model for a task that could be solved with a few simple conditional statements. He wanted to impress his peers with his knowledge of machine learning.

Conclusion

It's important to note that these reasons are not mutually exclusive. A single decision can be driven by multiple factors. Regardless, it should hopefully provide a framework for understanding why certain decisions are made.

Are these really suboptimal solutions, then? If we consider the human factors behind these decisions, it's clear that they're not always suboptimal. In fact, with that framing, they are often the most rational choice for the individuals making these decisions, even if they're not the most efficient or effective solutions for the problem at hand.

Understanding and acknowledging the motivations behind why people choose certain solutions over others can provide valuable insights for improving team dynamics, productivity, and even personal growth. It can also help us make better decisions ourselves.