Developers Are Not Commodities

As I write these words, one of the greatest security breaches the world has ever seen has just occurred; Equifax has had personal data for 143 million customers exposed, including names and social security numbers. That's nearly half the population of the United States. How did it happen? According to the NY Times, a weak point in the company's website software was exploited. 

This kind of stuff is no longer unusual. Let's not forget about the time Yahoo was breached, and exposed 1 billion user accounts, or Heartland Payment Systems was breached and exposed 134 million credit card numbers, or similar breaches at Sony, JP Morgan Chase, and many others.

The term "data breach" might make you imagine an Ocean's Eleven type scenario, where a team of highly skilled hoodlums work an elaborate plan to get past security systems and into a highly guarded data center, stuff the data into sacks, and slip out undetected. Or perhaps the breach was the result of super-advanced cyber-thieves, using technology that was developed and funded by rogue governments?

Either of these scenarios is better than the truth, which is that a lone hacker probably just poked the company's websites and intranets for known security vulnerabilities until they found one, which they then exploited to extract the data they wanted. The fact is its easy to write software that has bugs or flaws that open up these kinds of vulnerabilities. It's easy to not rewrite that buggy code or apply that patch when you aren't being told to do so, and you don't feel responsible for it. It's cheap too. You don't need a lot of time or funding or particularly seasoned (i.e. expensive) developers. Which is exactly why its so omnipresent.

This is the dirty secret not just behind these major incidents, but also behind the mundane ones we encounter on a daily basis, that cause us to be frustrated by projects that run late, that are buggy, that don't come out the way they were supposed to, or that are ridiculously expensive to maintain. 

I'm not the only one who knows this. I am certain that if I went into the IT department of any of these companies before they suffered a breach, I would find at least one developer who felt that more needed to be done to beef up security, or that more qualified developers needed to be hired by the company, but who were ignored by management. 

Watch what happens on social media after a breach; hundreds of developers will be posting "I told you so" messages. Pay attention to the lawsuits that happen afterwards; cost-cutting in the software department is always implicated (due to corporate greed of course).

Now what do I mean by "cost-cutting?" I am not talking about cutting travel budgets or office supplies. I am talking about cutting quality, which is at the root of the issue. Now I am aware that nobody sets out to "cut quality". Rather, they make the mistake of thinking that they can get the same level of quality regardless of who they hire as developers, which is unbelievably incorrect.

Here is the crux of the matter; great quality can only come about by hiring great developers and building great teams. Period. Anything less will result in, well, less. This secret is understood by a few exceptional companies, such as Apple and Google, which invest heavily in people and teams, but is ignored by many others.

As such, despite the pain, things are getting worse and not better. The common issue is that management thinks "a developer is a developer", and that, in the interest of margin, the game is about finding the least expensive developer that meets the "minimum qualification criteria".

Unfortunately, the people making such decisions often don't know the first thing about programming and have never written a line of code in their lives. I have seen many HR managers describe the "minimum qualification criteria" for a position without having the foggiest idea of what any of it means. It becomes a laundry list of keywords: .Net, Javascript, SQL, Hadoop, SAP, Python, and so on, with an arbitrary number of "years of experience" required for each: 3, 5, or 10. 

The fact is HR really can't tell a good developer from a bad one. It typically takes another developer, a seasoned one, to do that. To "play it safe" HR adheres to their checklist, and assumes it will do the job (Where does the checklist come from? That's another interesting topic). On top of this, the company wants to keep costs down and keep the salary as low as possible. That means no one who exceeds the minimum requirements (i.e. is "over qualified") is likely to get hired. It also means that someone who might be great for the job, but does not meet something on the checklist exactly, will be passed over.

This is all a great system for facilitating mediocrity, and we have it in spades. So that managers are rewarded for "keeping costs down", IT departments continue to shift to:

• Hiring junior developers when possible to reduce salary costs
• Hiring offshore developers as frequently as possible to take advantage of lower pay rates
• Hiring remote contractors who do not require office space and whose pay can be tied to an aggressive timeline

If this seems like much ado about nothing, let's start with the following: Are programmers basically the same? The answer should be an obvious "no", but I know for many people it's not, so let's explore it a bit further. It is easy to understand if you throw away the erroneous conclusion that development is a type of simple labor. Far from it. For the moment, let's pretend that we're talking about some other type of job, and that we're talking about something that directly affects the success and bottom line of a company--in terms of money, market share, reputation, and everything we usually worry about. Probably we will look for many subjective qualities. such as:

• A positive attitude
• Someone who takes initiative
• Someone who is detail-oriented
• A good communicator
• A proactive learner
• Works well with others

...and so on. We would likely rate these qualities at least as important as "years in the business". The first point I want to make is that all of this applies to developers as well.

• A developer with a positive attitude is better equipped to deal with changing requirements, challenging deadlines, and finicky stakeholders;
• A developer has to use their own initiative to figure things out on their own, identify problems on their own, and escalate important issues on their own;
• A detail-oriented developer will produce higher quality work, that tends to work properly the first time, and has fewer bugs;
• A developer who communicates well will be better able to interact with stakeholders and translate their requirements into a technical solution;
• A developer who has a passion for learning will keep up with new developments, allowing them to leverage new and better techniques and technologies in their work;
• A developer with a cordial personality is better able to handle disagreements with other developers.

It is a law of statistics and economics that people with these qualities will always be scarcer and command a higher price than those without. And why not? They provide higher value. A "go-getter" will contribute more in terms of ideas, productivity, and results every time. Those with lesser qualities will require more hand-holding, make more mistakes, create more rework, spark more conflicts, and so on. Having spent over a decade managing all types of developers, I have certainly found this to be true.

But there's an even more important trait that is relevant to this conversation: a good developer will keep a company from incurring the high costs of bad software. This is a proactive trait; one that spots and points out potential problems before they occur; and one that insists on doing a quality job, and fights for the time and resources to do it. This is critically important, because what management tends to initially ask for is usually incompletely thought through, and has an overly aggressive timeline, which, if simply accepted, will result in a rush job with the kind of behind-the-scenes quality issues we are discussing. It's true that management may still not listen, but there is not even a possibility of having a proper discussion--or remediation--without developers at the right level to provide it.

What determines if a developer is at the "right level"? In addition to the traits I mentioned above, a key aspect is how good they are at their craft. That's right, writing software is a craft, one that requires a degree of talent, skill, and aptitude to do well. The difference between developers in this area is not negligible; on the contrary it can be massive.

Let's consider this analogy; suppose you find out tomorrow that you have a deadly medical condition, and you need life-saving surgery to treat it. Will you be happy to go to just any doctor? Why does it matter where they studied, or how many years of experience they have, or if they've won awards or received good reviews? If you pay attention to that sort of thing, it's probably because you believe there can be great variation in the practice of medicine. There's a handful of really fantastic doctors, a small group of great doctors, a large sea of average doctors.

It might seem like an exaggeration to make this comparison to software developers. I am more concerned that it's an under-statement, and that the variation in developers is even larger. This is not just my theory; it is a classic concept in software development, often referred to as the "dynamic range". I think it is summed up well in this quote by Steve Jobs:

“I observed something fairly early on at Apple, which I didn’t know how to explain then, but have thought a lot about it since. Most things in life have a dynamic range in which average to best is at most 2:1. For example if you go to New York City and get an average taxi cab driver versus the best taxi cab driver, you’ll probably get to your destination with the best taxi driver 30% faster. And an automobile; What’s the difference between the average car and the best? Maybe 20%?  The best CD player versus the average CD player? Maybe 20%? So 2:1 is a big dynamic range for most things in life. Now, in software, and it used to be the case in hardware, the difference between the average software developer and the best is 50:1; Maybe even 100:1. Very few things in life are like this, but what I was lucky enough to spend my life doing, which is software, is like this. So I’ve built a lot of my success on finding these truly gifted people, and not settling for ‘B’ and ‘C’ players, but really going for the ‘A’ players".

What explains why the dynamic range in software development would be so high? Part of the answer is that software development is about problem solving, and people's ability to not just solve a problem, but to solve it well, to solve it quickly, and to do the work to solve it completely, varies tremendously. Keep in mind that in the real world, there is no "one solution" to a problem; there are many possible solutions. Some solutions will be clean and elegant; some will be confusing and sloppy; and some will look good on the surface, but be subject to issues that range from being costly to extremely damaging.

A seasoned developer, with many years of experience, may know how to tackle more difficult problems, see problems more clearly, and understand how to tackle them more effectively (such as the problem of security). On the other hand, a developer may have many years of experience but never get past solving basic problems, or learn better ways of solving those problems. Thus "years of experience" is not solely a reliable indicator of how "good" a developer really is.

Some developers go out of their way to constantly improve their problem solving ability and the effectiveness of their problem solving methods. This, I have found, is a hallmark of great developers. Great developers pursue excellence for its own sake, which I suppose can also be said for great people in any profession, but I find it is especially true in the software industry.

Without exception, when I reflect on the developers I have worked with who I trusted to do high quality work, they all possessed this trait. Not only did they care about quality, they insisted on it; they led others in the creation of it. They did not have to be told "to do a good job". Rather, they would be upset if they were not given the necessary time and freedom to do a good job. This is a far cry from the "average" developer who just does exactly what they're told to do, who is fine to take shortcuts to minimize their work, or who lets mistakes slip into their work through inattention.

I have also observed that great developers have a passion for their craft that extends beyond the workplace. It is a part of their lives. Again, without exception, the great developers I've worked with are always doing something "on the side", on their own time, to hone their craft. Some write books, some have websites, some write hobby apps, some go to seminars, compete or volunteer in software competitions, join clubs dedicated to programming languages, teach their kids how to program, and so forth. 

Now if you can find developers strong in these dimensions I've mentioned, trust me when I say that they are valuable assets worth hanging on to. Be smart and don't settle for average developers and then expect to create "industry leading software". That won't happen, and it's more likely you'll end up embarrassing yourself. 

I remember on one occasion the firm I worked for took over a project from a prestigious and highly regarded digital consultancy. For some reason, in spite of the big name, the client was not happy with their work. When my team looked into the source code we could see why; it was very amateurishly written, with many bugs, and brittle to the point it would stop functioning properly if even the smallest change was introduced. Modifying this codebase to do what the client wanted was time-consuming and expensive, and there was not much we could do other than re-write large parts of it. Incredibly, the prior developer had commented the code with his name, so we Googled him. It turned out he was an offshore developer, and based on his LinkedIn profile, a relatively inexperienced one at that. This was probably his first big project (and one clearly over his head).

This game is being played out all over the industry right now. Big firms are relying on the strength of their names to reassure clients and customers that their systems will be "top-notch", while at the same time turning around and offshoring the work to junior developers, hoping to pocket a bigger margin, and that no one will notice the difference. I call this the "Short Term Gain/Long Term Pain Dilemma", and it is firmly entrenched.

To make things worse, the gains from offshoring is less than expected. First, while it's true that top-notch developers can be found offshore, they also command higher prices (which defeats the premise of offshoring in the first place). Big companies who are offshoring are often looking for the "cheap" developers. These are usually developers with less experience, who are used to playing junior or support roles on teams, or who are fresh out of college. In their defense, many of these companies don't know what they're actually getting, because offshore firms are very good at convincing their clients that their staff is the best of the best. They are experts at pumping up resumes and pitching their staff. I've seen many companies take the bait and watched their projects gradually lose speed and get mired with problems, negating much of the "cost savings".

There's more to this than the strength of the individual developers. Another salient problem is that it's harder to build a strong team when part of it is on the other side of the world. Getting a team together in the same place, working side-by-side, and collaborating with one another will always beat a group of individuals working in isolation; it will beat a group working remotely; and it will beat a group separated by timezones or continents. I'm not saying it's not possible to do the other things; only that co-located teams have the edge in effectiveness, productivity, and working relationships. Physical distance creates psychological distance. It's just the way things work. It's harder to "feel" like a team, or to bond like one, when people are just a voice on the phone that you talk to once a day.  

If there's anything I've learned from managing software teams for over a decade it is this: great people and great teams are everything. Whatever it takes to have both is worth the cost in the long run, by a wide margin. This is more important than process, tools, platforms, and all the other things we get fooled into thinking are the "secrets" to success. With the right team, software development is easy, the quality is high, and the customer is happy. With the wrong team, software development is stressful, filled with problems, and is a constant process of putting out fires and dealing with upset stakeholders. One team virtually manages itself and holds itself to a high standard; the other must be micro-managed and still manages to create many frustrations along the way.

In our business, people are our biggest asset. Not investing in them, not listening to them, or treating them as interchangeable commodities is a good way to ensure that that your company will ultimately NOT be known for being innovative, experts in your field, creating quality products, or being a place where people want to work. As a final thought, I'll leave you with another of my favorite quotes by Steve Jobs: 

“It doesn’t make sense to hire smart people and then tell them what to do; we hire smart people so they can tell us what to do.”

Do you have the type of people who you trust to tell you what to do?