7-20-03 The Ideal Programmer

In January of this year, Scott Meyers (author of Effective C++), Bill Venners, Chuck Allison and myself convened a technical summit in Portland called "Writing Better Code." This was a small group that we invited from among people who had spent time thinking and writing about the issues around creation of code and projects. The list included:

• Scott Meyers
• Bruce Eckel
• Bill Venners
• Chuck Allison
• Joshua Bloch
• Alistair Cockburn
• Dave Thomas
• Andrew Hunt
• Pete McBreen
• Angelika Langer
• Chris Sells
• Kevlin Henney
• Randy Stafford
• Matt Gerrans

The summit began around Scott's observation that despite many years of practicing the craft of programming, the most fundamental concept in computing – what Dave Thomas and Andy Hunt call the "DRY" principle ("Don't Repeat Yourself" – which includes but means more than "don't write the same code twice in more than one place." It means: "there should be one authoritative repository for each concept in a program.") – is practically ignored (and perhaps not even understood) by a large percentage of programmers. Of course, there are lots of other principles which are also ignored or unknown, but Scott's observation was that if we can't even get people to understand and follow the DRY principle, what hope is there for anything more sophisticated?

Thoughts in the Machine

I think that a large part of the problem is that the concept of programming is diametrically opposed to the patience and effort necessary to produce good code. The concept is that we wave our magic paintbrush and the perfect picture paints itself in no time at all. Admittedly, we're a lot further along that path than we used to be, and I spend a fair portion of my own time pursuing tools to increase productivity. But regardless of how far along the path we get, one glaring fact remains:

Programming is about communication between humans.

I think most of us fall into the easy trap of thinking that all we have to do is successfully communicate with the machine – learn the necessary rules, no matter how arcane (and I know there is a group for whom "the more arcane, the better") – follow them, and get the end result, which is the working program (the code is only a step on the way to this).

The reason Extreme/Agile methods are so important is that they revolve around the idea of human communication: between the customer and the team, between the members of the team, and between current members of the team and future members of the team. In addition, the executable program is not the endpoint. Instead, the code (and associated, automatically-run tests) is the primary artifact. If you've read about the Extreme/Agile methods, you know that they have a propensity for treating all artifacts except the source code/tests as ephemeral, to be thrown away as soon as possible and not dragged around, wasting time under the pretense of maintaining them. The goal is to make the code as readable and understandable to humans as possible, so that:

1. It's possible to understand and review the code and its design before releasing it.
   
   
2. New programmers can understand the code without excessive effort, and thus come up to speed on the team in a reasonable time.
   
   
3. When the system needs changing, it's not too hard.

Programming is not about racing to throw together a solution. It's about changeable systems.

The ideal programmer knows that they want to create something that others (not just the computer and that programmer) will understand. This requires knowing when to spend the time in the right place.

I'm not saying that we don't want to move as quickly as possible. As I've said, productivity seems to me to be the greatest promise of new programming technologies. But even when I'm creating a support tool that no one but myself and possibly a few others will see, I know from experience that this tool will undergo changes as I discover new needs. It doesn't really matter what it is – if you are going to use it, it will probably need to be changed. A solution that I put together in a short time will either evolve under use, or spawn other solutions. Either way, understanding is the key.

Crisis of Management

Personally, I'm undergoing a small crisis. Over the years I've tried to work with people, and much of the time I've had varying degrees of failure. Every once in awhile there's a success. For example, one intern from last summer turned out to be very beneficial and give me leverage without costing me effort (and he still consults for me). Working with Chuck Allison, my coauthor on "Thinking in C++, Volume 2" has been a successful coauthoring experience, and we also had an amazing intern, Jamie, from the university where Chuck teaches. Most of the seminars that I've taught with others have been successes. But the failures have often made me wonder whether it's worth trying (perhaps I need to try harder or not try so hard). It has started to give me a little more sympathy for the miserable success rate of most book publishers (typically around 10% of books break even, and closer to 1% are financially sucessful, based on numbers I've heard). Sometimes it seems like people want to be treated like machines, plugged into a job that doesn't require a lot of thinking or creative effort so that they can think and create on whatever they really want to do (I often wonder what the world would be like if everyone followed their hearts and ended up doing what they loved, instead of listening to people who told them to get real and take on a practical career).

Numbers against you

Some daunting statistics came up at the summit (I can't offer references on these; maybe someone can email me references that bunk/debunk them). My favorite, because it's such a wake-up call, is that 5% of the programmers are 20 times more productive than the other 95%. Then there's the idea that the majority (probably that other 95%) of programmers don't read books on programming, and perhaps only have the language manual at their desk (this was naturally of concern in a group made up largely of book authors). The book Peopleware is filled with statistics from real studies which shock and surprise us when we think we know how people and projects work. And the big statistic, which you can understand if you believe the others: something like 50-90% of programming projects fail – don't do what they're supposed to, miss their window of opportunity, are cancelled or implode/explode by themselves. The statistic is hard to track down, since companies are not in the habit of trumpeting their failures, but if I just take into account the stories from the people that have come to seminars and that I've talked to at conferences, over 50% is not hard to believe.

Love it or leave it

How many people in our profession really love what they do? My perception is very skewed because I meet the people that read books and make the effort to come out to seminars and conferences. There are a significant number of people who never leave the home-office loop but still read books and magazines and struggle with new concepts, discussing them with co-workers. And there are those mythical unseen creatures, of whom I've only met when giving a presentation inside a company (usually a larger company, since it's a good place to hide) so that the effort is limited to walking from the cubicle to the auditorium. I recognize them because they ask questions that amount to "why should I learn anything new?" They don't want to read or struggle with new concepts because it is uncomfortable (It makes your brain hurt a little, which I think means you're growing new neurons. Think of it as an investment against dementia in later life). If you've seen the movie "Office Space," you know them – that wonderful character who was fired years ago, but keeps coming into work, keeps getting his cubicle moved and made smaller, and keeps wondering where his stapler is. I'm not worried about offending him (yes, him, because as my girlfriend keeps wondering about, we seem to be an unusally male-dominated field) because he doesn't read and so he won't read this. But I don't think this is a person who is part of our profession, but rather someone who just exists in our profession. And probably because, long ago, someone said "get into programming, it pays well."

A computer science professor noticed the resistance to learning (including his own) and asked a psychologist about it. He wrote a fascinating little essay about the difference between perfection-oriented and performance-oriented individuals. I have found myself to be either performance-oriented or perfection oriented at different times and in different situations.

Meeting of minds

The "Writing Better Code" summit was self-organizing, like a number of summits I've been doing lately. The summit idea came from a discussion I had with Martin Fowler where I was describing long-term, grandiose plans to create some kind of self-organizing conference and Martin interrupted me and said "why don't we try an experiment on a small scale?" (This is an example of why Martin is so good at what he does – he managed to break me out of my big, overwrought schemes and do a reality check on the concept). We've done the experiment, very successfully, for the last few years, as a series of invitational technical summits. Virtually every participant has come away saying this was one of the best (if not the best) learning experiences of their lives, and all we did was share information and experiences. I usually liken the summit experience to all the best hallway conversations that you get at a conference, without the typical noise (low-signal, marketing-oriented activities and events, etc.) of the conference itself. Ironically, the most successful discussion of the WBC summit came when I stepped outside the bounds of our self- organization and said I wanted to hijack the group briefly, and go around the room asking people how they interviewed programmers. Bill Venners turned this into an article on Artima.com, which has been one of the most popular articles on that forum.

Solving or Accepting?

Back when Gerald Weinberg was running the "Problem Solving Leadership" (PSL) seminars, I attended one that happened to be held here in Crested Butte. This was a 7-day workshop that involved all kinds of things related to management of technical projects and companies (the ideas actually applied to just about any kind of management, but Weinberg's audience has always been technical and so those were typically what he drew). One of the most preconception-shattering exercises involved spending most of the day simulating a company, including manufacturing, marketing, upper management, the works.

This was not a typical group of people who had just evolved into management positions. This was a self- selected group who had been reading new management books and trying to make themselves better by coming to this workshop. So you might expect better results than what we got. With the best of intentions, in less than a day, our two companies had rapidly devolved into the worst examples of any company you've ever worked for. Some people actually got very emotional and quit (remember, the company only existed for one day!). The typical labor/management stratifications occurred. Basically, it all went bad very quickly.

You might feel, as I did, depressed that a group of such well-intentioned and educated people could make such a mess of things in such short order. It seemed like a colossal failure. But Weinberg and his associates considered it a success. They expected this to happen, and they even had little places where they would apply pressure on the company if the company wasn't degrading fast enough. Their whole point was to show you how things failed. They weren't really trying to give us a formula for success, but rather make us see and understand failure (Weinberg's maxim from Secrets of Consulting is "Things are the way they are because they got that way ... one logical step at a time"). This felt a bit like last week's Designing Objects & Systems seminar, where Bill and I spent a lot of time emphasizing that design is an art and there are sometimes helpful guidelines, but not formulas for success, and there are many more ways to get stuck than there are to succeed.

Haunted

I have pondered the company simulation at the PSL seminar for years since. It still bothers me, sometimes a lot, that we crashed and burned so badly. It felt like a mockery of our attempts to do better. Similarly, the "Writing Better Code" summit seemed like a failure compared to the summits that Martin and I have put on. But those summits had a strong technical flavor; we were talking about and exchanging experiences on frameworks, architectures, and designs that we had all encountered, so it was very concrete and if we just learned something from someone else's experience it was often more productive that days or months of struggle on our own. On the other hand, the WBC summit was really about psychology: why do programmers write poor code and don't seem to care about it, and what can we do to convince them to write better code and to care? At best, we succeeded in enumerating the problems that we had seen, so compared to the other summits we reached no conclusions. But perhaps the struggle was the important thing, and like Weinberg's PSL company simulation, we all needed to have these ideas inserted so we could struggle with them over the ensuing years.

Sprinting

The best successes I've had when working with other people have been short term, mostly seminars but also brief bursts of focus on a project, like the "Sprints" that I've participated in for Zope3 and which are part of the Scrum process. I've also done this kind of thing with others on my own project, where the understanding is that we were going to work exclusively on a particular project for some short period of time, in order to move it forward. It seems like people can stay on track as long as the time is not too long, and since I am personally always juggling a number of projects, it's much more practical for me to put everything on hold and focus on one project with others, than it is for me to be constantly interrupted by the need to review or guide a project that is running in parallel to whatever I'm currently working on. This makes me think that the ideal manager is someone who doesn't have a project of their own, and is idle (that is, available) most of the time, and thus can stop whatever they're doing and come review or guide or un-stick your particular part of the project.

So I've managed to suggest a description of the ideal manager, but not so much about the ideal programmer. However, I suspect I'll get more ideas about this in the future. Stay tuned.