1-1-04 Why we use Ant (or: NIH)
For Thinking in Java, 3rd edition, I was determined to make my life easier by
employing others to do some of the spade work. Thus began "the internship
program," an idea I had long been convinced would lead to higher productivity
and less effort on my part.
I regularly have to remind myself that it wasn't a total disaster, by using a
bit of mental indirection. Because I managed to make just about every blunder
possible, in the biggest possible way, I'm left with learning experiences to
draw upon from years to come. From the interview process all the way through the
execution of the program, I'm sure I made most of the mistakes that more
experienced managers would recognize.
It's likely that most managers continue to make these same mistakes in some
form or another. In my early years, when I was working as an employee, I had the
common techie opinion that management was a stupid practice. Every Dilbert
cartoon has placed "the pointy-haired boss" into the common lexicon. Technical
people are the ones who get things done, and the best strategy is to promote the
pointy-haireds into positions where they don't gum up the works so badly.
What I have learned, first by accidentally taking courses for managers (I
admired Gerald Weinberg's book "Secrets of Consulting" so much that I wanted to
take a workshop from him, and the only ones he gave were really about
management, so what the heck. I morphed from a journalism major to a physicist
to an engineer in much the same accidental fashion), then by trying my own hand
at management in different forms, is that it's just very, very hard to get it
right. For quite awhile I didn't even know I was managing. So sure, most
people (like me) end up looking like stupid managers, maybe forever. I am sure a
number of people I've worked with still feel that way, but a college friend
evolved away from that attitude – over the space of about 10 years –
by becoming a small-business owner himself. We went from a total
misunderstanding to being able to discuss the issues and difficulties as fellow
travelers.
My biggest mistake with the internship program was in not remembering what I
was like at that age, and with my experience up to that point (a few summer
jobs, but primarily doing homework exercises, under deadlines, with no real
sense of quality of problem-solving. Basically, the world view of most of my
college professors, whose job was to process students, not build creative
thinkers). After the fact, my friend Matt gave me further insight. We were
physics undergraduates together, but he went the extra miles to get a Ph.D. I
think the reason it took him so much longer – that he loves having fun
– is the same reason that he's such a popular and effective teacher now.
Basically he pointed out that, when someone's working on a project, you pretty
much get whatever skills, abilities, and work habits that they have right now.
If someone has to be shepherded and given a great deal of feedback, for example,
those things probably won't change much for the duration of the project,
especially if it's short.
Many people use this to justify the "you've got to hire the right people"
approach, taking the extreme position that people can't change. I don't believe
that (otherwise I wouldn't be trying to deliver learning experiences), but I do
see that change comes slowly, and for a particular project you may be able to
change a few things if you use tricks like the morning scrum and the weekly
report, automatically running unit tests every hour, or tests that are run
during checkin, as well as training and consulting from outside the company to
provide new insights as to how and why things might be done more effectively.
But when you staff up a particular project, I think you have to take the
attitude that "past performance is the best indicator of future performance."
People can change, but it's safest to assume that they won't choose to do so on
your watch.
The internship program cost me far more, in both time and money, than I got
back from it. Except for the lessons, I would have been far more effective doing
everything myself, with one exception. One of the interns, Mark, grew up in a
family of programmers; I think this included all the people in his family. So he
had thought about these things and been immersed in the culture for much longer
than the norm, and he came not only with skills, but also the interest and
ability to pick up new skills. This is probably the most important feature in a
programmer, and also something that requires the longest to develop.
One of the valuable things that Mark accomplished during the program was the
creation of a program to generate Ant build.xml files. This was a variation on
something I had created for C++ to aid in the generation of makefiles (For the
first 2 editions of Thinking in Java, I also used this tool and makefiles). I
called my tool MakeBuilder, so it made sense to call this one AntBuilder. It
wasn't actually built from scratch, because I had taken MakeBuilder through many
evolutions and had studied the problem somewhat extensively, thus much of the
"specification" was well-tested. Mark took the same form factor and learned both
Python and Ant in order to create AntBuilder, which we then evolved through many
iterations. If you look at the comment tags in Thinking in Java, 3rd edition
you'll get an idea of what AntBuilder does – takes the tedium and errors
out of building antfiles. In the end, it turned out to be one of the more useful
tools during the development of the book.
Based on experiences I had with an updated version of MakeBuilder we used in
Thinking in C++, Volume 2, working on the TIJ3 solution guide, anticipating
needs for Thinking in Patterns, and considering turning AntBuilder into a
commercial product, I've been pondering some improvements for the tool,
including fundamental architectural changes that will make it easier to add new
directives and to adapt and add support for additional Ant features.
Mark, however, has been getting frustrated with the constraints of Ant: "I've
been trying to think of a good way to make AntBuilder more extensible in terms
of adding tasks and properties. ... We can keep adding more flags to AntBuilder,
but in the end, we're constrained by Ant's form. Some parts of the system are
well suited to Ant and some are not. Anything conditional is complicated and
that complexity seems to spread to other areas of the system. For instance,
because a class may be compiled or not based on the presence of a jar, a whole
separate target has to be created for any classes that want to use compiler
arguments."
His suggestion? Let's invent our own build system, and eliminate the problems
with Ant.
The creator of Ant writes here
about his regrets in (A) using XML and (B) not making Ant more powerful by incorporating
enough language constructs. I agree wholeheartedly on both counts, and yet I'm not ready
to undertake the project of creating a new build system, much as I would like to have a better one
for my own use.
We've all been tempted by this variation of "Not Invented Here." The idea of
building a solution completely from scratch and solving all problems suffered by
previous attempts is enticing. There are very important reasons not to do it,
though.
1. It's never as easy as it seems
Things always seem like they ought to be easy, and they almost never are.
This is the very reason that I try to arm myself with the biggest guns I can
get: notably Python, but as time passes I
understand that anything that adds even the smallest extra bit of ceremony must
be avoided, because you have to be able to keep it all in your head. I've been
told way too many times that "doing it this way is so easy" because someone is
ensconced in that particular world view, and all you have to do is recompile the
kernel or something like that and voila! But when you add layers upon layers of
this stuff and this kind of thinking, you end up with a maze of twisty little
passages.
So I don't go looking for extra work and complexity. It chases me down no
matter how hard I try to avoid it.
2. It's probably not the battle you should be fighting.
Building a new tool to solve a problem sounds cool, and sometimes it's
actually the right solution (as AntBuilder is, I believe). But reinventing
something that's already been solved, even if it hasn't been solved that well,
is something that should be avoided if possible. I have consulted on projects
where the technologists decided that they want to solve a much broader problem
than what was assigned, and usually it's a disaster. In fact, I would
characterize the desire to solve a bigger problem than necessary to be an
important flaw in the technical management of the project, whereas a desire to
"do the simplest thing that could possibly work" is an indication, to me anyway,
of a depth of understanding of how hard it is to just get something working, and
how likely it is that you'll fail.
Let's consider a well-known example. This group of techies sets out to
program TV set-top boxes, but they decide that C++ is too lame a language to
accomplish such a challenging task, and that they can come up with something way
better. Fortunately, they work for a company that rewards them by turning the
result into a cult (the only cult the company has left, so they're now applying
the name of the language to everything, even when it doesn't make sense).
Unfortunately, the product hasn't been a big moneymaker, unlike, say, set-top
boxes might have been, or any number of products they might have made if they
had gotten a real-time operating system and started building something with C++
(I know it's difficult, but it's not that bad). Yes, we got Java out of the deal
but a smaller company with less-deep pockets cannot afford to do anything but
solve the problem at hand, as expediently as possible.
Now, the observation Mark makes that Ant is not really the right tool for the
job is appropriate, I think. Ant sucks on a number of levels (so much that the
original inventor became disgusted with it and went off to create something
better, but has since coasted to a stop as far as I know), but the most
important one I'll sum up by saying:
Noise matters
The excitement of having a human-readable data format (XML) overwhelmed a lot
of people, many to the point that they decided (probably from having to hand-
code HTML) that it should be human-writable as well. But it's not even
particularly human-readable; it's probably best described as human-tolerable. If
you need to delve in and investigate what's going on with your XML, you can,
just like you can go look at Java byte codes, or assembly opcodes etc.. XML is
easier than those things, but it certainly isn't a great way to write.
As I've said before, lots of programmers will say "all you have to do is this
here, and that there, and this other thing, and voila!" Each one of those things
is a bit of noise, and reduces your productivity. I'll maintain that XML should
never be used for something that is written by humans, just as you shouldn't try
to use it as a programming language, although people have been so in love with
XML that they've tried to do both.
One of the things that works well about make is that the makefiles are
terse, and there are lots of ways you can make them terser. This concept was
completely ignored by Ant in its headlong rush to embrace XML, and this is why
AntBuilder is necessary: people shouldn't have to write build.xml files by hand,
and I sure don't want to. With AntBuilder I can easily regenerate the build.xml
files every time I add a file to my project, instead of being forced to put on
my Ant hat.
3. Lots of people have tried and failed
The web is littered with attempts to replace make. Ant "succeeded"
with Java because it was first and desperately needed. But if you hunt around
you'll see all kinds of various designs for build systems, in various levels of
development or abandonment. Most of them make fundamental missteps like Ant did
by choosing XML; misunderstanding the problem they are really trying to solve.
Another important problem with Ant is that its XML-influenced declarative
syntax is limited. A build system is really a little computer program. Normally,
you look for conditions (a source file is out of date with its target is the
primary one for make) and you execute commands, but sometimes you need to
do things that are more sophisticated, conditional logic, that kind of thing.
Both make and Ant fall down here because, while it's possible to do these
things it suddenly gets a lot more complicated because that's not normally what
they do.
Automated build systems are very important to me, so I've given this problem
a lot of thought. I finally concluded that what I would like is something that
ensconces a real programming language so that I seamlessly solve whatever
problems I encounter without a sudden increase in complexity, or switching to a
new paradigm. But I also want all the basic things – establishing
dependencies, for example – to be expressed tersely, much like it is in
make, for example (and why not make a lot of it look like make,
since many people already understand that syntax?).
I suppose if I were given a budget and a bevy of programmers, I would ask
them to take make, which is open-source, eliminate the silly tab-vs.-
space thing, and meld it together with Python, also open-source. make
would only retain the simplest of its functionalities (primarily the way you set
up dependencies and execute simple commands), and as soon as you wanted to do
something more complex you'd be able to drop seamlessly into Python syntax,
using any Python libraries you want. I think that would probably be my dream
build system.
Although I would like this, I'm not going to build it anytime soon, because:
4. Politics matter
As a small operation, if a new tool comes along that's obviously better than
an existing one, I have very little inertia and can switch over easily. I've
been incredibly frustrated upon hearing that larger companies might still be
stuck with JDK 1.2. Or even 1.1 or 1.0. It seems so awful to be forced to use
antiquated versions of Java, with all those early mistakes in the language. But
these companies have good reasons not to trust the newer versions of the
language – heck, I don't trust the newer versions, from experience. But I'm
not creating mission-critical systems that can be brought down because someone
at Sun was overworked during the week that my important feature in the latest
JDK was being fixed. These companies have good reason to move slowly,
frustrating as it is to someone who likes to use all the new, sexy features. And
they usually move slowly with all their tools, including build tools.
So they've taken several years to standardize on "the industry standard build
tool for Java," and alas this is probably Ant. And there are legions of smaller
companies who could change from Ant but already know how to use it and won't see
the point. Sure, we could reinvent the wheel from scratch, but it's unlikely
that this will help too many people.
Later:
Martin Fowler adds to the argument given here in
his weblog.
I've skimmed the Maven docs, and it's hard to know what it's actually supposed to be.
The docs say something like "we do all this stuff," but it's kind of a hodgepodge collection. And here:
"Maven allows a project to build using its project object model (POM) and a single set of
Ant build files that are shared by all projects using Maven thus providing a uniform build system. "
So it doesn't replace Ant, it adds to Ant. And it appears that Maven also uses XML, which I still
don't want to write by hand.
The fact that they're not clearer about what they are trying to do is not promising.
