24 May 05 - 10:05The "Any" process

"People trump process" is a wholesome truth of software development. Grady Booch offered the following formulation: "Good people with a good process will outperform good people with no process, every time."

Another frequently offered cliché: "If you hire only good people, then of course you will succeed with any process." I've heard this from more than one Agile critic - and, tellingly, from more than one Agile advocate. (If people agree on something, even though they're supposed to belong to opposite "camps", something odd is going on.)

So here's my take: there is no such thing as "no process". There's no such thing as "any" process - it's like the "any" key on the keyboard that so many computer users waste untold hours looking for.

Suppose that you've seen a group of people put together for a project, gave them an idea - rough or detailed - of the requirements - and the project was eventually declared "complete". Something happened in between. That's what a "process" is: it is a description that involves inputs, outputs, and some transformation of one into the other. If you can look at a project and describe it, at some level of detail, in these terms - you have a process.

What you don't necessarily have is procedures. That's a different beast, too often mistaken for "process". Procedures are rules ("do this"), with expectations of compliance ("every time") and penalties ("or else").

In some areas, "good people" can function without procedures. You need procedures for, say, conducting a nation's census. You don't need them for a picnic. You may not need them for every conceivable software project.

What you will have in every one of these cases is a process, that is, the possibility of describing what people actually do, at some level of detail, in terms of transformations. That's a process, which in some cases may be usefully described in detail as a set of more detailed processes.

If you observe "good people" on a software project, chances are that they are "good" because of what they know. In particular, they know which inputs, outputs and transformations it pays to pay attention to. They know how to put together smaller transformations to ensure that the larger one, from "idea" to "completed project", goes as expected.

When we say "people have a process", that's shorthand. We mean that there's a good way to describe what the people in question are doing in terms of inputs, outputs, and transformations. The description is "good" because it can be shared with other people, and because for those other people, knowing that description makes a positive difference in their performance.

So, the million dollar question is, if in general good people are "good" because their processes are effective, what do they have in common - if anything ? Are there any regularities to what teams of good people actually do, in the ways they transform inputs into outputs ?

That's the kind of "any" process which will reliably enhance the performance of competent people and make them "good people".

- Management - six comments / No trackbacks - § ¶

13 May 05 - 13:37Cobol is to Java as Java is to what ?

I mentioned earlier I've been teaching Java to people steeped in Cobol, mainframes, batch processes, and procedural programming. Just how steeped is an interesting question. The first two days of the training project are over, I've made the acquaintance of the group, not one "slow" person in the lot; I expect they've been keeping up with other technology, even from afar.

An interesting aspect of such a project is figuring out how to present, so that they make sense, aspects of the Java language that would strike no one as particularly "novel", that so many developers take for granted - but that nevertheless represent a sizeable gap with respect to Cobol.

For instance, based on previous experience with students who had a similar background, I knew I might need to explain how the call stack worked, before introducing heap allocation as a further complication. I showed the slide with a picture of the stack and asked if that was familiar. People nodded. I wrote, in near-Java, a recursive implementation of "factorial" on the flip chart - and asked if that would work. One student said, "Not it won't - you'd need an extra static variable, wouldn't you ?"

So I walked through one specific call to factorial, showing how stack frames were allocated, where intermediate values of the parameter would be go, where intermediate results would go, and so on. Some of the students knew this stuff from CS classes, some of them even remembered parts of it - but I needed these notions, and the visual representations that go with them, to be in the "active" part of their memory while we'd be looking at more specifically OO notions, such as object construction (which involves dynamic allocation, which involves contrasting the stack and the heap, and which also raises the question of the difference Java draws between "primitive" and "object" types).

None of this is strictly necessary. You can implement recursive algorithms in Cobol by emulating the stack; it "just" makes programs look tidier to have the mechanisms embedded in the language. The same could be said of many "features" of programming languages that Java programmers are quite happy to do without in general, and emulate when they need them; closures, first-class functions, and so on.

Then there's this idea that Java is the new Cobol. So, not for the first time, I'm wondering which language features of Java will come, in a few decades' time, to seem as antiquated as Cobol's static memory and lack of recursion now look. And, on the other hand, which "novel" features will slowly become accepted to the point where everyone takes them for granted. And on the gripping hand, perhaps every important feature a programming language could have has already been invented, and the best we can hope for is that we'll eventually converge on a particularly effective combination, or a few particularly effective combinations.

- Object Lessons - three comments / No trackbacks - § ¶

05 May 05 - 15:35"The Real World"

You may have heard that phrase before. "Your idea sounds neat in principle, but it'll never work in the real world." This can be disconcerting. Is that person accusing you of living in an unreal world ? (Some part of me - call him Calvin - might rub his hands and think, "I hope it has dinosaurs in it, and aliens and stuff.") The phrase sounds dismissive - it's tempting to think that your interlocutor is simply not willing to listen to you. Perhaps that's the case, and you're better off ending the conversation - but could you imagine a more constructive interpretation ?

The most generous interpretation I can come up for the use of that phrase - which, after a quick search through my outbox archives, I must confess I have used myself on occasion - is that it serves the purpose of "grounding" some aspect a conversation.

If you're saying something and I make a reference to the "real world", my intent is to advise you to become aware of your feet pressing on the floor, of a huge ball of dirt somewhere beneath, of gravity quietly working to keep the whole thing together, and so on.

You may take the advice, or disregard it. It's all one world - your ideas in it are no less real than the floor or the earth. Ideas can have a pretty big influence on the world. Impractical ideas even.

- The Universe And Everything - four comments / No trackbacks - § ¶

04 May 05 - 17:36Meet people where they are

This isn't an original thought; if you propose to someone else a change that you yourself went through with success, you'll need to put yourself in the other's skin and see how confusing, scary or uncomfortable that change appears to them - not to focus on how wonderful things look to you now, from the other side of that change. Dale's post on the topic is old enough by now to rate as a classic. But I'm in a situation now where this idea applies, in spades, so I'll contribute a concrete example.

I've been retained to teach object-oriented programming, Java, and Web programming to people who have been and are still writing COBOL batch programs for mainframe computers. Different worlds! These people went through some training already a year ago, attempted to migrate one small application to Java and the J2EE infrastructure. They didn't get anywhere. The Java training was so much water over their COBOL feathers. They told me that they'd purchased the training from one of the large training companies, so I can imagine why that is: they met someone who, instead of meeting them where they are, taught them Java from the "standard" background.

The thing is, if you look at some course material on object orientation, you'll notice that a lot of it assumes familiarity with concepts that are pervasive enough in C++, or even more "classical" languages like Pascal, that a lot of programmers and trainers, especially if they haven't been around long, take them for granted. Little things like the call stack, the heap, pointers, functions with return values and formal parameters, and so on.

The problem is that COBOL has none of these things. That trainer might as well have been talking Greek to the COBOL folks. It has static allocation, what amounts to GOSUBs for breaking programs into smaller pieces, no recognizable notion of local variables or function parameters, an interesting syntax, and remarkably powerful facilities that work in ways not at all obvious to someone raised moslty on languages of the Algol family.

When, before putting together my training schedule and materials, I started trying to run, then modify, then write little bits of code in COBOL, it felt like Greek to me - ancient Greek, that is. I was thrilled: that was precisely the point. The one way I could see of meeting people where they were in this case was to feel their confusion at a strange language, only with the tables turned. I then mapped out how I was going to lead them to objet orientation with COBOL as a starting point.

We'll see how it turns out; the course is formally starting next week.

- Private Life Of The Brain - four comments / No trackbacks - § ¶

03 May 05 - 23:30Perfect Technology

Possibly more often than I should, I indulge in a frustrating pastime: trying to set aside everything I know, or think I know, about software development, and working out from first principles what is certainly, definitely true about things like "design" or "requirements". Occasionally, it has its rewards - you hit upon ideas which simplify a whole complex area and let you think more clearly about it.

A while ago I read and appreciated an article on "Project Chartering" written by a delightfully eccentric consultant named III (pronounced "three"), whom I had the pleasure of meeting earlier this year in Phoenix, AZ. We discussed in some more detail many of the ideas in his paper.

One idea which particularly impressed me for its simplifying potential was the idea of "perfect technology". The thing is, everyone turns out to be able to give instant answers to a raft of questions about "perfect technology". Ask anyone: what is the storage capacity provided by perfect technology ? "Infinite" - the answer will come in an instant. How often does it break down ? "Never." What is its response time ? You get the idea, I'm sure.

What III suggests is that there is such a thing as "true requirements", as opposed to other things a customer might discuss, which might appear to be requirements or might even come labeled as requirements, but actually turn out to be something else - for instance, technology preferences or established procedures - and which will waste everyone's time if we try to analyze them as requirements. A good test, then, of true requirements is that they can be expressed in terms of "perfect technology".

Think about it. And go read III's article, it's quite interesting.

- Software Development - one comment / No trackbacks - § ¶

02 May 05 - 20:21Why good ideas fail (and how they might succeed)

Suppose you hear about a Better Way of doing business. An interesting example might be Extreme Programming, which suggests that software development contracts might be handled differently than they usually are, to significant benefits. I'm only using that as an example - there's a very general question you might ask about such ideas, especially ideas which have been around for a little while.

The question is: "If this is such a hot idea, then organizations using it should outperform others. Organizations are subject to a sort of economic Darwinism - the better performing ones stick around, the others die. So we should be seeing more organizations using this Better Way. Why hasn't that happened yet?"

You might be tempted to answer, "because it wasn't such a hot idea after all; looks good but doesn't work in the real world, back to the drawing board".

That's a premature conclusion, and you could be missing out on what's really a good idea - if you knew how to cash in. To understand why, all you need is a thin slice of Game Theory.

First, we need to discuss the Prisoner's Dilemma. In the simplest form, this is a two-player game with two possible moves, "Cooperate" or "Compete".

The game is about winning as many points for yourself as you can. Suppose the two of us are playing - if you choose Cooperate and I choose Compete, I win five points - and you win nothing ! Frustrating ? But on the other hand, if you choose to Compete while I Cooperate, you win the five points and I get nothing. If we both Cooperate, each of us wins three points. Finally, if we both Compete, each of us wins a measly one point.

The names of the strategies are merely suggestive - they have nothing to do with how best to play the game. What matters is the "payoff matrix", as below:

You'll want to spend some time thinking about the game, if you have never had an occasion to do so before. (Play it online at the link provided above, or play with friends and colleagues, read about it...)

The Prisoner's Dilemma yields a good model, though hugely simplified, of a situation in which good ideas fail. Imagine, if you will, an economy based on playing that game. Imagine further that everyone - all the businesses in that economy - typically play the Compete move. With any interaction, any two businesses make some little profit - one point.

In that context, Cooperation is demonstrably a Better Way: it would be possible to get at three points rather than one out of every interaction if everyone was using it. Why doesn't everyone start doing it ? Because it makes no sense for any one business to switch their strategy to Cooperation, unilaterally. They would be "rewarded" by going up against Competitors and gaining no points at all from any business they did. It would take a coordinated decision for everyone to move to Cooperation, but in a "free" economy there's no opportunity for such coordination. (The simplest game theoretical model of a "free" economy assumes that businesses don't communicate about their strategies - so you couldn't have a conditional strategy of Cooperating only with Cooperators. More realistic, and more complex models exist - but then I'm writing this precisely to motivate you to take an interest in them.)

This type of reasoning is representative of one common objection raised when discussing, e.g. Extreme Programming: "What you suggest works in theory, but in the real world customers don't trust software contractors and so they will prefer up-front specification to ongoing customer involvement; that being the case, you just have to go along if you want to do business." And this line of reasoning is quite valid, considered in the light of game theory.

This seemingly desperate situation is called a Nash equilibrium, after John Nash, the mathematician who studied them. They're an important result in game theory because they allow you to ignore most of the myriad combinations of M strategies in a population of N players: you can expect that any given population will stabilize at a mix of strategies that form an equilibrium. They help explain why populations of people or businesses may converge on strategies that are not necessarily Better Ways.

Robert Axelrod's book, The Evolution of Cooperation, tells the fascinating story of a series of experiment involving games of Prisoner's Dilemma conducted as computer simulations and "tournaments". I strongly recommend the book as your first stop if you're interested in learning more about the game, or about game theory. It contains little that looks like "theory" - equations or long logical arguments - but mostly focuses on the results of simulated games, pitting against each other strategies submitted by some of the best mathematical and scientific minds in Axelrod's distinguished entourage. The results have practical relevance to anybody interested in how Better Ways spread... or don't.

In particular, Axelrod shows how Cooperation can in fact - almost against all hope - win over Competition: our models (and Axelrod's simulations) need to introduce the notion of territoriality - i.e. the idea that anyone's interactions in the game will mostly be with "neighbours". (They're not necessarily neighbours in physical space - "idea space" could be a good way to think of it.) In Axelrod's simulations, it turns out that small clusters of Cooperating neighbours can eventually "invade" a population stuck at a Nash equilibrium.

Based on this, for instance, you might expect that the full benefits of Extreme Programming would arise if and when a "community" of businesses interacting mostly with each other in that particular way. That is, assuming that this particular way is indeed a Better Way; but that's a different question. At any rate, it is one that can't be settled by saying "if it really worked, everybody would be doing it by now".

- The Universe And Everything - seven comments / No trackbacks - § ¶