15 November 04 - 16:57Throw some Java salt over your shoulder

I'm fairly sure you could accurately gauge the maturity of a programming team by the amount of superstition in the source code they produce. Code superstitions are a milder form of cargo cult software development, in which you find people writing code constructs that have no conceivable value with respect to the functions that the code is meant to fulfill.

A recent conversation reminded me of an example I find particularly disturbing. Sample code for dealing with JDBC is particularly prone to being littered with this particular error, as shown below. (I suspect that is not coincidental; I'll be coming back to that.) I have elided most braces out for clarity and terseness - imagine that this is a cross between Java and Python:

import java.sql.*; 

public class JdbcSample {
  public static void main(String[] args) {
    Connection conn = null;
    try
      conn = DriverManager.getConnection("jdbc:someUrl");
      // ...more JDBC stuff...
    catch (SQLException ex)
      // Too often that is silently ignored, but that's another blog entry
    finally
      if (conn != null)
        try
          conn.close();
        catch (SQLException sqlEx)
      conn = null; 
  }

The "superstition" part is that setting the connection to null can have absolutely no useful effect; being a local variable, "conn" will become eligible for garbage collection as soon as it goes out of scope anyway, which the most rudimentary analysis of flow control reveals it will immediately after being set to null.

I am always particularly interested in finding out what goes on in the minds of programmers who write this kind of thing, because that will sometimes reveal the roots of the superstition. Most of the time, though, if you raise question in a design review the programmer will say something like "I copied and pasted it from sample code". This is how the superstitions spread - and it's also a red flag with respect to the team's practice maturity - but rarely an occasion to gain insight into why the superstition took hold, which is what you'll need to know in "remedial" training.

Now, the "null" concept, obvious as it seems, is a likely place for superstitions to accrete around. If you look closely, "null" is nothing but obvious. Comparing Java and Smalltalk, for instance, we find that they differ radically with respect to calling instance methods on null, or "nil" as it's called in Smalltalk; "nil" does have some instance methods you can call. Also, what is the type of the "null" value in Java ? It is a special type called "the null type", which looks like a sensible answer but incidentally breaks the consistency of the type system; the only types which are assignable to variables are the type of the variable or subtypes of that type, so "null type" should be a subclass of every Java class. (It actually works that way in Eiffel, as Nat Pryce reminds me - see comments.)

See also here for another example of a null-related Java superstition, also surprisingly common, as you can verify by Googling for "equals null".

In the case of JDBC, I would bet that idioms of resource allocation and deallocation inherited from non-garbage collected languages, like C, were the main force in establishing the superstition. Even people new to Java get used to not calling "dispose" or "delete" to deallocate objects, but unfortunately the design of the JDBC "bridges" between the object and relational worlds suffer from a throwback to idioms of explicit resource allocation/deallocation.

Owing to what many see as a major design flaw in Java, "going out of scope" cannot be relied on as an indicator that a resource is no longer in use, either, so whenever they deal with JDBC Java programmers are suddenly thrown back into a different world, one where deallocation is something to think about, like not forgetting your keys at home. And so, in precisely the same way as I occasionally found myself patting my pockets to check for home keys when I left the office, our fingers reflexively type in the closest equivalent we find in Java to an explicit deallocation - setting to null.

You may object that the setting-to-null superstition is totally harmless. So is throwing salt over your shoulder. While this may be true of one particular superstition, I would be particularly concerned about a team which had many such habits, just like you wouldn't want to trust much of importance your batty old aunt who avoids stepping on cracks, stays home on Fridays, crosses herself on seeing a black cat, but always sends you candy for Christmas.

- Software Development - six comments / No trackbacks - § ¶

14 November 04 - 22:34Overtime - I Would Prefer Not To

The blogosphere has been abuzz with the "EA Spouse" story, wherein the work conditions at a well-known video games publisher are revealed as better than those of a third-world sweatshop, but miles short of humane, honest or even very effective from a business point of view.

I've been having an extended conversation with a French colleague, who is interested in the "development dojo", an initiative to provide a radically different form of training in the techniques of software development. He is quite interested, but the schedule we are contemplating (6PM to 8PM once a week) would require him to leave work early. He is concerned that his colleagues and managers would not understand his leaving half an hour early.

I would like to think that it's just synchronicity. But I'm afraid that the coincidence of the conversation and the story over such a short interval suggests that the software industry is still far from mature regarding this issue of overtime.

What gets me is that, to a large extent, we do it to ourselves. I should know. Considering the consequences of chronic deprivation of reasonable time for rest and recuperation, there is really no justification for giving in to real or imagined pressures from management.

Often, the reason is that we lack basic techniques for saying "No" without making a big deal of it. An effective one is Bartleby the Scrivener's : "I would prefer not to". If asked to do something that you have decided is not in your best interests, just decline. Don't explain; don't argue; don't defend yourself; don't attack your interlocutor either - just decline.

- Management - No comments / No trackbacks - § ¶

13 November 04 - 20:19Community of curiosity and respect

Hal Macomber points to Jeffrey Cufaude pointing out one of the crucial skills for any form of management or leadership:

Until we can listen to others with whom we might violently disagree with some degree of authentic curiosity, openness, and respect, little is possible.

I have nothing to add to that directly, but right this moment it feels good to be a member of a community of practice whose practitioners are committed, in spite of the difficulties in doing so, to the greatest possible mastery of this particular skill. It's still a somewhat fuzzy karass(*), it doesn't have too many hang-outs yet, but it's there all right.

(*) I use the term with gentle irony, and as a nod to another member of the same community, and one of the recent additions to my blogroll.

- Management - No comments / No trackbacks - § ¶

12 November 04 - 22:00Systems from the inside

There's a world of difference between the confidence you have in a system that has been designed for you, but which keeps you at is periphery; and one that you actually participate in.

France has remained stubbornly old-fashioned in the design of its electoral system. It's paper ballots all over - although there have been proposals, even plans for a live test, of an electronic system.

I was a non-voter for many years - I was too young by a few months to vote in 1988, which with the impatience of that age I found galling, and afterwards left me with the impression that politics would take care of itself without my involvement. I took it up again about ten years later.

The first time I showed up to vote it was in the nearby school, my precinct's polling place, and incidentally where my first son would soon be going to school. I had my first encounter with the people manning the ballot box, whom in later years I would get used to seeing there at every election - they gradually became permanently associated with elections in my mind. And this first time, they asked me this: "Would you like to come around and give us a hand after polls close ?"

I turned to my wife, bemused - I wanted to check if it was OK, she would have to take care of the baby - also I had no idea what I was expected or supposed to do. She nodded "Sure, go ahead", and I accepted without much knowing what I was accepting.

Later I counted votes. That's how the system currently works here - votes are counted by volunteers, usually selected amount younger people. It's a well- organized system. Tables of four people were assigned several batches of a hundred envelopes; two out of the four to open envelopes and read out loud the vote, two to write down check marks on ruled paper and write down tallies, cross- checking each other's tallies at the end. All four of us signed the tally sheet, which was then sealed as a valid count of votes.

It's interesting, in the light of my experience with this sytem, to consider the post-U.S. election news - candidates asking for a recount, discussions abuzz with rumors of fraud due to compromised e-voting machines, and so on. We've had elections that were close (the 1974 presidential race was decided by a margin of barely 1%, less than half a million votes) but I don't recall that fraud was ever an issue. I'm pretty sure that the widespread practice of having the voters "use" the counting system in that particular way changes how people handle defeat (or victory) here.

And the comparison certainly gives me no great desire to see e-voting systems deployed here.

- The Universe And Everything - No comments / No trackbacks - § ¶

11 November 04 - 20:29Architecture and pattern density

I've just finished Steward Brand's How Buildings Learn, which I recommend to anyone who intends to keep using the term "architect" in the context of software. There's a lot in there about emergent design, "deploying" early and often, negotiating scope, ongoing maintenance, economics, planning for the future without overspecifying...

There are many, many paragraphs which tempted me to quote at length. My favorite short quote from the whole book is this one:

It's a shame to do anything for just one reason

This is related to "pattern density", a notion which Christopher Alexander apparently discusses with respect to buildings-architecture but which I think is misunderstood in the domain of software design.

- The Universe And Everything - No comments / No trackbacks - § ¶

10 November 04 - 22:24Recording design decisions

Over on Shape, a recent thread has been discussing the use of keeping a journal, in particularly to record one's decisions and how long it takes to get feedback on one's decisions. In relation to my previous post but one, it's also an interesting question to ask about design-related questions.

I don't keep a journal, but I find myself using notebooks a lot nowadays, primarily to take notes of what clients tell me in meetings. And every so often I start keeping a logbook, and I keep doing it for a few months; usually moving to a different job (or out of a job) ends the cycle. I took it up again recently.

It doesn't involve much work, at the granularity I do it. A day's worth of entries could look like this (transcribed from a real entry that I selected almost at random from a "logbook" of mine) :

20/05/2003
* Stand-up (9h30->10h20)
* Document "Smoke Test" procedure (10h30->11h45)
==
* Discuss bugs (#237, etc) with QA Head (13h->14h)
* Help team implement ATs for #268 (14h->16h)
* Help team implement ATs for #275 (16h->17h30)
! #268
! defect in #275 caused by #135
MEMO: should our "estimate accuracy" metric reflect time spent fixing 
defects introduced in tasks whose estimates are being tracked, e.g. 
defect in #275 caused by #135, defect in #271 caused by #248 ?  

Every time I do this, I find myself inventing notations - the "==" transcribes a twin horizontal bar that marks the time I take for lunch, and I can easily spot days when I did actually have lunch. Some days don't have the twin bar.

Logbooks are about activities rather than about decisions. But they could be of value to people interested in looking back at what they actually did - and asking whether what they did actually carried out their decisions.

I wonder how I might go about recording my decisions. It seems to me that I rarely take momentous decisions that would warrant an entry like "Today, after much thought, I decided to...". I can think of about two such in the past two years.

It feels more like work is a constant stream of micro-decisions. Small choices of how to respond to various events, against the background of an overall picture of what I want to do and where I want to go, which I try to clarify as much as I can. The choices translate into activities, so it feels like writing "I did this" is equivalent to "I decided to do this".

Nowadays work includes more training and consulting (and marketing) than coding and designing, although there may be opportunities to get my hands dirty again. If that does happen, I'd be on the lookout for design decisions and recording them - it would be an interesting experiment.

- Software Development - one comment / No trackbacks - § ¶

09 November 04 - 21:07Instant recipe for project success

The first part of the recipe is a well-known quote by Thomas J. Watson Sr, founder of IBM:

If you want to increase your success rate, double your failure rate.

Failure is a great giver of feedback. If all you ever meet is success, you'll never find out what critical mistakes you typically make. So the above is a true recipe for increasing your successes - but that's not the 'instant' part, which is a bit more subtle and not as well known. The 'instant' part is not Watson's but mine (at least in this formulation), and is simply the following - something that any manager, on any project, with close to no effort, should be able to put into practice on the very next business day:

If you want to double your failure rate, all you have to do is halve your project length.

If your next project is a six-month project, and you want to make it two failures instead of one, do two three-month projects instead.

Caveat emptor: this recipe is "instant" but it isn't "guaranteed". It is rumored that shorter projects are more likely to succeed than longer ones. If that happens, I won't give you your money back, but I will let you trade in my instant recipe for the Corollary to Watson's dictum - "If you want to increase your success rate, halve your project length".

- Software Development - No comments / No trackbacks - § ¶

08 November 04 - 17:14Popperian Software Design

No one who reflects thoughtfully on the practice of software development disputes that good design is important to good software. (Not necessarily all-important - there are those who will argue that design matters a lot less than, say, success in the marketplace or in use. We'll ignore that for the time being.) There is less agreement on how to tell what constitutes good design, and there is agreement least of all as to what process results in good design.

Consider design as exercising rationality, or intelligence. Daniel Dennett classifies "degrees of intelligence" according to a hierarchy of models named after famous theoreticians - Darwinian, Skinnerian, Popperian, Gregorian.

I'll just give a quick overview of the hierarchy - go read Dennett's books for more detail. This is best explained in terms of beasties of various levels of sophistication. Darwinian creatures have no intelligence at all - think bacteria. There is variation in how well they survive the rigors of their environment, so that the best are selected - the ones equipped with the "technology" most appropriate for that environment. Skinnerian creatures have behaviour - they can do several things, not just sit there in a Petri dish. And they can learn - behaviour which produces good results is "reinforced". Popperian creatures such as the brainier animals don't blindly try out behaviours - they "think" about them first. They're named after Karl Popper who argued that such thinking "permits our hypotheses to die in our stead". Finally, Gregorian creatures have an extra level of sophistication - they don't just think, they also acquire mind-tools to improve their own thinking; tools such as language. That's, essentially, us humans.

Now that phrase - "permit our hypotheses to die in our stead" - is a severely compressed version of a longer quote, which I was able to track down, not all the way to the original source, but at least to one of two Web pages out there where it appears in entirety:

On the pre-scientific level we hate the very idea that we may be mistaken. So we cling dogmatically to our conjectures, as long as possible. On the scientific level, we systematically search for our mistakes... Thus on the pre-scientific level, we are often ourselves destroyed, eliminated, with our false theories; we perish with our false theories. On the scientific level, we systematically try to eliminate our false theories - we try to let our false theories die in our stead. -- Karl Popper

When we discuss "design" in the context of software development, and in particular in any discussion that ends up contrasting "up-front design", "analysis", "domain modeling" and so on with the class of so-called "emergent" processes favored by agile methods, I suspect that underlies the discussion is a Popperian model of design, and a critique thereof.

On what might be termed a "pre-scientific level" of software development, we dive into code - a phrase commonly seen to express the concerns of people who favor the more classical approach with full-blown analysis, modeling and so on as a prerequisite to implementation. What perishes with our "false theories" - our wrong assumptions as to the nature of the problem, or the structural difficulties faced in implementing a solution - is the project itself. That is, we end up being over schedule, over budget, with crappy code and the customers hating us nerds in IT or in engineering for failing to deliver.

On the "scientific" level, or so the Popperian view would have it, we design so that "our designs die in our code's stead". A design is a hypothesis: "a solution structured thus can be built in the allotted time, will perform correctly, and will solve the customers' problems". For a design to die in the code's stead is for the hypothesis to be falsified - we must show that the design is solving the wrong problem, or that it cannot fulfill its function, or that it is unfeasible to build it with the scheduled effort.

This Popperian view of the design process has several features which can clarify our thinking and our discussions about design. The first is that within its context - the comparison with Darwinian processes at the bottom of Dennett's tower, and its role in explaining intelligent action - the Popperian view stresses the goal of good design: adaptation to a given environment. This can get rid of disagreements about what constitutes good design - a piece of software that "survives" has good, or at least adequate design. Which probably means something rather demanding - the project being seen as a success, the software being put into actual use, without the very first item on the agenda being the development of a replacement.

Further, Popperian design eliminates. It kills hypotheses. So we should be able to verify that a Popperian design process is in operation by finding the remains of discarded theories, hypotheses, or designs. If it seems that a project ended up implementing the very first version of its "design", then something else was going on. As a corollary, design documents should document not just the current design, but the list of design alternatives which were found to be likely to cause the project to "die", if they had been implemented.

That last - if they had been implemented - leads to the crucial property of a Popperian design process - it is able to make predictions as to how a design alternative will turn out if it is implemented. There are many ways to make such predictions, any of which are in principle valid in a Popperian view.

For instance, you might have a "model environment". This would be similar in intent to a wind tunnel for testing the shape of airplane wings - except that you would be "modeling" the relevant properties of the context that the software will operate in. That would be an excellent reason to have an Analysis phase on a project. And a Popperian view would definitely clarify what Analysis is, what we expect of it, and what kind of evidence to look for that the result of Analysis activities was used in this particular way.

But this is not the only way a Popperian elimination could go. You could perfectly well rely of formulaic rules for eliminating design alternatives - say, checking for conformity to design patterns or the Law of Demeter.

However, two things have to be true, I think, for a Popperian view of design to work. First, it must be the case that your predictions are by and large correct. That is, your "analysis model" must correctly rule out invalid design hypotheses, and rule in valid ones. Or your "design rules" must have some correspondence to regularities across various software environments, so that one could verify "if you violate the Law of Demeter, it is in fact more likely that you will end up with a project failure".

Second, and also crucial, it must be the case that the effort involved in eliminating a design alternative is less than the effort involved in actually realizing that design alternative, and seeing - in practice - whether it succeeds or fails in the target environment for the software. (Where "environment" means users, other technical platforms, political considerations, and so on.)

This last hints at how "agile" development can be seen as a critique of Popperian design processes. The claim is that good testing, factoring of the software in flexible and modular pieces, and good teamwork are making it cheaper to try out design hypotheses in actual code. The claim might be true or false, in general or depending on context, but at least our discussions can be based on something that has an empirical basis; we do not need to resort to arguments that up-front design is "voodoo", a meaningless ritual, or that agile development is "non-rational", a discourse that prohibits thinking.

Of course it can also be that design isn't best understood as a Popperian process. But so far, that way of looking at it is the best I've come up with to make sense of the countless discussions of "design" I've seen or been part of.

- Software Development - No comments / No trackbacks - § ¶

07 November 04 - 16:22Bullock's Principle

More or less by accident, I just came across one of Jim Bullock's more trenchant observations on development methodologies, buried fairly deep inside a fairly old Shape thread:

A bad, naive instance of the other guy's way of doing things always looks worse than a refined, thoughtful instance of your way of doing things.

Obviously something I wish were taken into account a lot more in discussions of Extreme Programming, et al - more importantly, something I'd like to think I can always keep in mind when writing about same.

- Software Development - No comments / No trackbacks - § ¶