Software development is all design work from a Lean point of view

A criticism I've heard made about some major voices in the Lean Software Development community is that they emphasis the reduction of waste. Waste is most commonly viewed as rework, or in the throwing away of spoiled goods, although there is also waste in opportunities from not getting quickly to market. A naive view of software engineering would treat refactoring and abandoning prototypes as waste, and consequently avoid practices that lead to those things.

However, everything that I've read so far emphasises some balancing principles in Lean that are part of making it work well in software development. The key is recognising, as traditional Lean does, that design and production are different domains and that different values need to be prioritised. Specifically, rather than a slavish and naive view of waste, there should be significant attention given to making decisions at the last responsible moment and having real options. 

Real options can be seen in the response to problems: If I come to you with a piece of work that isn't acceptable you could just say "no", or "that's wrong", or some other simple negation of the work. But much better from a Lean perspective is to give me more parameters for right, which in software might mean pointing out further requirements, or pointing out tests I need to be able to pass. There is no need at all for the negative part of the response. In fact, even if you follow up with all the good points, the simple fact that you've started with a criticism of my existing work will lead me to defend myself, lead me to try and validate myself by rejecting your ideas. If it actually matters to you that I respond to your advice then don't put me on the defensive, instead empower me with more knowledge and real options.

Software development benefits from making use of the last responsible moment to decide, rather than either locking in decisions early or avoiding making decisions until there are no options left. The later the decision, the less time between deciding and delivering, the more likely that decision is to be based on all the relevant facts and to be usefully out there before new facts turn up.

One of the big insights of Lean, and particularly important for Lean Software Development, is that real option are the things you need to keep available in order to make any decisions.

So I've been thinking about this, how can I maintain real options during code development so that I can defer choices until I have more information and choice closer to deployment?

Lean itself offers some good answers, like make it easy to revise decisions (refactoring) and having side-by-side prototypes. Refactoring shouldn't be seen as fixing mistakes or just tidying up at the end, it's the exploration and refinement of better designs and implementations after the requirement is understood, we do it to improve the product because without those improvements the product isn't acceptable to the users or becomes to expensive to maintain over time. Lots has been said about refactoring so I won't go into it more than that.

When I say prototypes in software I don't mean short lived experiments to be thrown away and subsequently reimplemented with better form, I prefer to call those things spikes and they are valuable learning exercises. What I really mean and want to talk about are side-by-side prototypes as real development options carried on in parallel, prototypes as the output of the design phase which suggests all developer work is with prototypes.

Sounds expensive, developing two or more real versions in parallel. More pragmatically you can try to keep real options about decisions in your domain that are easy to revise, like working up many user interface wireframes while keeping a minimal skeleton UI in place in code, or working up several different object designs on paper as Eric Evans encourages. But at some point you should try this with actual code, I've found it enlightening. 

In order to have parallel implementations in code, without the pain of branches and so forth, you need to be able to switch between both versions. That might be as big as having two versions of a page or dialog, or it might mean having a system of options and switches that allow you use one or the other. 

I'm finding there's an obvious benefit and a very important less obvious benefit of actually trying this. Obviously, I get to look at alternatives side by side. When I make a serious attempt at both alternatives the real requirements start to stand out from implementation details, so in the end the losing version informs the winner. But beyond that the code around the alternatives ends up improved. In order to plug in alternatives, and switch between them quickly, there needs to be a well defined and coherent place for them, a sensible provision of services to them, and a clear notion of what role they play. So, as well as improving the quality of the specific code being developed, side-by-side prototypes become a force for quality at higher levels and larger scales in the software.

Right now to me that would be a win across the board. Clearer design would make it easier to integrate new staff as we grow, real options would give them the chance to prove themselves rather than slogging down what at times seems the only possible route.

JavaScript: Meta-programming Example

Currently in vogue is to have code writing code, this is often called meta-programming. There are many different techniques, and many objectives when applying the technique, and it's often very powerful.

Here is a scrap from a little expression engine I wrote in JavaScript that uses a map of regular expressions to represent the various tokens.

    var lexeme_patterns = {
        "integer": /^\s*(\d+)/,
        "plusminus": /^\s*([\+-])/,
        "multdiv": /^\s*([\*\/])/,
        "open": /^\s*(\()/,
        "close": /^\s*(\))/
    };

There are a number of functions that I want that correspond to each of these tokens. For example I want a function that that I can ask if the next token is of a particularly type: lexer.is_plusminus() or lexer.is_integer().

Rather than manually construct these functions, I generate them using new Function(..) during the construction of my lexer. That way when I add a new token I get the functions I want automatically:

    function extend_for_token_set(proto, tokens) {
        var is_token = "return this.current_token['type'] === '!!!';";
        tokens.forEach(function(token) {
            proto["is_"+token] = new Function(is_token.replace("!!!", token));
        });
    }(lexer, _.keys(lexeme_patterns);

Is it any use? Is it safe? Well, I find it useful to eliminate manual steps as I evolved my expression engine, and if I wanted to change behaviour there was a minimum of code to change, while still getting all the function I wanted for an easy to read style in my parser. But! it is tricky to figure out what code like this is doing if you don't already know what it's supposed to be doing, and that is a big negative.

So, good or bad, I don't want to say, but I do think it's worth knowing how to do it; it's a powerful tool in the modern programmer's tool kit.

XP System Metaphor - Redux

Talking to people, it may be that I'm taking the idea of the metaphor in XP too seriously. That it was an underdeveloped idea about architecture that amounted to little more than: We should be able to easily talk about your over-all system. We've learnt a lot more about the importance of language in design through Domain Driven Design. We should be aware of the history of it all and the lessons we're learning about how to do these things better. We should pay attention to things that are hard to talk about and bring them into our discussions.

That said, my experience is that a big system is much more approachable if you have a solid abstraction of it as a piece of software in terms that can easily discussed. Bonus points if you can keep clear the boundry between the concerns of making an application and the concerns of the particular business domain. And a special secret prize if you can use concrete language to talk about the bits of your system that don't correspond to anything in the outside world. I've had success with a metaphor for the system that is a distinct contrast to the business domain.