Software development is all design work from a Lean point of view

A criticism I've heard made about some major voices in the Lean Software Development community is that they emphasis the reduction of waste. Waste is most commonly viewed as rework, or in the throwing away of spoiled goods, although there is also waste in opportunities from not getting quickly to market. A naive view of software engineering would treat refactoring and abandoning prototypes as waste, and consequently avoid practices that lead to those things.

However, everything that I've read so far emphasises some balancing principles in Lean that are part of making it work well in software development. The key is recognising, as traditional Lean does, that design and production are different domains and that different values need to be prioritised. Specifically, rather than a slavish and naive view of waste, there should be significant attention given to making decisions at the last responsible moment and having real options. 

Real options can be seen in the response to problems: If I come to you with a piece of work that isn't acceptable you could just say "no", or "that's wrong", or some other simple negation of the work. But much better from a Lean perspective is to give me more parameters for right, which in software might mean pointing out further requirements, or pointing out tests I need to be able to pass. There is no need at all for the negative part of the response. In fact, even if you follow up with all the good points, the simple fact that you've started with a criticism of my existing work will lead me to defend myself, lead me to try and validate myself by rejecting your ideas. If it actually matters to you that I respond to your advice then don't put me on the defensive, instead empower me with more knowledge and real options.

Software development benefits from making use of the last responsible moment to decide, rather than either locking in decisions early or avoiding making decisions until there are no options left. The later the decision, the less time between deciding and delivering, the more likely that decision is to be based on all the relevant facts and to be usefully out there before new facts turn up.

One of the big insights of Lean, and particularly important for Lean Software Development, is that real option are the things you need to keep available in order to make any decisions.

So I've been thinking about this, how can I maintain real options during code development so that I can defer choices until I have more information and choice closer to deployment?

Lean itself offers some good answers, like make it easy to revise decisions (refactoring) and having side-by-side prototypes. Refactoring shouldn't be seen as fixing mistakes or just tidying up at the end, it's the exploration and refinement of better designs and implementations after the requirement is understood, we do it to improve the product because without those improvements the product isn't acceptable to the users or becomes to expensive to maintain over time. Lots has been said about refactoring so I won't go into it more than that.

When I say prototypes in software I don't mean short lived experiments to be thrown away and subsequently reimplemented with better form, I prefer to call those things spikes and they are valuable learning exercises. What I really mean and want to talk about are side-by-side prototypes as real development options carried on in parallel, prototypes as the output of the design phase which suggests all developer work is with prototypes.

Sounds expensive, developing two or more real versions in parallel. More pragmatically you can try to keep real options about decisions in your domain that are easy to revise, like working up many user interface wireframes while keeping a minimal skeleton UI in place in code, or working up several different object designs on paper as Eric Evans encourages. But at some point you should try this with actual code, I've found it enlightening. 

In order to have parallel implementations in code, without the pain of branches and so forth, you need to be able to switch between both versions. That might be as big as having two versions of a page or dialog, or it might mean having a system of options and switches that allow you use one or the other. 

I'm finding there's an obvious benefit and a very important less obvious benefit of actually trying this. Obviously, I get to look at alternatives side by side. When I make a serious attempt at both alternatives the real requirements start to stand out from implementation details, so in the end the losing version informs the winner. But beyond that the code around the alternatives ends up improved. In order to plug in alternatives, and switch between them quickly, there needs to be a well defined and coherent place for them, a sensible provision of services to them, and a clear notion of what role they play. So, as well as improving the quality of the specific code being developed, side-by-side prototypes become a force for quality at higher levels and larger scales in the software.

Right now to me that would be a win across the board. Clearer design would make it easier to integrate new staff as we grow, real options would give them the chance to prove themselves rather than slogging down what at times seems the only possible route.

JavaScript: Meta-programming Example

Currently in vogue is to have code writing code, this is often called meta-programming. There are many different techniques, and many objectives when applying the technique, and it's often very powerful.

Here is a scrap from a little expression engine I wrote in JavaScript that uses a map of regular expressions to represent the various tokens.

    var lexeme_patterns = {
        "integer": /^\s*(\d+)/,
        "plusminus": /^\s*([\+-])/,
        "multdiv": /^\s*([\*\/])/,
        "open": /^\s*(\()/,
        "close": /^\s*(\))/
    };

There are a number of functions that I want that correspond to each of these tokens. For example I want a function that that I can ask if the next token is of a particularly type: lexer.is_plusminus() or lexer.is_integer().

Rather than manually construct these functions, I generate them using new Function(..) during the construction of my lexer. That way when I add a new token I get the functions I want automatically:

    function extend_for_token_set(proto, tokens) {
        var is_token = "return this.current_token['type'] === '!!!';";
        tokens.forEach(function(token) {
            proto["is_"+token] = new Function(is_token.replace("!!!", token));
        });
    }(lexer, _.keys(lexeme_patterns);

Is it any use? Is it safe? Well, I find it useful to eliminate manual steps as I evolved my expression engine, and if I wanted to change behaviour there was a minimum of code to change, while still getting all the function I wanted for an easy to read style in my parser. But! it is tricky to figure out what code like this is doing if you don't already know what it's supposed to be doing, and that is a big negative.

So, good or bad, I don't want to say, but I do think it's worth knowing how to do it; it's a powerful tool in the modern programmer's tool kit.

XP System Metaphor - Redux

Talking to people, it may be that I'm taking the idea of the metaphor in XP too seriously. That it was an underdeveloped idea about architecture that amounted to little more than: We should be able to easily talk about your over-all system. We've learnt a lot more about the importance of language in design through Domain Driven Design. We should be aware of the history of it all and the lessons we're learning about how to do these things better. We should pay attention to things that are hard to talk about and bring them into our discussions.

That said, my experience is that a big system is much more approachable if you have a solid abstraction of it as a piece of software in terms that can easily discussed. Bonus points if you can keep clear the boundry between the concerns of making an application and the concerns of the particular business domain. And a special secret prize if you can use concrete language to talk about the bits of your system that don't correspond to anything in the outside world. I've had success with a metaphor for the system that is a distinct contrast to the business domain.

XP System Metaphor

The metaphor seems to be the most puzzling of the 12 XP practices. I'd been resistant to it because it seemed to obscure matters.

I'm quite taken with DDD and for me the heart of DDD is a well protected model that captures domain knowledge using the language of the experts. It seemed that the XP metaphor contradicted DDD's notion of ubiquitous language because the examples I'd been shown tried to replace the language of the domain with the language from the metaphor: the suppliers in our system are cows on a farm... no, not really.

Now I'm beginning to think DDD complimented by a system metaphor may be a brilliant solution. But it will depend on using the metaphor for the right part of the program.

I think the metaphor should be sought out for the parts of the system that are not clearly business domain, and further I think the metaphor should be strong and quite different to the domain. The metaphor represents the system architecture in all the ways that it is independent of the domain so I've started calling it the system metaphor.

With a good system metaphor the boundary between domain and system becomes clear: when the language of the metaphor and the language of the business mix there is an obvious clash of images. With a good system metaphor the architecture is made explicit in code: architecture is no longer an accidental fact of using certain libraries, accessing certain services, and calling one thing after another. With a good system metaphor explicitly representing the architecture and properly tested like any other code the system architecture should made easier to change and evolve independently of the domain.

Ron Jeffries gives an example of an agent-based information retrieval system using a beehive metaphor. The domain of information retrieval would use the language of information retrieval experts and those things which were part of the agent system and not part of the information retrieval would be coded using metaphor language; you know you're dealing with the system and not the expert's domain because you can see beehive words. Of course if the agents and the information retrieval were intimately linked in the domain then I'd avoid the metaphor and stick with domain language, but then I'd be looking for a metaphor to capture the system architecture around them.

Before I ever seriously looked at XP I had some good experiences with a distinct system metaphor. I was building a modular system for network printing, allowing specialist modules be plugged into drivers or moved to the printers themselves. What I built was pipe system, and in discussions and code I emphasised the metaphor of plumbing.

The plumbing metaphor allowed our specialist users to stay focused on their domains (colour processing, optimising streams of graphics primitives, font handling, etc) and to see their system needs differently to their domain problems. Thanks to the clear over arching structure that came from the metaphor, the program was very accessible to new developers. It was easy to explain, we'd draw pictures of pipes in sections that sparked imaginative thinking, (what happens if I change the order of pipe sections, or loop back on myself...), and it facilitated testing, (tapping the flow from a specific driver and pouring it back in elsewhere for generating test data, which later evolved into a network transmission system). We were able to effectively refactor the architecture without much fuss (changing the transfer mechanism between pipeline stages to improve error handling without touching the domain modules because the abstraction had kept them nicely separated).

But how do you find a system metaphor? You go looking for it. You also create the space for it. If the system metaphor is supposed to help you keep the system and domain separated then strive to keep them separate, try to talk about them with different language. If it's to makes sure that the architecture is something appropriately tested and structured then strive for those ends. And look for language which keeps the domain and system separate.

Be conscious of what goes wrong when system and domain get tangled, business logic in your controllers, changes to the user interface that break business functions, things that mostly happen because we don't distinguish between the domain and the system facilities we're using to deliver a program around that domain.

In the end it's a creative step, so get creative, try drawing the system without any boxes and arrows and without any words, think poetically, explain it to someone you respect who isn't a programmer.

And if you've had any success in this area, or if any of this leads you to write better programs please get in touch, I'd love to hear your stories.

JavaScript: Curry Does The Work

curry is a common higher order function which isn't part of JavaScript on most browsers. However Prototype, Underscore, and many other libraries do offer it. The most up to date JavaScript implementations offer the bind function which can be used for the same job. I'm not going to show you how to implement curry nor attempt a generalised lesson, this is about how curry helps me with a refactoring.

In my previous post there was some really clear redundancy in the two wrapper maker functions. Lets make try to make that go away:

 
  function wrapper_maker(before, after) {
    return function(wrapped) {
      before();
      wrapped.apply(null, arguments);
      after(); 
    }
  }

Which is great for the context push wrapper:

  var contextualised = wrapper_maker(context_push, context_pop); 

But how are we going to get those arguments for logging functions in place? The function curry does the work, (I'll use the Prototype version, it's pretty typical).

Basically curry makes a new function that behaves as if the arguments given have been pushed in front of other arguments so the following two bits of code do the same thing:

  logger('abc');

  var loggerX = logger.curry('abc');
  loggerX(); 

This can be powerful when you want to pass a function that is generalised with parameters to another function that expects a function without parameters, which is what we need:

  var logged = wrapper_maker(logger.curry('start'), logger.curry('done')); 

And just like that curry gives a simple solution. It works when we want to add parameters to callbacks but don't control the code that will call our callback.

Refactoring to make our wrappers with wrapper_maker means we can add functionality to all of them without the duplication we would have needed in the previous post. Here's a refactoring that makes wrappers that pass back the wrapped function's return value:

 
  function wrapper_maker(before, after) {
    return function(wrapped) {
      var result;
      before();
      result = wrapped.apply(null, arguments);
      after(); 
      return result;
    }
  }

JavaScript : Functions All The Way Down

Sometimes doing work with functions makes for more lines of code, but it also makes patterns of function calls reusable.

Here's an example that sandwiches our function call between some logging and some context management:

  logger('start');
  context_push();
  do_stuff(data);
  context_pop();
  logger('done');

This kind of code gets dangerous if you have to repeat it. The problem is that we really want precisely that order of the surrounding functions. It's easy to transpose lines, or to update the pattern in some places but not others.

Functions come to our aid allowing us to write something like this:

  logged(contextualised(do_stuff))(data);

That will guarantee the sandwiching, the first call will be matched by the last call, the next inner pair will match, and so forth. But actually writing those functions will add some code.

  function logged(fn) { 
    return function() {
      logger('start'); 
      fn.apply(null, arguments); 
      logger('done');
    }
  }

  function contextualised(fn) {
    return function() {
      context_push(); 
      fn.apply(null, arguments); 
      context_pop();
    }
  }

The real value here is reuse. We can easily wrap any function now. And we don't have to call our function immediately, we can take the wrapped result and pass it as a parameter where ever it might be useful.

We can even make a general purpose function that create tools like logged and contextualised. I'll show an example of that when I talk about currying functions.

JavaScript: eliminate temporary variables with higher-order functions

Functional Programming: it's the new cool! Any number of new languages striving to prove their functional programming credentials, and some old ones getting a boost. For most of it's life JavaScript has been trying to prove itself as an object oriented language, but you can also do some very neat functional programming style tricks with it. I think that the functional programming side of JavaScript is it's real strength.

So what's the big deal with FP? Didn't OO teach us that functions are the source of evil unstructured code? The FP response is that OO makes it too tempting to tinker with state, if you base you programming on storing and manipulating state then you will end up in a bad state sooner or later. Programming in an FP style often ends up with very little state and that state can be changed in only a very few places and narrow windows of time.

The following example uses a temporary variable to hold the current time and then makes an anonymous function to return it, the anonymous function is passed in turn as a parameter to another function. Old school procedural style code:

  var temp = getTime();
  fn1(function() { return temp; });

If you want to program in an FP style you've got to learn to find that temporary variable ugly, it's state, and it's an opportunity for mistakes. Your first instinct might be to inline the call to getTime() like this:

  fn1(function() { return getTime(); });

But that would be wrong, getTime() is accessing changing state, the clock. The original code meant for it to be called just once when the anonymous function was first made. So we're going to isolate that call to changing state with something in a functional style:

  function returner(x) { return function() { return x; } }
  fn1(returner(getTime())); 

Same functionality, no temporary variable. But it's not getting rid of the temporary variable that makes this functional programming, it's that I'm using a function that returns a function. That's the essence of higher-order programming: functions as the data that other functions work upon.

Things to notice: the helper function, returner() doesn't know anything about time, it's really just a tool for locking down some state, turning it into the return value of a function. Specifically we're using the stack to hold the state with the name x, and then we're closing on the value of x with an anonymous function so that we've still got access to it after the stack unwinds. As you do more in a functional programming style you might start building up a library of functions like these, and you'll discover that there are some standard functions that are effectively the patterns of the functional programming world, things like map, reduce, curry and others which I'll talk about in future.

One way to practice this is try writing your program with functions that have only one statement in the body, the return statement, and no var statements at all. All the work will be in the parameters you pass to your functions and what you can do in your return statements.

Take your time, this can be mind bending. You'll find as you apply this technique you'll have less need for variables and wrapper objects to hold state. Less state means less opportunity for errors because you can't be in a bad state. Having both the code organising benefits of OO and the state avoiding mindset of FP is a happy place for a programmer to be, and JavaScript give us that in a very simple package.

JavaScript: varargs

The bad news is there's no simple varargs syntax in JavaScript. The good news is there are some workable patterns that can help if varargs are the right thing for your function.

arguments is an array like object, not an actual Array. It contains all the arguments passed to a function regardless of the named parameters. We're going to take that array like thing and turn it into a real Array, ignore the part that corresponds to the names arguments, then pull that array apart into variables. We'll assign some default values if we didn't get enough arguments.

  function fn1(a) {
    var args = Array.prototype.splice(1, arguments),
        b = (args.length > 0) ? args.shift() : null,
        c = (args.length > 0) ? args.pop() : {};

    fn2.apply(null, [a, b].concat(args).concat([c]));
  }

We use the splice method to get a real Array, and notice how we start at 1 to skip over our 1 named argument.

We use shift to take the left most of the varargs. Notice how we check if there are any arguments available and if not assign a default of null. We use pop to take arguments from the end of the list.

If there were more arguments they will be left in the args variable.

The body of this function calls another function, fn2, with the same arguments used for fn1. I'm using apply which takes an array of arguments so that I can put my variable number of left overs back in the middle.

JavaScript: Selection by Mapping

Very often we find ourselves branching in our code, and the most obvious way to do this is in an imperative style with if statements.

if (x === "case 1") { fn1(a, b); }
else if (x === 3) { fn2(a, b); }
else if (x === someObj) { fn3(a, b); }

That type of code gets tricky to read. There's lots of noise and repetition in the if and else statements, in the argument list on the function calls, and in the (x === ...) in the conditions. All that repetition can lead to errors. It's trickier to see what different about each case. There's also the opportunity to sneak in complexity increasing special cases.

The key to simplifying this is recognising that two things are happening: mapping of a condition to a function, and invocation of the function. We can separate those two things so the mapping of condition to function looks like this:

cond["case 1"] = fn1;
cond[3] = fn2;
cond[someObj] = fn3;

And the invocation looks like this:

cond[x](a, b);

In practice you'll be able to give a good name to cond that can help document what's going on as well.

However, variable arguments can be tricky, but we'll look at that in a later example.

javaScript tips and hints

I'm going to start writing my favourite JavaScript techniques here. I get better at what I do by trying to capture and name these technique. I hope they can help you get better at writing JavaScript, even if it's by reacting against my style.

My main focus will be higher-order and meta-programming techniques. I'm particularly interested in how list processing and functional programming style is such a natural fit for JavaScript. I probably won't spend lots of time on object oriented style although it's good to understand how object orientation can help with sharing and structuring code in serious JavaScript based applications.

Also, little and none of this will be written with reference to the DOM. JavaScript is bigger than the DOM with server side and mobile programming becoming important. Writing well engineered JavaScript is bigger than the browser. But writing browser based JavaScript is important and these techniques are still relevant there.

If this type of stuff is interesting to you then I strongly recommend Stoyan Stefanov's book JavaScript Patterns: Build Better Applications with Coding and Design Patterns.

Fear of State

I've been fooling around with the Node.js native MongoDB driver for a month or so now. I love the concept but the interface lets me down.

When I ask about it everyone wants to show me object oriented wrappers, and every one of those wrappers has timing flaws requiring discipline to call things and access things at the right time. Two made it possible to try to use a database connection object to access a collection before the database connection had been made. Most left data in an object but required you not to try to access it until the asynchronous fetches complete.

Deeply unsatisfying, deeply risky, particularly when the sample code doesn't have those problems. It has a different problem of egregious redundant code...

I think the sample code is on the right track and the object wrapping style is a red herring. Passing functions to functions, and for reuse use higher order functions which take your unique code as an input functions and return a function which wraps them up with the required boiler plate.

Functions all the way down; the functional style appeals to me. The functional model presents the state to the appropriate handler when it's available and not before. But the interface makes it hard to write this style and led me to hacking the driver to make it easier. Why is it hard and what was I doing to make it easier?

Essentially through all these apparently nested but actually asynchronously activated functions there is a need to pass forward certain objects: the database connection, the collection handle, cursors, and similar things.

I wanted to create my unique code first and pass it in, but it needed to pick up state along the way and use that state several levels of asynchronous calls further in. So I made the functions pass on parameters, passing in a bit of context and getting it back a parameter to the callbacks.

I'm not going to maintain the hack so I'll abandon that project and try again some other time.

Don't ask the users of your library to look after objects that have special rules about when and how to use them. Make it easy for them to get what they need at the right time, make it hard for them to get things when it would be the wrong time to access them.

It seems Uncle Bob agrees with me it seems.

JavaScript: The Way Of The Future

JavaScript has been around for ages, but it's coming into it's own with powerful, highly productive tools. The latest generation browsers, ECMAScript5, HTML5/CSS3, PhoneGap and Titanium, WebOS, node.js (on the JVM I'd rather program rhino or ringo than groovy), and an enormous number of excellent supporting libraries.

Thomas Fuchs has blogged that we should help make JavaScript documentation better and I rather agree. He also suggests blogging about JavaScript particularly about topics that make it easy for new JavaScript programmers to pick up skills quickly.

So I will! But what topics should I include?

• Testability and TDD of JavaScript is important to me.
  
• The separation of business logic and DOM access.
  
• Functional programming style, and how it simplifies things.
  
• Learning with server-side JavaScript to improve client-side code.
  
• Explaining some odd stylistic idioms, (such as single var, or bracketed immediate function definition and call).
  

Yesterday I put together a lightning talk of some non-beginner techniques. But they might make good topics as well. Sometimes getting your head around a few tricky problems can make the less complex stuff much easier. Running a few miles each day and getting up the stairs will be much easier.

Not just generalists

We have a development culture that says developer work is ideally directly associated with a user story. And that more or less any developer could work on any user story.

We like to think that if there are technical problems they will be solved by the developers on the user story that is encountering the problem. Technical problems include things alien to the actual user story: tooling and IDE plug-ins, code and project organisation, continuous integration requirements, and deployment complexities.

Implicit in our arrangement is that any developer should be able to solve any of these problems. That isn't the truth for us.

In reality not all programmers are equally able to solve all the problems we face. We hire developers after testing their programming and design skills, and we pay some attention to their interpersonal skills and customer focus. There's a lot of stuff we don't test for, and if we have better than average skills with that other stuff then it's luck not planning. And we have made some poor quality systems that hold us back because of it.

Our CTO says that we are a group of generalising specialists, which to me means we should be first of all specialists who can also serve in general roles. I think that's an aspirational statement: we don't look for specialists and our hiring process wouldn't favour them. Similarly if you go with the flow of our development process you'll become more and more generalised. So far as I can see our baseline is generalist.

And yet we do have developers among our number who are interested and able to do well at many of these things that are problematic for us. Specialists and people interested in specialising.

So my puzzle is how to maintain our user story focus, because I'm proud of that at the end of the day, but also how to encourage developers to usefully pursue their interests and apply their particular skills where they add value. Genuinely, how to be what my CTO would like to say we are, specialists who can work in general situations.

Frankly I don't want to feel vaguely guilty when I say that I don't like some of the stuff I do, and I don't want to feel guilty for doing the stuff that I do like when it adds value. I want to feel allowed to add value rather than just ticking off story cards.

I imagine a situation with developers respected for taking some hours of the week to step away from specifically story centred work and giving attention to technology questions that they particularly care about.

When I say step away, I mean this literally. For my own part I get up from my regular work when I hear people having trouble with the front end and go and get involved. Sometimes I just listen in to understand new things. Sometimes I actively contribute. Sometimes other people call on me to help. Sometimes I identify problems and go fix them myself. It's not all of my time, it's not as much of my time as I'd like, but I enjoy it and it's good for overall productivity.

If that's valuable and it's just me then it's a company vulnerability: we miss the opportunity to get value from efficient use of specialist skills. Also, I'd love people who have big ideas to have a sympathetic forum to share them and also to be informed balancing opinions for big decisions. So I'm thinking about working groups (my CTO wants advisory groups) intended to satisfy those roles, shouldering the load together and encouraging each other to better, without being a drag on our growing development team. Let's see how it works out.

I should say, the simple words "Working Group" are important. "Group" because I don't want specialisation to promote single points of failure. And "Working" because it's about actually doing the work we see needs doing rather than feeling we shouldn't do it because it's out of story.

Super-Star Programmer: The Game

Years ago I was a keen participant in a game design forum called The Forge where they'd made a taxonomy of player priorities: gamist, narrativist, and simulationist (GNS). Different game systems favour different priorities and are more or less satisfying to different players depending on their priorities. [1]

I think formal software development processes are games: you follow the rules to get the experience of being a software developer, a super-star programmer.

The game itself

We all have a notion of super-star programmers and we set up a whole lot of rules and rewards that guide us toward behaving like super-star programmers.

There are lots of things we think super-star programmers do: They make use of computers to prove things about their work. They know about the up coming tasks. They interact with interested parties throughout their work. They make it easy for others to find out what they're doing. They actively learn and share their knowledge. They seek out feedback and create opportunities to act on it. Their code works all the time. They stay on task and their work is relevant to requirements.

We prescribe various practices to get those outcomes in our work: various meetings (planning games, retrospectives, root cause analyses, stand ups); various reporting systems and information radiators; Kanban; process steps that require consulting with specific individuals or teams; TDD and pairing; Continuous Integration; and other practices that we consciously build in to our process. All these together are our game.

It doesn't always work: all these rules don't always produce the playing experience we hope for. And it works differently for different people. A given process may be effective at guiding some developers and be quite disheartening to other perfectly professional developers.

We'd like to be super-star programmers who just do all this stuff without the need for the explicit process. That said, game playing can be one of the best ways to learn hard things.

Lessons from game design

GNS gives us some clues about why processes sometimes go wrong and other times they go right.

The gamist priority: gratified by the rewards and adhering to the rules in order to get those rewards. On the one hand they will stick to the rules, often with lots of energy and enthusiasm for the system itself. On the other hand they are likely to exploit flaws in the system. If you create an environment were some important activities are outside of the rules, then they may ignore those activities. "Gaming the system", sums up the risks of relying on process with gamists.

The simulationist priority: these guys really want following the rules to deliver the goods, and they can adapt to quite intense and restrictive systems. They'll work with flaws in the system if they understand the objective. They'll make the effort to do it right rather than just doing a minimum to satisfy a rule. By contrast they can be quite annoyed if after abiding by the rules the results don't follow as promised. The most diligent players can be the harshest critics of the game and are easily frustrated by imperfect systems.

The narrativist priority: attentive to the arc of activity, the big picture and human concerns. At best they'll take the simplest framework and make it work. But they are notorious for abandoning the rules to make the game suit them. Rules that say what you're trying to achieve are important to them. Conversely insisting on a process without giving attention to the desired outcome will seem like manipulation and frustrate them in very short order.

(I always self identified as a narrativist with some simulationist leanings.)

How we play in our office

Looking around my office I see all of these priorities to varying degrees among my colleagues.

We have a step in our process to talk to QA. The simulationists are likely to actually take time to talk to QA when that step requires it, after all it's what we're supposed to do. But the system gets abused by the gamists who might take a few passing comments as justification to say they've satisfied that step. Meanwhile the narrativists get annoyed because they did talk to the right people but at some other time when it seemed to fit in better.

Thinking about when we've tried to introduce promising new practices. The gamists take to them with gusto, making them work to get new rewards from a new rule. The simulationists trudge through, annoyed that it seems we don't really know what will deliver the desired outcome. The narrativists grow frustrated at yet more things slowing them down.

How about stripping back or simplifying a system when it's going smoothly? How quickly do the gamists stop doing that good thing? The simulationists grumble about changing the rules when they were working just fine. While the narrativists feel liberated by being able to do it differently and perhaps even better out of the opportunity.

And so we suffer seemingly endless rounds of process tuning, never satisfying anyone.

It's not a game!

In the gaming world we could always say that if you don't like the game then don't play. The situation in the office is harder.

For all my love of games, playing this game is secondary. Yes, it helps me when I'm not up to super-star programmer standard, it helps keep us all on the same page. But what matters more is actually striving to be a better programmer, not just playing at the game.


[1] I've taken some liberties in my interpretation of GNS for sake of illustration, I don't intend to argue the fine points of GNS theory here.

First days of node.js and mongodb

I'm putting together a little web application to manage Delphi Method style collaborative priority setting and decision making. I'm using it as a chance to try out node.js, mongodb with a native mode driver, and express.js.

Altogether they make a sweet team. I'm loving the lack of the typical language impedance mismatch in this type of project. And the possibility of sharing code between client and server without adding the bulk of something like GWT or cappuccino.

However I'm not happy with the mongodb driver interface style for node.js. Consider the following variation on it's simplest example, which checks if a record exists and adds it if it doesn't:

var db = new Db('my_db',
                new Server(host, port, {}), 
                {native_parser:true});
var query = {'key': 'value'};
var record = {'key': 'value', 'data', 'stuff'};

db.open(function(err, db) {
  db.collection('test', function(err, collection) {
    collection.find(query, function(err, cursor) {
      cursor.toArray(function(err, items) {
        if (items.length > 0) { 
          db.close();
          sys.puts("query already exists");
        } else {
          collection.insert(record, function(err, docs) {
            db.close();
            sys.puts("record has been added");
          });
        }
      });
    });
  });
});

This mongodb driver follows the node.js idiom for non-blocking IO, providing a function which will be called when the IO is complete. Each of the nested functions above is not being executed immediately. They are deferred until the mongo database interaction is done, giving control back immediately to the caller, playing nice in a cooperative multitasking environment.

But the simplistic code style presented above and in sample code encourages code duplication. And frankly naming anonymous functions is one of the first things to do if you want other folk to be able to read your JavaScript code. Let's pull those anonymous function out and see if it's any clearer.

var database = new Db('my_db', 
                      new Server(host, port, {}), 
                      {native_parser:true});
var db;
var collection;
var query = {'key': 'value'};
var record = {'key': 'value', 'data', 'stuff'};

openMyDb(database);

function openMyDb(database) {
  database.open(openTheTestCollection);
}

function openTheTestCollection(err, database) {
  db = database
  db.collection('test', findRecordsWithKeyAndValue);
}

function findRecordsWithKeyAndValue(err, coll) {
  collection = coll;
  collection.find(query, convertingTheCursorToAnArray);
}

function convertingTheCursorToAnArray(err, cursor) {
  cursor.toArray(insertKeyAndValueIfRequired);
}

function insertKeyAndValueIfRequired(err, items) {
  if (items.length > 0) { 
    db.close();
    sys.puts("query already exists");
  } else {
    collection.insert(record, closeDb);
  }
}

function closeDb(err, docs) {
  db.close();
  sys.puts("record has been added");
}

Notice how there needs to be a db and a collection variable to share those things between the functions. But that means that this is no longer safe in face of parallel operation.

The nasty nested version was necessary because those nested contexts were creating new variables needed for safe parallel operation. I can't reuse the nested functions, but I can't safely use the non-nested functions.

A solution might be something like a push forward context along side the function parameter:

openMyDb(new Db('my_db', 
                new Server(host, port, {}), 
                {native_parser:true}), 
         {'key': 'value'},  
         {'key': 'value', 'data', 'stuff'});

function openMyDb(db, query, record) {
  db.open(openTheTestCollection, 
          {'query':query, 'record': record});
}

function openTheTestCollection(err, db, context) {
  context.db = db;
  db.collection('test', findRecordsWithKeyAndValue, context);
}

function findRecordsWithKeyAndValue(err, collection, context) {
  context.collection = collection;
  collection.find(context.query, convertingTheCursorToAnArray, context);
}

function convertingTheCursorToAnArray(err, cursor, context) {
  cursor.toArray(insertKeyAndValueIfRequired, context);
}

function insertKeyAndValueIfRequired(err, items, context) {
  if (items.length > 0) { 
    context.db.close();
    sys.puts("query already exists");
  } else {
    context.collection.insert(context.record, closeDb, context);
  }
}

function closeDb(err, docs, context) {
  context.db.close();
  sys.puts("record has been added");
}

This would give me a safe way to reuse these functions and make it possible build up some nice reusable chunks for things that are being duplicated at the moment, (like opening a database and a given collection).

I can easily imagine the driver itself copying things into the context, like I've done above with db and collection, and that would slim things down even further.

Perhaps I should fork the driver and try out those changes myself, as I've presented them they wouldn't even break any existing code.

Not Fungible

If developers are not just fungible, and development is not merely a collection of standard developers solving a problem, then what is going on? If you can't just replace a developer then that developer must be bring something unique to the task.

In reality developers come and go on a project. But we make accommodations, we reorganise ourselves to allow that developer to contribute their particular touches, the project changes because we changed developers. I think we under play the extent to which a new developer or the loss of a developer changes our work.

If we're not fungible then a development team is an accommodation that allows a bunch of developers to contribute their distinctive stuff to a unique solution in the effectively infinite space of possible solutions. (Okay, finite computer resources and acceptable execution time mean we're exploring a finite space of solutions, but it's a big space.)

If I was managing the project then I'd be trying to allow these distinctive craftsmen and artisans and artists and puzzle-solvers and engineers and explorers and scientists and philosophers to make something which brings sufficient value to enable us all to make living. The perspective becomes one of trust: I trust that these people will solve my problem.

It takes a lack of trust to prescribe how developers will do their work: if I define their roles to make them fungible then I'm almost definitively preventing their unique contributions and preventing them contributing their best. I suspect out of fear or some misguided fumbling after security we shape projects so that specific developers don't make a difference: but that way we demotivate developers and prevent them making their best contribution.

What happens if I really accept that I'm not fungible? I'm not an interchangeable resource filling a seat. The projects don't simply have quotas of seats that need to be filled.

Then I'm a creative person, and this particular workplace is where I currently exercise my creativity. The project would be different if I were not here. And any project I move on to will be different because of me.

So what do I bring? And what do I want to bring?

Just Playing Games

I used to be very interested in game design, particularly the principles of reward systems. Basically a game mechanism is a system to reward (or penalise) behaviour and a game is about the behaviours it rewards.

Many games are unsatisfying or unsuccessful because they claim to be about one thing but reward something else. Dissonance between intent and practice is often very distracting. Sometimes players end up with a tacit social contract to not play by the rules but rather to improvise within the game's framework to get the experience and rewards they care about. Subverting a flawed game that way often requires being very motivated and having a firm vision for how things should be.

It seems to me that many of our practices and processes are like games. They aim to limit our behaviour or reward us for particular patterns of behaviour. But sometimes they are blunt instruments that can overly restrict behaviour and not reward the behaviour we really care about. A classic misguided reward is flagging that something has been done at the end of stage when what is really desired is to engage in some activity throughout the stage. For example we allow progress when we can flag that we've spoken to QA but that's just a proxy for involving QA throughout development.

And so we explain there are nuances and principles and that you're not doing it right if all you're trying to do is get past the check point. Imagine a game that says don't do what the rules say, instead improvise to do what the rules intend. At the end of the day the system has been set up to make the game about progress through stages, we reward progress through stages by making that progress visible, so indirectly we reward taking a minimum of time to get through the check points.

Similarly different types of computing work can deliver rewards in different ways. I've found build systems to be all or nothing affairs, indeed anything that involves searching for the right configuration tends to have no visible result until it is complete, a big payoff at the end of all the work. By contrast programming tends to deliver incremental progress from ongoing work, no grand climax but rewarding ongoing tinkering. These different patterns of reward for effort fit or conflict with different peoples motivations.

Being aware of what types of rewards drive different people is important in getting good work done rather than boring or frustrating them. Similarly being aware of the rewards implicit in any given task may make it a chore or treat to different people.

We should being aware that our explicit processes are fundamentally games: constraining behaviour and rewarding players. We need to be attentive to what behaviour is rewarded and how a developer might subvert the intent of the process by observing the game rules to the letter.

Perhaps we need to ask ourselves more deeply what is happening when good work gets done and aim to reward that directly, rather than playing games that need subverting by playing to lose but doing the job right.

We shouldn't assume that the games we've been playing are the best games to play.

Who is the process for?

One process to rule them all and in the darkness bind them.

I've been puzzling a lot about process recently. Why are we more interested in Process over People. We talk about processes, we tinker with and tune and polish processes, we treasure our precious processes, gollum, yes we do, my precious.

I've been asking the question of anyone who'll give me a minute, "why do we give more energy to process over people?"

My own answer is the most cynical: I find it hard to speak well of our clients and I'm indifferent toward our products - I give energy to my process to keep me working in a professional way even when I'm not motivated by my clients or product. Another answer is that processes are easier to change than people and programmers are almost by definition control freaks.

Processes are safety nets to ensure that good practices are given appropriate time and effort, knowing that we can't be trusted to always do the right thing.

So it makes sense that I should have a process that compensates for my weakness, but my weaknesses are not your weakness, so why do we imagine one process will suffice? Is that one process the set of all the things that we need to compensate for everyone's weaknesses, (assuming those things are not contradictory)? Or is the process just compensating for our common weaknesses, (perhaps giving us an excuse to not attend to our personal weakness, after all I'm following the process)?

Structured process should be a tool that you shape from a knowledge of your weakness and that serves to compensate for those weaknesses.

I value a team process. But I recognise that not everything in the process is there for my benefit, and some things I really need are not there. As a professional I aim to augment the team process with my own personal practices so that my work is better. As a professional I recognise that playing along with things that are not necessary for me can help others and improve the whole team's quality of work.

But how to strike the balance? How to keep the balance in the face of change?

Doing things the old way

We've been talking recently about why we get stuck in processes that no longer serve our needs even though they were the right thing when we started.

There were a few things that came up: tests written in particular ways, particular sign-offs in our development cycle, the idea that we must have certain types of tests, writing code in particular ways, and so forth.

There was a bit of defensiveness when unnecessary things were highlighted. We didn't set out to be inefficient, and we wanted to explain the context that made these things good ideas. Understanding that context and understanding where we're at now are both important for positive change. Lack of understanding just fuels defensiveness.

An experience of mine is a good metaphor:

Years ago I broke my left knee in a motor accident. I was very careful to do as I was told during my recovery and have since restored symmetrical strength and flexibility. (I even went on to do several years of regular fencing practice.)

Recently I've started Aikido classes and a basic training technique requires falling backward when thrown: lower yourself down with the rear leg and roll smoothly on your back, a very safe and effective way of falling. I can do this easily with my right leg but I always hesitate with left leg -- I hop back awkwardly or I fall and land heavily on my butt. I learned to defend my left leg after the accident, and I'm still unconsciously defending it long after the need has passed.

Now I've got pressure on me to change. I'm getting to the mats early and practising that backward roll slowly and gently so I know what it should feel like. I'm paying attention when it turns out right in class and trying to remember those successes more than the failures. I speak to other students and the instructor so that they give me the chance to practice it right. Eventually the right move will become unconscious. Who knows but maybe losing the habitual defence is the last stage of healing for that old injury?

Some of our quirky processes may be the same. At a time of need we carefully included steps which now are not needed. We've learned our lessons, but we've spent so long valuing them as necessary defences that it's hard to let them go. In fact, without some external perspective or change to bring them to our attention, we don't even notice that we're doing anything odd.

The puzzle is how to drive positive change. If my broken leg metaphor is right we need things like: a problem that highlights the cost of now unnecessary practices, recognition of why they were valuable and re-assessment of our current condition, recognition of our growth and trust that we won't backslide, and alternatives that we can practice that might also allow us to grow even further.

In the end an up to date set of development processes and techniques should make our day to day work more comfortable and effective, that's worth the effort.

Design Facilitation

We've a problem of implementations that simply doesn't match our expectations, not so much the low level detail but an over arching problem of coherence, poor quality design.

We dealt with lower level design by various means: testing and TDD, steady refactoring, and some good libraries (collections, list processing, dependency management, etc).

But at a higher level it can be hard to distinguish the business knowledge from the contingencies of implementation. At times it feels like we were incrementally feeling our way into the product, blindly sneaking forward without any wider view to order our route.

And we're talking a lot now about how to resolve that problem. We've got a number of new hires, and they are very good, so getting them to insist on thorough analysis and their best principles and not merely going along with the flow. But also to be more conscious about design, and align implementation to the way we talk about the product, and how the users talk about it, basic DDD stuff.

We talk about taking time to design, having some sort of map, rather than just hacking with a machete down the path of least resistance.

Also we have a few guys who really want to improve their design skills, who want to make that a key part of their career. I was talking with one about how to get him involved with these things and one thing that occurs is to have someone involved in design discussions who is more responsible for facilitating the discussion than for trying develop the design.

Or maybe that's just a bit over the top.

Kanban, TDD, and Iterative Development

Where I work we make a lot of use of Kanban type systems. It's a key tool in our process and has helped us manage bottlenecks and make activity more visible.

We also favour Test Driven Development which encourages a very fine scale of iteration, and we get benefits from that.

The red green refactor cycle of TDD is pretty good at tuning interfaces and driving looser coupling between modules. Our Kanban process is pretty good at finding bottlenecks in our delivery process and helping us reallocate resources or use our time differently.

Recently we've had some serious production bugs, relatively new code (six to nine months old) being extended with new functionality is causing us grief. Whatever we did in our initial implementation we were missing the secret sauce, we implemented a design that is now hard to fathom, and work in the area is blowing out well beyond initial estimates.

Our designs don't make ongoing use of the code easy.

We were treating Kanban + TDD as our iterative process and we hoped we would get design improvements from our iterative processes, but we weren't really. At the scale of design and iteration of design, neither Kanban nor TDD have given us what we want.

The design bit in TDD is a bit of a lie, it's the hope that if you pay attention to the little stuff then the big stuff will look after itself. Evolution can be a cruel and terribly inefficient process, and there's no guarantee about the future fitness of it's outcome. A program emerges from TDD and satisfies the tests but nothing more is guaranteed.

Moreover, it seems to me that Kanban works against constructive iteration. Of course a larger feature may be broken down as a series of tasks on a Kanban board, and that will be an iteration of sorts, but the motive for task breakdown in Kanban is managing and coordinating resources; creating opportunity to revise earlier decisions is not really Kanban.

Fundamentally, Kanban is monolithic within it's model of a single task. If any iteration is visible within the task cycle you see tokens being pushed back, and anyway you look at that it's negative: it's regression, it's a bug, it's something being rejected. If I take iteration to mean go back and repeat, then from a Kanban perspective iteration within the cycle is a sign of problems.

I feel we're missing something between TDD and Kanban, or parallel to them, that promotes giving attention to higher level code quality issues, quality design and integration, and implementations that align with domain models. Particularly we're missing anything that promotes iteration over higher level design decisions. I'm reluctant to just propose adding more baroque details to TDD or Kanban because any success had with those version is likely to be confused with the basic practice, and when people are weary or don't understand the elaborations they'll slip back to the default practice. I look for methods that support me in my weakest moments, rather than methods that require me to be on top of my game in every moment.

Around the basic development activity (which for us is TDD) I'd like to explicitly promote fine grained analysis and design, review of progress with all sorts of relevant parties, and refactoring to explicitly improve design. Something that encourages multiple iterations for any given task, reinforcing that for any given task a single design step is probably not enough, something that highlights the positive cycle of reconsidering design. Something that revisits the integration of units and their interactions and not focusing just on requirements satisfying interfaces. Something that rewards the developer for going back and having another look.

These concerns are sometimes at a higher level than a naive TDD. I'm concerned with how the collection of objects and services fits together, the coherence of the system as a whole, thinking about future ease of work, clean mapping between implementation and our understanding of the business domain. But at times it's low level but invisible to TDD, it's behind the interface. With TDD alone a thing can be tested and correct and yet have an utterly impenetrable implementation from the perspective of future development.

What might the solution look like? I don't know just yet, perhaps a check list with multiple columns, each column is a fine design iteration, start the next column if you have to put a cross in the box you've already ticked, and then treat getting into more columns as a good thing. I'll be trying out something like that and see where it takes me.

P.S. I know I've presented a very naive view of TDD, but the name itself suggest a simple interpretation. The name TDD gives no clue that you should look for more subtle and nuanced practices even though that's what Beck described in his books.

Test Driven Development and Code Quality

I've been digging into some hairy code the last couple of weeks. It's new code, so there's a lot of testing around it, but it's complex, partly of necessity because it is a hard part of our business, but it seems harder than it needs to be. One of the things I've noticed is that no matter how good or bad it is the tests are better.

Tests are an extra layer sketching out a single scenario over the production code, so they are both more abstract and more specific. Any given test isn't trying to generalise all the scenarios and behaviours of the code being tested, in that sense they are more specific. Also they are at least an extra layer of abstraction over the production code.

It may also be important that tests normally form a topmost layer with no distorting pressure from their caller. By way of contrast consider a web application, the code has a very concrete front end and back end, typically the HTTP interface and a database. Wedged between those things is our poor beleaguered layer of abstracted business logic, all too easily squeezed and distorted by both upstream and downstream pressures.

Production code, the actual program, must encompass all the possible uses, in that sense it must be generalised. I hold that generalisation is next to optimisation as a source of complexity and confusion in code. Any developer looking at a generalised piece of code must assume that it supposed to be able to do all the things that it does: being more specific reduces the things the developer has to take into account when working; needless or accidental generalisation forces the developer to think about things that are irrelevant or unnecessary.

It seems the pressures on production code drive it to be concrete and general, while clarity seems to be served by being abstract and specific, there are two axes here and I think we get them mixed up at times.

And we have a model of abstract and specific code sitting right beside the production code: the tests! But it seems that the quality of those tests isn't feeding back into the production code in a way that makes the production code easier to understand or maintain. The good practice of testing is helping to produce correct code, but not necessarily sustainable code.

But we may have some pieces of the puzzle, some seeds of solutions:

• Wrap general collections so that there is clearly documented set of operations that hopefully communicate the business meaning of interacting the with structure. Isolate and limit misleading generalisation.
  
  
• Perhaps we need principles such that a test should never use domain vocabulary that isn't also in the code. If the test wants to interact with something, then the name should be coming from the code, if the name is not there then the code is missing an abstraction.
  
  
• Similarly, if the test is creating a composite object, then perhaps that represents an abstraction that is missing from the code. And if the test is wrapping an action or a series of steps in a named function then perhaps that represents something missing from the code.
  
  
• Make your implementation match the way you talk about the business: if your intuition or knowledge of the domain tells you something and the code contradicts that expectation, unless the expectation is actually wrong, then the implementation is bad, because being arbitrarily different from expectation requires additional intellectual effort to work with. (That last one is an insight from Domain Driven Design, emphasising ubiquitous language and consistent with models expressed in code.)
  

Driven Development

There is a trend in the agile community to name development philosophies as driven, for example: TDD and BDD and DDD. Names for principles and methods to help us stay on track and to guide us toward good practices.

I thought it might be worthwhile to name some of the other development styles I’ve seen, to help us stay on track and to guide us away from bad practices, (beware irony ahead):

FFDD Favourite Feature Driven Development: when the determining factor in the code you write is some cool feature of the language you use or a preferred programming technique. Everything is a list to unrepentant lispers. Operator overriding is another favourite. They say everything is a look up problem at one well known company. Closely related are Favourite Language Driven Development and Favourite Library Driven Development.

FPDD Favourite Pattern Driven Development: a very popular variety of the FFDD family of development methods—because everything is better with Template Methods.

PDD Personality Driven Development: where your design decisions are based on following someone else's notions, not making your own decisions nor taking into account the current circumstances. If the personality in question is outside the team it’s Guru Driven Development and from inside the team it’s Charisma Driven Development.

PoLRDD Path of Least Resistence Driven Development: polaroid development can have many symptoms: doing things because it’s the way it’s always been done, or wanting to commit changes in someone else's name because you've worked to their priorities. After all it's their product, they're the owner, client, lead, etc.

DADD Decision Avoidance Driven Development: might be seen as a whole company variety of PoLRDD and is often presented as iterative development, affectionately known by some as Hot Potato Driven Development.

LoCDD Lines of Code Driven Development: and it’s more subtle variant proportion of lines of code delivered to production driven development, also known as POLOC DD. The driving principle is that if you don’t need it you can delete later but every keystroke spent on tests and infrastructure is stolen from production.

SDD Seniority Driven Development: the old guys do new stuff and the new guys do old stuff, which is often organised in new development teams and maintenance teams and accompanied by such practices as “I know this system better than anyone and I'm sure this patch is okay so just pop it into production”, and “it worked on my machine”.

CPDD Cool Puzzle Driven Development: overlapping with both FFDD and PDD is the philosophy in work environments dominated by technophiles of doing what you are best motivated to do, of course some dull things have to be done, but that’s how new hires learn the system.

Now that's out of my system

The last two posts were essays that I wrote a little while ago to clarify my thoughts, a bit big for blog posts I guess, but maybe they'll be the right thing for someone sometime.

So, more bloggishly, (to keep the spelling checker happy I guess I could say, "in a more blog-like style", but I like "more bloggishly"):

I've been chatting with a friend who's been disappointed by some experiences in a new job. They have a demanding client and a backlog of bugs, the new starts are being thrown at the maintenance problems, things the experienced folk think should be simple fixes, and it's not going well.

Sadly, simple fixes are normally only simple if handled by the longest serving staff. No one else should feel confident that just patching the problem somewhere is going to be fine without good testing or some formal verification.

And, indeed, those experienced old hands may be right, these may be problems needing only simple patches to resolve. But anyone without their experience of the peculiarities of a system and it's history, and without a safety net of a thorough testing process, can't make those little fixes. Without experience of a system the only professional thing to be done is explore it thoroughly, test as much as possible, and proceed with caution.

I've noticed this before: people with deep experience of a system come to be unconscious of what they know, and assume that things are simple and obvious, (and sometimes get damned obnoxious when it's not so for others).

So if you want to optimise for quality and development throughput the new starts get the problems that can only be solved by thorough exploration, broad testing, and general caution. And the long serving staff get the little maintenance chores.

Heh, in my world they'd still all be collaborating closely and back filling tests over the legacy functionality when those little maintenance tasks come up. But in my world dividing developers into an underclass of maintenance developers and on overclass of new feature developers doesn't happen, that's optimising for hubris, a trap I escaped with my mental and physical health in tatters...

Complexity, keep it to yourself.

"Tell don't ask" is a heuristic aimed at reducing a particular kind of coupling complexity: operations by other objects on the exposed state of a target object. "Tell don't ask" solves this by pushing the operation into the target object and hiding the associated state in there as well.

The more objects that can see and act on the states of other objects the more complex your system becomes: the exposed states of an object are multipliers of complexity. A big part of controlling complexity is limiting the exposed states of objects, or from a different angle, limiting exposure to the states of other objects. The symptom of code that could benefit from "tell don't ask" is often called "feature envy" where one object spends a lot of time looking at another to do its job.

We're all aware of exposed variables as exposed state but anything you get back through a function return value, a call back, even the things infered from exceptions are exposed state. Exposed state is anything about an object that can be used to by another object to make a decision.

The most complete "tell don't ask" style would be a void function which throws no exceptions and has no later consequences giving the caller no possibility of behaving differently in response to the action they've triggered; there's no visible state, no return message, nothing for the caller to act on, and that makes it much easier to reason about the correctness of the caller.

The next step up allows a boolean return value with the caller able to follow two branches. Returning a number allows many branches, exceptions are also returns that cause branching, and so forth. It's easier to think about just two branches rather than many branches. It's much easier to think about no posibility of branching, (but beware of downstream consequences, if a void call now causes observably visibly different behaviour later, then that first call is also exposing state).

If changes in the target object's state are visible then anything with access to that object can change it's behaviour in reponse to the initial operation, multiplying complexity depending on who is exposed to the object's state and how much state is visible to act on.

There are two perspectives here: the caller and the target.

When developing a caller object you want to be able to reason about it, to assure yourself that it is correct, and to know that others looking at it later will feel sure it's right. Being exposed to less state in other objects helps keep your object simpler, so you should be trying to expose yourself to the least number of objects possible and you should want them to have simple interfaces that don't allow needlessly complex possibilities as a response to their operation. If nothing else hide the state of bad objects in facades to show how it could be done.

A developer writing a object that will be called by others should be trying to be a good citizen, should be trying to make it easy for the callers to be simple. Offering wide open interfaces with access to complex state forces the callers to at least think about the possibility of responding to that complexity, and that makes their lives harder: general is not simple.

There are lots of other design heuristics, refactorings, rules of thumb, and so forth that lead to reduced complexity through reduced coupling:

• "Tell don't ask"
  
• "Just one way to do something"
  
• "Don't expose state"
  
• "Reduce the number of messages"
  
• "Be shy" or "Defend your borders" or "Limit an object's collaborators"
  
• "Be specific" or "Do one thing well"
  
• "Wrap collections and simple types"
  
• "Reduce the average number of imports"
  
• "Generality is generally not simple"
  

Sometimes you have to open yourself to coupling, after all programs are really only valuable because they respond to things, but there are ways to reduce risks. Broadly, isolate coupling in time and prevent coupling by side effect:

• "Return the least amount of information possible"
  
• "Expose less state"
  
• "Expose only immutable things"
  
• "Complete construction before starting operation"
  
• "Defensive copy on get and set"
  
• "Fail early"
  

Cows, librarians, and the evolution of god-objects

In an objected-oriented farm there's an objected-oriented cow and object-oriented milk, so, does the object-oriented milk send an un-milk message to the object-oriented cow, or does the object-oriented cow send a de-cow message to the object-oriented milk?

A little vocabulary: in an OO system objects send messages to other objects, a message contains enough information to provoke an action or change.

In a language like Java the built in message formats are function calls, return values, and exceptions. A simple void function is one message from caller to callee. A function call with a return value is two messages, (the return value is a seperate message, albeit part of a well defined dialogue). Thrown exceptions constitute a third class of messages. (Also, exposed state is message broadcasting and to be avoiding if you care about controlling complexity.)

An alternate version of the riddle: in a library do books send messages to shelves saying shelf me, or do shelves send messages to books saying you are now shelved here?

Ah-ha! we say, (some might say "mu"), the riddle is broken, there's an object missing from the system, the librarian coordinates the action, putting books on shelves, sending messages to books telling them they are shelved and other messages to shelves telling them they now contain a book.

And of course the farmer fills a similar role on the farm.

So lets do some more stuff on the farm. And let's keep in mind that useful heuristic, "model the real world", it helps keep us on the straight and narrow in many design problems.

Bringing in the sheep from the top paddock: the farmer opens the gates, the farmer whistles at the dog, etc. Harvesting fruit: the farmer calls for temporary workers, the farmer provides the workers with baskets, the farmer takes the workers to the orchard, the workers fill the baskets, the workers leave the baskets with the farmer, the farmer calls the market, etc. Ordering new seed stock and fertilizer: the farmer makes a list of what's in the shed, the farmer makes a plan for which fields need to be sown, the farmer dials the supply store, etc. Then there's mucking out the stable, tending the ewes at lambing time, etc.

If we follow the "model the real world" heuristic in a domain with a dominant active agent we end up with a lot of complexity in a big class representing that agent, (and we get some weedy little collaborating classes around the edges).

Sometimes we make god-objects because we don't see the world in terms of individually active objects, distribution doesn't come naturally to humans. We see the world as our play-pen filled with toys which are only interesting when we're playing with them, and we model accordingly. Fat-controller-classes correspond to normal human thinking, but they're not good for managing complexity.

Sometimes we're like rental tenants, when the plumbing's broken we call the owner, when there are electrical problems we call the owner, etc, and this owner becomes the god class for household maintenance.

The point is that a heuristic shouldn't be slavishly followed into unmanagable complexity. "Model the real world" can to be watered down, logic can distributed into other objects, even if that arrangement diverges from real life. We have balancing heuristics to help head off runaway god making, things like "don't model the user", and "don't make manager or controller classes" and the "single responsibility principle".

Sometimes a design heuristic like "always consider another two alternatives" is useful to keep us from "the world revolves around me (and my favourite object)" type thinking.

The goal is to manage complexity, to make stable and maintainable code. We've got the reality of change giving us opportunities to find alternatives that fix problems. Good ideas are good but we should be wary of following good ideas for so long that they become bad ideas.

And sometimes we have to do odd things, like maybe writing cows that tell milk to de-cow itself.

"Mu"

My Projects

My current projects include:

Delphi, a tool for iterative refining lists of priorities: a web app fronting a cute algorithm for merging lists of priorities. But the real jewel here is using node.js on the server side, which gives me validation code shared by the server and client, no more duplicating that logic; and, by structuring code as a pipeline and exposing different pipeline entry points on the server I can migrate processing between the client and server.

AudioTools.js: an audio synthesis library, following the model of generating wave forms in the client and dynamically embedding them as Base64 encoded data URIs. In it's heart of hearts this is a hack because the audio support of browser clients is sadly neglected.

The Guitarist's Ear Trainer: one of the key musical skills is being able to identify the sounds you hear, but most of the very good software is about naming the sounds, I want to make the naming of sounds a secondary concern, going directly from sound to identifying where fingers go on the guitar neck is my priority. And I want it to work on mobile phones so I can get some practice in while I'm travelling to work.

And some code I've cobbled together from various sources that I may put out there, (I'm reluctant because I'm relying on cut and paste from jQuery, prototype, and underscore):

patch.js: a library that extends the standard classes of earlier versions of JavaScript with the basic functions you find in the most up to date versions. I depend on good list processing functions, and they are part of the latest versions of the language, but there are dozens of names given to them in various libraries, and a philosophical reluctance to extend the core classes. Generally I don't like polluting the core classes and the global object, but I'm comfortable with extending the core classes the now established standard names.

Looking into the future:

GA-FLRB: a fuzzy logic rule base with a genetic algorithm system. My honours year project involved writing a cellular automata system with an FLRB as the automata, and neat little language to describe the FL rules, still have a soft spot for such systems.

JavaScript

Love it or hate it JavaScript is one of the big languages.

I like the Tomas Fuchs paraphrase of Bruce Sterling: "JavaScript is the duct tape of the internet". With CommonJS and the growth of server side JavaScript, it will be much more.

It's a language that quietly accepts laziness and ignorance, saving up the pain so that it can all rain down on you at some unforeseeable moment in the future. It's a language that rewards discipline: I recommend developing discipline.

JavaScript has become my language of choice, for all it's imperfections, because:

• It's the only language that's available on all the platforms I want to target: major mobile phones and desktops.
• It's remarkably simple, scheme in C-drag, so I focus on getting things done rather than fooling around with the language and tools.
• I like the forward facing side of development, it's a language that's frequently used to tune the user experience and that plays to my motivations.

Of course there are essential tools, work flows that need to be optimised, and developers skills that haven't been widely applied to JavaScript that should be:

• Editors and IDEs; test frameworks facilitating the writing of tests and the execution of tests; debuggers and profilers; libraries and toolkits; command line JavaScript implementations.
• Isolation of dependencies: the chief dependency is with the DOM, the second is with DOM wrapping libraries, the third is with widget libraries, all of which should be wrapped so that you're working with logical elements rather than (other people's) stuff on your page.

Thoughts on a Portfolio

I was screening a candidate today, an international phone call and a review of some of his code on github.com, he made a solid showing. For myself I've never presented my code as a portfolio although I've often said that I think a good professional should. That got me to thinking about what I would like to have in a public repository of my work, how I would like to be judged. What do I want my portfolio to say about me?

• I should want to work for a company that writes code like my portfolio.
• It should be the sort of code I would like to be paid to write.

It's about looking professional, and it's code, so I'd aim to get people actually playing with my work. There's no better advert for a programmer than the quality of their working programs.

A few things come to mind regarding the presentation in the repository:

• It should look real, like you were genuinely using it or it's been made for someone.
• Any reasonably experienced person should be able to click through and figure out what it's about without any wrong steps.
• The tests should be an obvious and effective first place to look, (I'm always impressed when I can figure out the motivation and typical use of open source by scanning through some tests).
• If you're one of many contributors then your contribution needs to be substantial and easy to find.
• It should be relatively easy to deploy and run for anyone who takes a look at it, ideally both a launch script and instructions.
• If it's a web application then consider hosting it somewhere, if it's a stand alone application make it downloadable and installable (and removable).
• Don't present non-functioning work in progress in your portfolio (maybe you need a separate repository for that).
• Don't present old school work unless you are a recent graduate.

And in terms of the code, (some of these points are my responses to code I've read while reviewing various portfolios):

• It should be clever without being obscure.
• It should in general follow the common idioms of the languages, be obvious and relatively future proof.
• The code should speak for itself, comments that just repeat function names or similarly obvious information are redundant; good file, module, class, function and variable names are better.
• Comments should explain what isn't obvious in code, like larger scale motivations or algorithms, or expected patterns of use. Although tests should also carry that information.
• As a portfolio it shouldn't just repeat the same thing, so variety in languages:
  • Something in the Java/C++/C# vein,
  • Something in the Python/Ruby vein,
  • Perhaps something in the emerging languages vein like Scala/Closjure/F#
  • Perhaps a restricted environment tool like JavaFX/Flex/JavaScript.
• At least two different types of applications:
• A retained state stand alone application,
• Plus a web app,
• Perhaps a smart phone trinket,
• A library or some utility scripts could also be worthwhile.

It's a portfolio, put your best foot forward.

By Way Of Introduction

Thanks for dropping by! This is a first foray for me into the world of topical blogging.

I'm starting this more or less midway in working life: my first job was at MacDonalds which I started just before I turned fifteen, and if I retire at sixty-five then I'm just a year or so shy of half way there.

A few years ago I went through a professional mid life crisis of sorts, left a good job in R&D at Canon and went to teach English overseas. I leant Spanish well enough to make good friends and to get back into IT while living in Madrid, I ended up working in a large bank, a job that made Dilbert's world look positively functional, I learnt a lot.

I saved some pennies and headed to London a took a job based on how I wanted to work and what I wanted to learn. I've been here a year and half, and it's still a good job. A lot of this blog will be about the priorities I've developed through those experiences.

It's about the profession of software development, the developers toolkit I use and the way we work. It's also about technology in my day to day, the toys and tools that I want for myself.

Right now I'm not looking at being team lead or any senior role, although I actively take on any load to help the folks who do have those roles, so the perspective will be of a largely content senior developer. It's part of me learning to be a better developer, pulling knowledge from different sources, being conscious of my own character: a tendency to think of developers as humans with human motivations; conscious of the balancing act between ideal solutions and effective compromise; drawing on experiences that include academic research, name brand hardware development, big bank back office, and super-agile small-company web applications.

My own hobby programming will enter the picture. Right now that includes a web based tools, things that can work on mobiles phones as well as the desktop: delphi method tool for capturing requirements, particularly for becoming conscious about "forgotten" legacy requirements, and an ear trainer for guitarists that I want to run on smart phones.

I daily use a very pretty little MacBook at home, and have a thumping great big Linux box under my desk at work. I'm still tossing up between Android and iPhone . I've spent years working with Microsoft Windows. Ultimately I'm technologically agnostic, I've been around long enough to know it'll change, and that's it's good enough to be mostly satisfied, and better not to be too attached to specifics and details.