Valid States Only, Please!

I've been thinking about why debugging can be hard and some higher level design heuristics that can help. One thing that really gets me down is invalid states. This problem has a slightly different face in functional programming versus object oriented programming. Sometimes it's not clear cut that a state is invalid, but rather we may have interim states that have no real business meaning.

Lets start with the classic list processing style of functional programming. It's pretty common to take a list of elements conforming to some design, they represent things in the domain in some given state, and the initial states are normally pretty sensible. Then the list is processed via a series of list processing stages, maps and filters, reductions, and so forth. The problem is that these primitives don't distinguish convenient intermediate steps that are just implementation details from stages that are really significant and meaningful states from the domain perspective, valid business states.

The solution is chunking together list processing primitives to make higher order list processing functions take lists of valid items and produce similarly valid results, hiding the intermediate workings, (valid here means representing a state that is meaningful in the business domain).

Failing to break the list at the important points leaves any future developer puzzling over the meaning and correctness of the intermediate lists, particularly if they want to change the process by perhaps adding additional intermediate states or shifting process so that subsequent states are different, or if they are wondering why an object arrives at a particularly stage of processing in a given state.

An analogous situation plagues object oriented programs. A method transforms an object, it does so via a series of intermediate steps. If the method makes those changes on the object directly then the object goes through a series of intermediate states. This can cause all sorts of confusion if you're trying to modify a transformation, it leaves the developer puzzling over what precise intermediate state the object might be in if they what to make change at some given point, effectively the developer has to run the whole program in their head to understand the context of any given sub part of the process. I'll just mention parallel programming, no need to say more I hope.

The solution here is to avoid changing parts of the object, instead, as much as possible calculate the new state in local variables and method parameters, then change the object in a single step. Two phases, one of data crouching and then another of state changing.

Too many months or maybe even years of my working life have been spent trying to understand the state of programs that allow wide access to intermediate states and don't distinguish intermediate states from significant stable states with real meaning in the business domain.

So, some rules of thumb:

• Don't go tweaking state ad hoc, (getters and setters are for tools not programmers), state changes should correspond to meaningful business events.
• Aim to hide, even from other methods, the intermediate states that come about as you implement a method.
• Don't let raw list processing primitives become exposed in your business abstraction layer.
• Chunk list processing primitives together into higher order operations which take you from one business state to another.
• Know what constitutes a valid state, and don't let your objects ever be in invalid or intermediate states.

Getting Stuff Done

I'm planning another blog to track my other interests: songwriting and music in general, and perhaps gigs and other social stuff. Anyway, I've been reading The Frustrated Songwriters Handbook and re-reading Writing Better Lyrics and getting some great ideas and general motivation from both. They are very different books in almost every imaginable way except one.

Both books deal with the problem of slow startup time for creative activity.

The central activity frustrated songwriters are encouraged to try is the 20 song game: set aside a day to write 20 songs and then show them to other songwriters who have done the same. Yes, 20, in one day - then do it again. I'm planning my first effort soon, maybe this weekend, if not then in the week after Christmas.

One of the very first exercise in Pat's book is called object writing set aside 10 minutes each day to write concretely about senses and experiences surrounding some particular object. Stop at 10 minutes. Even if you're bursting with ideas, stop at 10 minutes. After just a few days you start to learn to skip the pretence and centring and all your little rituals, all you can do is as he says dive deep for the good ideas fast.

Both these books are addressing problems of being blocked, of procrastination, of inefficient work practices, and such, that plague songwriters. It's particularly a problem for songwriter because it's normally something you need to fit around the rest of your life. Obviously as an amateur, but even for professional performing musicians distractions like preparing for the next gig, or taking students, and so forth, get in the way of writing music.

These songwriting exercises are not directly about producing songs, they're training techniques that help you hit the ground running when you write.

It seems to me that a very similar problem effects many professional programmers. There's a lot of code that I want to write, more than just what I must write in the office. I want to polish up portfolios pieces, I want to explore new ideas and learn new languages, I want to demonstrate all manner things, and I never seem to have the time, family and friends need my attention, household chores must be done, and so on and so forth.

So, as programmers, are there similar exercises that can make small chunks of time more effective for practicing what we do?

Here's a challenge, the equivalent of object writing, half-hour programming sessions. Make yourself a promise that for some reasonable period of time, (a week, a month, ...), you will work on a project for 30 minutes each day. At the end of 30 minutes, no matter where you're at, stop. If you're in the flow, stop, and let that remind you to get rolling even faster tomorrow. Maintain the commitment to 30 minutes every day for your chosen period, don't work longer today and then give up tomorrow. Outside of that 30 minutes you might need to fix up your tools and environment to make sure you can get the most out of that 30 minutes.

Here's another, the equivalent of the 20 song game. Write 6 programs in one day. They can be anything you like. It would be more interesting if they were different types of programs, maybe even different languages. But the should actual work as deployed programs. For example, write: a game, some data visualisation, a to-do list, a network monitor, a shopping cart, and a web photo light box, all in just one day. Can you wake with no idea what you're going to write, and just make a bunch of programs. Can you do it again next week? What bits of your environment will you need to polish and tune to achieve that 6 program target, what will you need to practice so that you don't waste time next time? Unless you think they are exceptionally cool, they need only be shown to other programmers who've taken the same challenge.

Good luck, here's hoping for a more creative and constructive future.

Red and Blue Architects

I've been thinking about what I expect from a software architect in the highly skilled egalitarian teams doing iterative development that I'm part of, where we're all able to make major changes in response to changing requirements and discoveries in a changing environment.

The traditional software architect's role has been to present a right way of doing something in advanced of stubbing your toes on a problem. That presumes we know enough about our product and it's environment in advanced to make those decisions. It also tends to under-rate the capabilities of other developers to solve problems. Let's call this lot red architects. That was actually true, and good architects in that style were valuable, when I worked in the hardware business: the experts really did have deep knowledge that wasn't available without experience of the product line and general business domain, and the operating constraints of the hardware really were pretty clear in advanced. I haven't worked in that world for many years.

Now I'm working in startup land, it's much faster and looser. The problems of bad architecture still plague us but architecture delivered by red architects just hasn't really worked. We need a different sort of architect, I'll call these new folk blue architects.

Blue architects don't tell me what to do. But they insist that a certain class of question gets asked and answered. Questions that couple the business needs to the hardest technical problems so that they can't be ignored and everyone sees the value in solving them. These are the guys who ask how we're going to handle cache regeneration when we go through database failover, or how we're going to separate our background report from our web front end when the data volume makes the background processes so costly they'll impact the servers responsiveness. They ask what we're going to do when we no longer have a window for downtime because we've become globally successful. Questions that steer us away from the superficial and primitive toward effective and satisfying solutions.

Blue architects guide the whole team to produce better implementations by asking penetrating questions.

Of course everyone with some technical nouse can ask blue architecture questions. But I expect the guy who claims to have architecture specialisation to be better at it, to ask deeper questions, and to frame those questions in ways that lodge in the team consciousness. An architect fails if they can't communicate their insight; I expect anyone claiming specialisation to actually deliver.

In the last few years of work the red architects haven't had much success. On the other hand I've seen individuals deliver blue architecture questions to great effect. Let's have more blue architects.