petermr's blog

The Island of Skye has the most dramatic mountains in the UK, and Sgurr Nan Gillean is one of the most visible and well known. For me the Pinnacle Ridge (left) [1] epitomises the reality of computing.
There is an often quoted Gartner’s_Hype_Cycle for technology with a graph like this (not Open). After the

• Technology Trigger comes the
• Peak of Inflated Expectations then
• Trough of Disillusionment,
• Slope of Enlightenment and
• Plateau of Productivity.

Great fun for consultants. I sometimes even believe it. However our software projects usually behave like this.

• The idea. The lash-up demo. Everything works. Everything moves fast…
• The first public wonder demo. People are wowed…
• Everything breaks. Upgrades in the browser kill the demo. Everything takes 10 times longer than it used to. It really does.
• The endless fractal Ridge of Refactoring…
• …and occasionally a stunning view through the clouds

What is refactoring? A metaphor might be maintaining a garden (weeds, pruning, all the seeds of decay). Lots of work goes in but little seems to change. Sometimes (as when parts are ploughed up) it seems to go backwards. Refactoring is making your code better without having anything new to show people. It’s time-consuming, often solitary, and filled with local troughs of disillusion. Frequently you go downhill and when you have climbed the nest peak you are only a little higher than before. But it is a little higher
Here’s a simple example. I want to print some lines to the output (don’t worry about the Java, the idea should be clear)

String[] ss;

...

for (int i = 0; i < l.length; i++)

System.out.println(ss[i]);

This code goes through each string by incrementing the index i and printing it. I have written thousands of little loops like this. It’s not wrong, but it’s not as good as it good as it could be. Why?

• I haven’t delimited the loop (without {} it is easy to get edit loop structures wrong).
• I might edit the code and change the value of i within the loop.
• one of the entries in the array might be null

and several other ways of introducing time-bombs
So I edit it:

List moleculeNameList;
...
for (String moleculeName : moleculeNameList) {
CMLUtil.output(moleculeName);
}

I haven’t changed the functionality of the code – my colleagues won’t notice anything different. I might have had to make many hundreds of edits like this (“blood on the floor at midnight”). There will be many times when I am disilllusions – the code no longer works. Often I feel like a hermit crab who has discarded its safe shell and hasn’t yet found another. I can’t go to sleep until it’s fixed. I don’t know how long that will be. Programmers make mistakes at 0300. There is fog all around (this is very common in Skye and in the Cuiilins the compas points in the wrong directions).
Finally it all compiles. And the Unit tests (more later) run. I get the green bar. Now I can go to sleep (or at least to bed).
So what is better, and why did I refactor? Simply, the code is easier to read and less likely to break. I can output the result to somewhere other than the screen. If someone in the Blue Obelisk or elsewhere wants to re-use it it’s easier and quicker. It’s close to a community component. And that is what we are striving for.
The good news is that Refactoring is very well appreciated in the community. Eclipse has special functions for refactoring. With a little investment in time it can go through all your code and make simple changes (rather like this one) on hundreds of instances without making a mistake. Now that’s productivity!
[1] See http://en.wikipedia.org/wiki/Image:Pinnacle_ridge%26gillean2.jpg for authorship and (Open) copyright of image

In recent comments, JamesM raised the idea of “spike solutions“.
I had never heard of these (nor had Wikipedia) so I asked and found they came from the XP school – here is a reasonable description. Particularly:

Spikes are good when you are knowledge-limited, not time-limited. — KentBeck

or to put it simply – when you haven’t a clear idea where you are going.
This happens (or should happen) a lot in research. Essentially we start out to see if a particular approach goes somewhere useful. You can’t write tests in advance because you don’t know what you are testing. Here are examples from my own work.

• I would like to be able to interpret chemical diagrams as molecular connection tables (i.e. as atoms and bonds rather than pixels). I know it had been done before by commercial companies but the code was unaffordable (and I suspect) of limited applicability. And I had what I thought was a smart idea – to look for common patterns as the images within a paper would probably come from the same software. I had no idea whether I would use neural nets, Fourier transforms or brute force. So I downloaded a number of images which looked of good quality, and lashed up some code to try to isolate glyphs for machine learning. In fact the glyphs turned out to be less isolated that I had thought – the imagaes weren’t monochrome but had a lot of antialiasing to soften them. This meant that some image processing was necessary – so I had to create some simple filters (and I sometimes don’t reuse code because I like implementing algorithms to see how they work). I made it to the start of the next stage but there were no quick wins and not many wins at all. At that stage I stopped the activity and moved to using text parsing instead.
• Frequently when you deal with external data sources – especially unstructured ones such as text – you get a Zipf‘s law distribution of problems. You get a good feel for how the bulk of the data behaves, and code for that. You might at this stage decide you wish to validate your code against Tests. However with every new data item there is a chance that it uses something slightly different which might break the test. So you are actually testing the data against the tests, not the code. At some stage you need to draw a line and declare an arbitrary conformance spec. There are then at least two tests. One for the code against a small “platonic” data set which makes sure that you don’t break core methods during refactoring and one for the data which tests conformance against this platonic spec.
• I wish to use an external tool (database, library, etc.). I don’t know how this works or even whether it works and if so, will it do what I want. So there is quite a lot of glue code that makes connections, creates sample objects etc. We can create test objects but we can’t test the tool until we know how it behaves!

Sometimes these things actually work! So at that stage the discipline has to be to modularise and refactor the experimental code. Obviously it is post facto, but at this stage it should be possible to write the tests – and so there will usually be some catch-up in testing. But do it as soon as possible because it will get worse if you leave it!

We have a new autumn intake of researchers into our Centre and are aware that there are constantly changing demands on the software and informatics skills needed. In Big Science projects there is provision for infrastructure and training and well-developed methodology for the creation of software. We’re working out what is appropriate for “smaller” sciences like chemistry.
“Small” is not inferior – in fact it can have advantages, allowing faster and more diverse activity. But there is less formal support and software usually has to be done in the margins of projects. Software per se has little positive formal reward in science as it is the research in citable papers that matters to the evaluators, regardless of the value to the community.
So how do we develop a good, modern, software environment and lead people to best practices? Today I’ll start with Extreme Programming (XP) which talks a lot of sense (I quote from Wikipedia):

Extreme Programming Explained describes Extreme Programming as being:

• An attempt to reconcile humanity and productivity
• A mechanism for social change
• A path to improvement
• A style of development
• A software development discipline

and

… five values are:

• Communication
• Simplicity
• Feedback
• Courage
• Respect (the latest value)

and summed up in 12 practices, grouped into four areas, derived from the best practices of software engineering:

Fine scale feedback

• Pair Programming
• Planning Game
• Test Driven Development
• Whole Team

Continuous process

• Continuous Integration
• Design Improvement
• Small Releases

Shared understanding

• Coding Standards
• Collective Code Ownership
• Simple Design
• System Metaphor

Programmer welfare

• Sustainable Pace

Now, XP is aimed at teams of developers in commercial organisations creating saleable products against whose success the team can be measured. Whereas small scientists are often working singly on projects with no positive software metric. So can XP (and it has many critics) be relevant?
I think some of it can. The five values are de facto attributes of a successful Open Source project, so by adopting Open Source (especially on a distributed global model) you have to adopt these. You cannot grow a successful group if they do not communicate, write simple code, give feedback (bugs), have extreme courage (more than XP demands), and have respect. So if we can translate the Open Source values into local practice then we have imported these values, regardless of the size of the projects. Of course there has to be some shared goal, but most research departments probably provide some of that (there will, of course, be some individuals working in such new areas that they have no natural companions).
Of the 12 practices some are only applicable to commercial and quai-commercial organisations (perhaps in Big Science). So my list is something like:

• Pair Programming. Where possible someone else should work alongside you some of the time (“mentoring” could be a better word). The second person need not be an expert programmer, but may be good at designing information or act as a rubber duck.
• Test Driven Development. Absolutely essential. Junit tests (more in later posts) have revolutionised my programming – I couldn’t libe without them.
• Whole Team. Not easy, as not everyone belongs to the same team, but valuable if possible.
• Continuous Integration. Yes. Things change so fast that we cannot work with infrequent large releases. Working on Sourceforge we are used to nightly builds and welcome them. Of course the nightly builds have to pass the Junit tests!
• Small Releases. Again with the Sourceforge mentality this is standard practice. It does require careful attention to APIs – too many changes and the re-users get disillusioned. For example I decided to refactor the namespace for CML (there were just too many variants) to a single namespace for all time. One of my valued users told me that he would just about tolerate this, but any more and I was dead meat!
• Coding Standards. Difficult to enforce socially, but happily the tools (at least in Java) implicitly set standards. Tools such as PMD are very useful and we can hopefully standardise on a set of style guides which are not too picky
• Collective Code Ownership. Again any Sourceforger gets used to this. But it can be more difficult within a real-world group.
• Simple Design. Fundamental, but not easy to learn or teach. Like architecture. You know it when you see it! So emulation is a good approach and constant code review by others. The balance between YAGNI and anticipation of requirements is difficult.

I add to this things that seem obvious to the commercial developer but by no means so natural to the Small Scientist

• Use an integrated development environment (IDE).  These are now Open and very impressive. We use Eclipse for Java and are going to recommend it to everyon in the Centre.
• Standardise on libraries. Again that is difficult in some cases, but we can now do this with Blue Obelisk for chemical informatics. After all, some of us have spent enough time developing it!
• Present software projects to the assembled group even if they are on different projects. Be honest – what went wrong is often more valuable than what went right.

So – if you have read this far – we would be very grateful for any feedback from other Small Scientists in similar positions.