Post Office Horizon Project – critical components need checks and balances

The Fujitsu Horizon system for the Post Office turned into the Horizon Scandal as a result of poor coding and a reluctance to fix software issues. (Let’s leave aside the appalling dishonesty of the supplier and Post Office when it came to court cases, resulting in the wrongful prosecution of hundreds of innocent subpostmasters.)

David Mc Donnell, a former Development Manager on the Horizon IT Project, identified the problems with the code itself, he said:

‘it was so bad. It was beyond anything I’ve ever seen. Even in the 25/30 years since that project, I’ve never seen anything like that before. Some of the stuff that we found buried in the code was unbelievable. There was unreachable code… It was a mess

https://www.postofficescandal.uk/post/inquiry-phase-2-star-witness-dave-gives-it-both-barrels/

The EPOS team was delivering a small component within the overall contract but EPOS was clearly a critical component to get right and to prevent wrongly accusing post masters.

McDonnell’s rather obvious solution was to get some better coders in and re-write the cash account from scratch. He was overruled by the Horizon Programme Manager, Terry Austin

McDonnell described it using a Lego analogy:

‘if you understood that it was built out of Lego bricks, you could replace the Lego bricks one at a time starting
with the most critical, the most important, which I would argue was the cash account. Here, you could even — because it was a batch process that wasn’t part of the counter client/customer interaction, you could rewrite that as a separate module and have it running as a shadow process on the counter. You could run the cash account twice at the end of the day or whenever, as a secondary confirmation, and use the replacement module to check the validity of the first one. Once you’d proved that it worked, you could take the old one out and just continue with the new one. This was not a large task. It was not something that – I couldn’t understand why they didn’t do it, because it was such a – it’s not a small piece of work but relatively small, and you could have done it without introducing any danger to anything else on the counter.’

https://www.postofficescandal.uk/post/inquiry-phase-2-star-witness-dave-gives-it-both-barrels/

Because the Horizon Contract was awarded as a single large contract, the visibility of failing components was hidden from scrutiny. The Programme Director was trying to bring the programme in on time and budget, so there were incentives to keep this failure hidden – possibly this is why the Programme Director refused to have EPOS rewritten.

Presumably the vast majority of the funding for Horizon was spent on terminals, networking, servers, training, management consultancy etc. rather than software development. Fujitsu may have been the right company to roll out EPOS terminals to thousands of Post Offices, maybe they were the right company to provide training for thousands of subpostmasters – but they were clearly not the right company (or team) to develop the EPOS component. It would have been far better to award this contract to a specialist with strong development and testing skills. (There may also have been an off the shelf component that could have been used for EPOS.)

I think the core issue for me is that a single component, the EPOS system, had no checks and balances. It required an independent, second component to carry out checks and balances on the first component as David Mc Donnell mentions as a “shadow process”.

I raise these issues and how to reduce risk in large development projects in my article Building Block Architecture – specifically:

• Break down the Delivery Contract into components and procure separately
• For critical components, set up two sub projects in parallel so data and calculations can be checked (ideally using automated testing).