Damning Report On States IT Failures

Independent report lists lack of routine maintenance, a failure in back up systems, outdated equipment and no disaster recovery plan for staff to follow - and it's cost £250,000 to put right.

PWC has put together a 31-page report that looks at how and why States IT systems failed several times between November 2022 and early 2023.

It found problems with systems and maintenance in both of the IT rooms at Frossard House and Edward Wheadon House. It shows there were failings by both the States own IT team and new digital partner Agilisys.

The report says that from 2016, when it was decided to outsource States' IT, there was minimal investment in existing services.

Timeline of key events

In June 2022, an air conditioning unit in one of the computer rooms failed and wasn't fixed. In November, the second air con failed, leading to the overheating of servers and systems and then quickly an outage affecting the gov.gg website, schools and States internal computer systems.

PWC says it interviewed staff and analysed reports and other paperwork to get to the root of what happened. Its report has a timeline of events and a list of recommendations.

PWC identify a lack of resilience in the networks at Frossard House and Wheadon House, including air conditioning and a generator that was left in place, but didn't work. It also found monitoring and alert systems in the data rooms were ineffective. Maintenance wasn't carried on systems seen as critical to the IT infrastructure.

Once the servers started to fall down, PWC found that there was no manual or rehearsed IT Disaster Recovery Plan, which made the job of trying to fix the issues more difficult.

The list of recommendations runs to several pages and says the States must understand that their old systems will be in place for some time and that they need investment to ensure they continue working. It also suggests that responsibility for systems is clearly assigned to a single party.

The President of Policy and Resources Peter Ferbrache says around £250,000 was spent dealing with the outages:

"It isn’t acceptable that we’ve had to spend a quarter of a million pounds on something that shouldn't have occurred. 

If we had spent £25,000 on something that shouldn’t have occurred that would’ve been too much.

But the fact is that we’ve had to address the situation, and with our difficult financial position going forward, we are still going to have to address this situation, as it is a matter of real priority."

He added that they will keep an eye on the cost of implementing the recommendations:

"We’re not giving anybody an open cheque book, saying just spend whatever it is, but we will devote all the resources we need to devote.

Whether that’s employing extra personnel, extra suppliers, or buying equipment advancing systems, to make sure that we give the most resilient system that we can."

He hopes to avoid outages in the future:

"Lack of maintenance is something that’s inexcusable and should never have happened.

Nobody is seeking to defend it, but we’ve already made significant steps in addressing it, and we’re going to continue to do that."

A redacted section of the report on States IT failures