2 months on from WannaCry Ransomware Attack

July 7^th will mark 2 months since the Ransomware #wannacry malware attacked systems around the world affecting over 230,000 computers globally. While the HSE were able to avoid serious impact there were still many lessons that could be learnt.

We talked to Fran Thomson, Director responsible for Engagement and Delivery, and Darach Glennon, Director responsible for Customer Service Experience.  We reviewed the week and the outcomes that have been learned.

The wannacry malware worked on infected computers by encrypting files, a ransom was then demanded to release the files back to the owner. This is not the first attack of this kind nor will it be the last. Hacking has become a significant criminal and political activity, as a consequence attacks are on the rise and gaining increased visibility. Spam and malware is a huge percentage of email flow across the internet. Protection is improving as attacks occur but equally the attacks are getting more sophisticated as protection improves.  In light of this, it is not surprising that wannacry happened or that it had such an effect.

On the afternoon of Friday the 12^th May while the Digital Health Governance Group were meeting reports began coming in from the media and contacts within the NHS of the attack on global IT infrastructure by the ransomware wannacry virus. The HSE took the decision to immediately increase monitoring of all systems by the customer experience team and the Office of the Chief Information Officer (OoCIO) were notified to be on alert. Further media reports emerged showing the extent of the virus in multiple countries and the escalation of major problems worldwide. It became clear that healthcare was a global target. A “MI – Major Incident” was declared as a precaution. At 6pm HSE leadership team requested the OoCIO to shut down external access to email immediately.

The ultimate goal was to protect patient care and every decision was centred around that. The priorities became: secure the internal network and access, consider external vulnerabilities and manage the wider health sector information systems. The strategy throughout the process was to ensure that communications were clear and frequent to all affected by the event. This was to ensure that everyone was aware of processes and updates as the incident unfolded.  

That evening and over the weekend teams were mobilised across the country. Different teams were assigned different strands of the emergency work and communicated through conference calls that were initially happening every two to four hours. There were multiple actions taken in response to this multi-faceted problem. These can be seen in the figure below.

The aim over the weekend was to ensure that the information systems could be turned on where ever possible on Monday morning and minimise any disruption to patients and users. It was imperative that this message was communicated to the wider health service, to make sure they were aware of the attack, were confident that it was being dealt with and knew what steps they should take when they returned to work on Monday. A communications campaign #THINKB4UCLICK was initiated to get these message across, this took the format of sending information via posters, videos and animations on social media, which were shared with NHS to use and present to getting the message across through traditional channels such as radio and television.

By the end of Monday 15^th anti-virus software updates, that addressed the vulnerability exploited by this virus, had been deployed to approximately 52,000 health system PCs and 2,350 servers. In addition newly released Microsoft patches were rolled out to over 28,000 machines. Over half of the 1500 vulnerable Microsoft XP machines had been visited and 1,300 health servers rebooted. The remainder would be visited over the next 48 hours. The teams visited as many sites as possible switching on machines, to allow for patches and updates to be downloaded and hanging information posters before staff returned to work on the Monday. 

On Wednesday 17^th May a plan was actioned to reboot nearly every server in the HSE estate. This was required to ensure the anti-virus changes took effect. After this it was determined that it was safe to turn back on access to internal email followed by external mail.

By Thursday 18^th May all remote users were back on line and after assessment the major incident was stood down.

There was only one incidence of wannacry identified in a HSE funded facility in Wexford. This facility was not linked to the wider health service network but as a precaution it was removed from the incoming email list and the affected hardware was removed.  

Since wannacry the national helpdesks returned to business as usual and continued to deliver support. At the same time work began on the longer term repairs and reviews that were deemed necessary from the incident.

For example, the 1500 machines running XP are still in the system because they are running very specific programmes linked to pieces of medical equipment (devices that deliver diagnostic imaging (NIMIS) and Bio-Medical Device control machines) Their vulnerability is now being assessed and protections are being reviewed.

The entire HSE estate of technology will be reviewed, there were certain systems that were radically overhauled during the MI and steps are being taken to prevent that being necessary again.

There was and is a range of defensive and protective tools in continuous use: anti-spam, encryption, anti-virus, firewalls and other products but this alone was not and will not be sufficient. Since the attack, we have been reviewing all relevant services: How we do business, how we communicate, where the weaknesses in the estate are, how we can patch quicker and again, our security policies and practices and detection protocols.

Partners worked hand in hand with us throughout the attack to ensure that their technology was protected, to assist where possible and are also reviewing their systems for the future.

This was the first major test of the critical incident management service, introduced in 2016.  This team were at the centre of the response to the attack and collaborated with the wider HSE to protect the information systems.  There is no doubt that we need to continue to be vigilant for these types of attacks. This MI was a major learning curve and the lessons learned have been taken on board. Moving forward there is an increase in planning to prepare for these types of incidents both in the medium term and long term.

It was a team effort from the entire HSE that responded to this attack, people went above the call of duty to help from the Director Generals’s office, IT support in hospitals right down to every single person who followed all the instructions issued, it was team effort that kept the system safe and ensured that we were able to bring it back safely and quickly.

Since the attack, we are encouraging all staff to avail of our Good Information Practices Module. You can find out more about it here, and click through to do the module. It only takes 20 minutes and will help to continue to keep our systems safe.