Written by: Elaine Naughton

Contributors: Ursula O’Sullivan, Richard Keating, Darren Finn, Gordon Graham & Kevin Walsh

Introduction

 

Microsoft terminated their support of Windows XP operating system and Office 2003 products on April 8^th 2014. Many Irish public sector organizations were affected by this announcement. Many organizations including the HSE had computers running Microsoft XP and Office 2003 which after April 2014 could be exposed to increased risk of viruses as well as security problems and indeed putative costs. An agreement was negotiated by the Department of Public Expenditure & Reform (DPER) for extended support with Microsoft until April 2015. The HSE were one of the organizations included in the extended support arrangements.

The HSE were also aware that Windows Server 2003 and Microsoft Exchange 2003 would also terminate support in 2015, a large project was undertaken. This body of work (known as the Microsoft Product Upgrade Programme – MPUP) was undertaken in conjunction with third party contractors. The work included the upgrade of servers running these products as well as the Windows XP desktop upgrades.

Over the course of 14 months a significant amount of work has been undertaken to upgrade each product groups. This is undoubtedly, one of the largest IT infrastructure projects ever undertaken in the State. The purpose of this document is to summarise the current status while also outlining the next steps required to achieve a single national operations infrastructure for the HSE.

The Challenge:

With the knowledge that Microsoft intended to terminate their support arrangements for the standard Operating systems in place within the HSE, the ICT Department was faced with the mammoth task of having to upgrade all machines to a stable and supported versions of Windows.

A review of circa 46,000 PC’s and 2500 servers nationwide highlighted the following:

• 26,000 PC’s within the HSE required an upgraded from Windows XP, or earlier versions of Windows, the desired agreed upgrade required was Windows 7
• 6 Microsoft Exchange Infrastructures were running legacy Microsoft Exchange Server 2003,
• the Directory Services in each of the 8 former board areas were running Windows 2003 server
• the majority of the other 2500 servers in existence were running Windows Server 2003 or earlier.

The Solution:

The MPUP Team was established and leads identified with responsibility for each of the identified workstreams below:

• Desktop
• Microsoft Exchange 2003
• Windows Server 2003
• Active Directory
• Application Remediation

Desktop:

To ensure that all HSE PCs were upgraded prior to the expiration of the extended support, a ‘Rapid Remediation Deployment’ model was put in place by HSE ICT. The Programme was run on a National basis under the management of the MPUP project team, and each Regional Infrastructure & Operations Manager, together with the service users, scheduled detailed implementation plans for each area and hospital. This enabled agreement of timeframes and schedules for PC upgrades with users groups, in their area of responsibility.

This model set clear targets for the number of PCs to be upgraded each day while putting the necessary ICT Infrastructure, Software, Resources, Tools and Procedures in place to achieve it.  Regional deployment teams were assembled to carry out the upgrade programme supported by a third party contractor.

The status as of June 2015 was as follows:

The successful completion of key phases of the MPUP is significant for the service and infrastructure teams in particular who have worked tirelessly to achieve ambitious goals set out, within very tight timelines.

 

Application Remediation:

There are a significant number of applications in use across the organization as one would expect of an organization as diverse as the HSE. The task of gathering information on all applications proved a challenging one. A third party contractor was employed to scan all desktops to identify the list of applications. This scan produced a list of 1000+ applications of varying priorities and with varying support agreements (from none to full maintenance and SLA cover) in place.

Applications were categorized based on support arrangements as follows:

• Category 1 – Applications supported In-house by the Applications division
• Category 2 – Applications for which support arrangements are in place with a 3^rd party provider
• Category 3 – Applications with out of date, unknown or no support arrangements

 

It was agreed that within the timeframe remaining that only Category 1 applications would be in scope.

Over 90% of these applications were reviewed in the timeframe. To address outstanding non-compliance issues, over 50 remediation projects were initiated, chiefly targeted at Windows Server and Office compatibility. These remediation projects will continue to be worked upon during 2016

 

Windows Server 2003:

Regional Server Management staff identified, upgraded and migrated Servers where possible into the National domain, HEALTHIRL. Regional (legacy) Citrix Farms were upgraded and migrated into the National Citrix Farm and standardized. National Virtual infrastructure was mobilized where possible, while also introducing a standard template for server builds across the organization. In excess of 1,000 servers have been upgraded to Windows Server 2008 with 1,323 Windows 2003 Servers remaining because of application remediation deferrals. The largest application to have been migrated is iPMS where nearly 500 servers have been upgraded, including the Citrix environment, supporting 3,500 concurrent users.

Directory Services:

There are eight regional domains in the HSE, of which each contain an average of 10-15 Domain Controllers, which all ran Windows Server 2003. The project work stream involved each local Domain Administrator working over a period of two weeks with a third party contractor, PFH Consultants as well as the NICTDS team lead to build new servers. Once built, the new servers had to be promoted to Domain Controllers and legacy servers demoted. This also involved replacing legacy hardware with national virtual infrastructure where possible, and migrating DHCP, DNS and other key Directory Services to the new servers while ensuring minimal disruption to end users. The current status is that domain controllers bar a maximum of one in each area, have been replaced. The remaining legacy server exists to cover deferred applications which have authentication requirements that pre-date Windows 2008.

Exchange 2003 – Exchange 2010:

A review of the Microsoft Exchange environments in the regions commenced in 2013. Due to the large volume of mail in the regions it was decided by the project team to upgrade the 6 existing Exchange 2003 regional email environments to Windows Server 2008 and Exchange 2010. New Microsoft Exchange environments have been built and are in place in all but one domain. To date over 13,000 mailboxes have been migrated in the South; 12,000 mailboxes in the East; 5,700 in the South East; 4,700 in the Midlands; 3,200 in the West. The remaining regional upgrades and migrations continued throughout 2015/16

Next Steps:

As work continues with servers and application upgrades and migrations, the next phase of the project is now in train. In this phase the HSE will realise a single national infrastructure. This will enable leverage of the national ICT resource pool as one team, which can in turn support one infrastructure.

Partnering with Microsoft a new programme of work was agreed, known as the Enterprise Consolidation and Application Modernisation programme (ECAM). The scope of the new ECAM programme is as follows:

 

• Continue the build out of the national domain, in line with Microsoft Best Practises, which will be capable of supporting all users, computers and servers within the HSE
• Prepare the HSE to migrate all regional domain users and computers into this national domain
• Build a central E-mail/Messaging infrastructure, based on Microsoft Exchange, allowing among other things, shared calendaring, efficient mailflow and proper integration with key infrastructure applications like MS Lync etc
• Produce a rationalised catalogue of all applications, and assess and migrate a prioritised subset of those applications to Windows Server 2008, with the remaining applications subject to a review of how to proceed, taking into account the success experienced to date.

 

Conclusion

The Microsoft Product Upgrade Programme (MPUP) has been a priority programme for the HSE. It has delivered significant benefits for the HSE organisation in terms of updating and upgrading the devices and technologies used to deliver our services to the public.

It could not have been achieved without the full co-operation and patience of all - service users, teams and mangers. This type of work sometimes seems ‘invisible’, but for the duration of the upgrade it caused significant impact on the availability of ICT resources for users and indeed had an impact on other ICT projects and works.

The new programme of work will still require significant input by local SME’s and Infrastructure & Operations resources in particular. However, the end result will allow for the streamlining, standardization and greater regulation of ICT services across the organization which is also in line with the new Operating Model for the Office of the CIO.

 

Contacts

 

ECAM Programme Manager/MPUP Business Manager:

Ursula O’Sullivan    @tweet_ursula      Ursula.osullivan@hse.ie

 

ECAM Enterprise Consolidation Project Manager:

John Lehane           @johnwomble       john.lehane@hse.ie

 

MPUP Programme Manager:

Jack Somers           @survivorsomers  jack.somers@hse.ie

 

ECAM Applications Modernisation Project Manager:

Ray Daly  ray.daly@hse.ie