Royal Liverpool Children's NHS Trust
Providing full Disaster Recovery capability for critical hospital IT services
Download this case study as a PDFRoyal Liverpool Children's NHS Trust in Alder Hey, Liverpool, is one of the largest and busiest hospitals in the country, treating more than 200,000 children each year. In addition to being the largest children's hospital in Europe, Alder Hey, as it is usually known, is a major teaching facility with over 600 medical students at a time benefiting from its facilities.
Alder Hey is an international centre of excellence in medicine, looking after children from 17 North West health authorities, 2 health authorities in North Wales and Shropshire, and providing pediatric support to the Isle of Man, as well as being a tertiary referral hospital for many conditions affecting the young.
The need for Disaster Recovery (DR)
A modern hospital employing thousands of people is heavily dependent on its IT services for a broad range of critical clinical services and for the administration of day-to-day operations, such as patient admission and management, finance and logistics.
Although continuous availability of IT services is clearly crucial to the running of a hospital, comprehensive DR capabilities for computing systems are, surprisingly, not widespread in the healthcare world. Until recently, if anything catastrophic happened to the systems at Alder Hey, there was simply no timely fallback option.
The Information Management and Technology steering group at Alder Hey did not feel that this was an acceptable situation for the hospital, and strongly recommended that a DR solution be put in place. "We had to find a technical solution that would provide a high degree of general resilience and enable rapid failover of critical services to alternative systems in the event of a major loss of systems," reports Daniel Dryhurst, IT Systems Manager at the hospital.
Finding the solution
The IT management at the hospital had the challenges of determining the optimum DR solution, finding a suitable IT supplier for the technology and implementing it in a cost effective manner. "We felt there were two approaches to be considered - either mirroring each server with an identical backup, or implementing server virtualisation and replicating the virtualised environment," says Daniel.
The first task was to conduct a survey of all the existing services and produce a picture of the hospital's IT requirements to be used in the search for a solution. Two of the hospital's existing IT suppliers, one of which was ISC, were then approached with the requirements and asked to come up with proposals for a resilient infrastructure with DR capability.
After careful evaluation, Daniel and his team chose a design prepared by ISC. The proposal submitted by ISC specified the creation of a virtualised server environment using VMware technology running on six powerful HP ProLiant servers, three in the production environment and three in a separate DR room.
The specification also included a NetApp Storage Area Network (SAN), populated with fault-tolerant RAID storage, which provided a centralised data system for the servers, and a Quantum tape library catered for backup.
The complete configuration would be replicated at a separate location and updated in real time. The intention was to ensure that, while both environments can be used concurrently, either had sufficient capacity to run the entire workload and could, therefore, take over in the event of a complete system outage. "The ISC solution fully met the hospital's DR requirements and was the most competitively priced," states Daniel.
Virtualisation enables the sharing of physical servers by multiple applications, consequently it has the added benefit of reducing the quantity of hardware needed. Also, the new servers and storage would enable Alder Hey to replace as many as 10 aging machines, with the potential to retire more in the future, resulting in crucial cost savings.
"Space was very limited at the hospital, so the efficient use of resources, which is a feature of the virtualised approach, was a major advantage of the ISC solution. And a virtual server configuration is inherently resilient, as services can be quickly switched from a failing machine without significant loss of service," comments Daniel, adding, "ISC's consultants also impressed us with their technical expertise, which was a key factor in the decision to work with them."
Implementing the solution
The installation had to be completed to tight timescales and disruption to hospital operations had to be minimised, so careful design and planning was essential. The existing systems had to be audited to assess the processing and storage capacity needed. This enabled Daniel to produce charts of system resource utilisation, which was invaluable in helping ISC design the new, virtualised environment and plan the application migration strategy.
ISC procured the equipment and carried out the installation efficiently, making sure all systems were configured correctly. Once the new hardware was up and running, migration from existing systems took place between 7 am and 9 am each morning in order to avoid inconvenience to the user community. The hospital's large file server, however, needed 36 hours for the transition, because of the quantity of data involved, so this was carried out over a weekend. Applications were redirected to temporary storage while this took place.
ISC technicians led the installation and migration process with Daniel's team shadowing their activities to learn about the new systems. "The project was executed with minimal impact on hospital operations and was completed on time," says Daniel. "Excellent planning and clear documentation from ISC were key factors in its success."
The benefits
The chief benefit the new IT infrastructure delivers to the hospital is, as intended, the ability to continue functioning effectively in the event of a major incident, such as a fire, in the building housing its computer systems. ISC's approach to meeting this primary requirement does, however, provide additional advantages.
Because of the built-in fault tolerance of the virtualised server environment and the resilient data storage, IT services are proving much more reliable. And if a hardware component should fail, it can be easily replaced without affecting services. Also the powerful new HP servers used and the high-speed data access provided by the NetApp storage system has greatly increased system performance.
Sharing common storage using sophisticated SAN technology, as opposed to using multiple instances of server attached storage, makes for better data security and greatly simplifies the processes of backup and restore. Further benefits include the improved system management and monitoring provided by the centralised administration tools associated with the HP, VMware and NetApp products used.
Although not the main driver for the project, cost savings have been achieved by retiring a number of old servers, as the associated support and maintenance contracts are no longer needed. Freeing up valuable space and reducing power consumption also helped.
"The management and clinical staff at the hospital are greatly reassured by the knowledge that the IT services they depend on have been safeguarded," observes Daniel. "By adopting the virtualised approach put forward by ISC, with its built-in ability to scale, we can also cope with future growth without compromising resilience."
"ISC listened carefully to what we wanted to achieve and put forward a well thought out and competitively priced disaster recovery solution. The technical people were extremely good and they worked closely with us to implement the solution within tight deadlines."
Daniel Dryhurst, IT Systems Manager, Royal Liverpool Children's NHS Trust
