Back to Newsroom

Tuesday, 12 October 2021

No electricity, no IT, no phone calls, no surgeries: IT failures can affect hospitals severely and, at worst, may endanger human lives. Nico Meinhardt, IT manager of the St.-Antonius hospital in Eschweiler, can tell us a thing or two about it. During the flood disaster in mid-July in the West of Germany, the lights went out in his data center. By chance, an msg employee, Dennis Zielke, came into the picture – which made it possible to establish an emergency data center in no time. In this viewpoint interview, the two participants describe their experiences.

We all have mental pictures of the destruction and suffering caused by the flood disaster. How did you personally experience the events?

Nico Meinhardt: The flood hit us full force. We didn't get as badly hit as the Ahr valley or the "Erftkreis" county, but it was still pretty powerful. I remember it very clearly. Our hospital is in the city center and close to the Inde, a small river with a depth of perhaps 70 centimeters.  From the afternoon of July 13, the level rose successively. At a first, small crisis meeting we couldn’t have foreseen that we would be facing something like this.

Have there been comparable flood events in the past?

No, never like this. Nevertheless, we had taken advice from experts in the past and prepared ourselves for this. For example, with high sheet pile walls. A flood scenario was drawn up in which various protection measures were taken.

Were you on site the whole time?

In the evening, I watched the water level rise and rise on my laptop from home. Around midnight, the level had risen to 3.70 meters. From this level onwards, there were no more measurements because the gage had obviously become a victim of the flood. The further rise could no longer be recorded. I finally asked our technical manager Michael Deussen on site how serious the situation was. Our data center is located on the second basement level, in a bomb shelter with two double bomb shelter doors. In the past, we assumed this place to be safe.

This changed in the course of that night?

At first, Michael still said that they had the situation under control. Then at about half past one in the morning, I learned that the hospital had already turned into a small island. In the meantime, the water reached the top edge of the sheet pile walls. It was about half past two when our managing director called and literally said: “Mr Meinhardt, shut down the systems, we have to abandon the hospital.” A short time later, we were swamped. The two lowest levels were flooded, the courtyard was flooded, our hospital park was flooded. A torrential river was running through the hospital. Unbelievable! You only see this in reports on TV. The masses of water simply pushed firmly anchored metal fire doors around like little toy doors, and they came to rest somewhere. It was terrifying.

What damage has the flood left?

Heating, transformers, power distributors, emergency generators, data center, telephone system, cooling units and gas units were located in the cellar or basement. As well as the laboratory, part of the radiology department and two specialist departments. Water pipes were burst, sewage pipes were burst. There was practically nothing left, everything was ruined. The damage is roughly estimated at over 70 million euros.

What happened to the patients?

There were about 300 to 350 patients still in the hospital, including about 25 intensive care patients who were highly critical and bedridden. You can't just discharge them and usher them out. There was no way in and no way out. That's why the most important thing for us at first was to take care of the patients. The hospital had to be evacuated, things were that dramatic. The only access to the hospital was actually via a kind of shuttle service, organized by the Eschweiler fire brigade. We then drove through the city with an Unimog. On the way, you could see the full extent of the disaster: smashed windows, flooded shops, cars that were bobbing back and forth in the water like toys.

Was anyone in the hospital harmed?

No, we managed to safely evacuate all patients unharmed.

And what was the situation with the IT infrastructure?

I knew nothing about  the condition of the IT infrastructure at that time. But there was hope that our bomb shelter and the IT infrastructure might have survived the whole thing largely unscathed after all. Regardless of this, we immediately began contacting our partners and planning a possible rebuild of the IT infrastructure with them. Because one thing was pretty clear: A complete rebuild would mean a lot of work.

And this is where msg came into the picture?

Exactly. Actually, it was a coincidence: Dennis Zielke and I know each other from our student days. We were in touch about the “Hospital Future Act” (Krankenhauszukunftsgesetz (KHGZ) and wanted to discuss msg's portfolio at a meeting. In a phone call, I described our situation to him. Because of the situation, the meeting didn't take place but Dennis immediately offered his help and that of msg. That's how the whole thing got rolling.

What did the descriptions trigger in you Mr Zielke?

Dennis Zielke: I couldn't get it out of my head over the weekend. First thing Monday morning, I approached the head of our unit and asked him if there wasn't anything that msg could do. Then, I got in touch with colleagues who specialize in emergency operations for cyber security and the like and who have experience with such situations. Because there too, it's about time, speed. And about getting the business back up and running when the infrastructure has been destroyed due to a hacker attack. We briefly discussed what we could do. We already knew how may servers and what kind of hardware was needed – and we provided and delivered this to them in a care package as quickly as possible.

Mr Meinhardt, did you expect msg to react like this?

Nico Meinhardt: I was a somewhat positively surprised myself. The speed and quality of the trusting cooperation serves as a prime example. I don't think you can do this any faster. When I saw the damage done, I thought: It will be months before bit sand bytes will rush through the network again. Everything was ruined: Main and sub-distributors, server farm, storage and the network.

Was there a contingency system and/or backup in place?

We have a second location in another wing of the building. Besides the backup of the hospital information system, we also store other critical data such as digital patient records or PACS data (patient images) there. However, it was most unfortunate that we had just switched and renewed our backup system. The cutover had not yet been completed. The flood got ahead of us and we got caught off guard. We were lucky to have tapes, i.e. backups, that we had stored in a safe in the IT rooms. Like the way it used to be. That saved us. The task now was to restore the data and rebuild the infrastructure as quickly as possible.

How did msg help you with this?

What we needed was server hardware, storage, SAN switches and a tape library to read the backups. And best of all, someone to help us set up the server farm. We're almost completely virtualized with VMware, ESX hosts needed to be rebuilt, and the data center needed to be rebuilt - this is where we desperately needed help and someone familiar with Veritas' outdated backup software Exec.

What came next?

In our first call, we discussed our needs and the sizing of our environment. msg provided the care package I mentioned, which arrived shortly afterwards. That was just over a week after out first call. That is also one of the reasons why we are already so far ahead today. Almost 100% of the systems have been restored. The main building is completely networked again. We can work digitally again, have been able to function again for three weeks and are providing outpatient care for our patients. We will start providing inpatient care again from October 4. It is impressive to see what has been achieved in such a short time.

How did you feel about this unusual project, Mr Zielke?

I wasn't actually on-site. I was there only virtually and on the phone. But that didn't make me feel any less sympathetic and engaged. It was an emotional experience. And also because I know that msg is competent. There's real substance to what we do here.

So did a partnership between St.-Antonius-Hospital and msg emerge from the disaster?

Nico: Yes, because we know how to appreciate this willingness to help. An excellent start for a promising partnership.

Thumbs up - the urgently needed servers have arrived.
Nico Meinhardt, IT Manager at St.Antonius Hospital