When the HIS Lights Go Out

Key Points

• Article discusses the contingency measures envisioned in KBC Zagreb in case of total or partial HIS outage

• Important issues highlighted in case all technical risk mitigation measures failed

• Scenarios described: total HIS outage or subsystem failures/disconnections for short and longer period of time

• Procedures for off-HIS operation taken into account: full offline mode, document archive availability, off-HIS medical documentation, information restoration after HIS recovery, communication with subsystems (lab, x-ray, pharmacy, kitchen etc)

• Special case of off-HIS documentation proposed: external PDF archive

Offline Operation Contingency in Case of Total HIS Outage

Any discussion of information security is based on three pillars, mnemo-technically described with the notorious acronym CIA (ISO 2013):

• Confidentiality

• Integrity

• Availability

and while the integrity and confidentiality of patient information is of crucial importance in healthcare information systems, relatively little attention is paid to availability issues.

The ISO/IEC standard 27799 (ISO 2008) defines an electronic health record in the health information system as follows:

“repository of information regarding the health of a subject of care in computer-processable form, stored and transmitted securely, and accessible by multiple authorized users”

The terms “availability” and “accessibility” are used interchangeably and with similar meaning. “Availability” is somehow broader term, again defined in ISO/IEC 27799 (ISO 2004):

“the property of being accessible and usable upon demand by an authorized entity”

The lack of availability is always a nightmare for an organisation, whose operation relies heavily on core business information systems. This applies 100% for the considerations on Hospital Information Systems (HIS) within a hospital organisation.

The common sense evaluation question for any HIS functionality scope is:

“What happens in your hospital when HIS is down?”

If this is answered with “the hospital operation stops for non-emergency patients and procedures”, evaluation shows that this hospital runs an integrated HIS with a broad scope of functionalities, covering most vital hospital processes. The word “stops” signifies that the HIS is a business-critical system for that hospital.

If so, very high standards for system availability are to be implemented in relation to the infrastructure (central hardware and computer network), as well as to some vital application and system software functionalities. Typical examples of HIS outages root causes are:

• Failures in non-redundant HW or network components;

• External and environmental causes (power supply, physical agents);

• Severe database locks or application errors;

• Adverse activities caused by malware.

ISO/IEC 27799 identifies 25 information security threats in the healthcare information systems, with some of them relating exclusively to confidentiality and integrity of information.

Following are to be taken into account as root causes of threats to availability:

1. Introduction of damaging or disruptive software

2. Misuse of system resources

3. Communications infiltration

4. Connection failure

5. Embedding of malicious code

6. Accidental misrouting

7. Technical failure of the host, storage facility or network infrastructure

8. Environmental support failure

9. System or network software failure

10. Application software failure (e.g., of a health information application)

11. Operator error

12. Maintenance error

13. Willful damage by insiders

14. Willful damage by outsiders

15. Terrorism

Despite awareness of these root causes and implementation of corrective measures, residual risk of failure, which cause outages of unacceptable duration, is sometimes unavoidable.

Services to patients in hospitals represent a vital business function as defined in the ITIL R framework (ITIL 2007) and therefore, must be treated as the most important function within hospital institutions. Upgrading availability in order to get a fraction of a percentage better availability in most cases represents a high growth in costs. Thus, the natural question imposed by this zero downtime demand is “how can the desired availability be achieved?” Even mirrored systems can have a downtime.

System operators tend to prevent most of these risks mentioned with redundancy and proper support. Both cost money, which in times of crisis are to be carefully examined and held within acceptable limits. Because of the fact that no risk mitigation program can reduce the risk to zero, hospital management interprets costs associated with improved security as a means of insurance.

In this situation, the guaranteed number of “9” in availability percentage has always to end with the question: “What do we do if and when the event of residual risk happens?” Ceasing the hospital’s IT operation due to an HIS outage is not acceptable, so what is the alternative?

The answer in the University Hospital Centre Zagreb (Croatia) is: “Off-HIS operation!”

Generic architecture of this proposal is described in figure 1. This concept tackles three groups of processes:

Regular HIS operation;

Off-HIS operation during the HIS outage;

Restoration of information system after HIS recovery.

During on-HIS operation the “Ark” is automatically filled in with all HIS documents generated in the PDF form.

During “off-HIS” operation: The archived documents for patients treated are retrieved from the HIS-Ark.

New documents and orders are generated in off-HIS manner (local templates) and stored in the HIS-Ark or locally in case of catastrophic network failure.

After restoring HIS operation, new documents from HIS-Ark are restored to HIS database for regular usage.

Aimed to be able to work with reasonable efficiency and patient safety during the HIS outage, some contingency of medical document management is to be established. This does not imply redundancy in system components (avoiding SPOFs – single points of failure), but rather the entire ongoing operation of a hospital during an unplanned HIS outage longer than a defined period of time, such as 30 minutes or 1 hour. Furthermore, this is not about a manual mode of operation: fetching existing patient information from the paper archive and recording new patient data on paper (by pen or by means of local word processing).

The University Hospital Centre Zagreb’s concept is to generate an external document storage facility named “Ark” (as in Noah’s Ark). This can be a standard Document Management System (DMS) outside of the hospital IT infrastructure and maintain a somewhat lesser version of HIS user roles.

This ark is filled in with all documents generated in HIS for every patient in real time. This procedure is feasible and economic, because in any case all documents are visible as PDF versions, and export to a distant location such as cloud based storage is related to a moderate cost.

Two issues are to be taken into account:

• Patient identification

• Confidentiality

In the often elaborated healthcare information installations that are in place, the HIS cannot be easily replicated where access rights are granted according to department attributions and time elapsed after last visit, as well as the permanent logging of every access to patient documentation is assured.

In an emergency operation through “HIS-Ark” some compromises are to be made, and in that case all users within the hospital will be able to get to all patient documents, with access logging over DMS system, and still be able to treat the patients.

The most important issue in an off-HIS situation remains the identification of the patient in order to allow for the storing of documents in the proper “container”, as well as the retrieval of the documents during a downtime. In Croatia, citizens are given a unique citizen ID, this will be used for primary identification. Search mechanisms through name and DoB (date of birth) are available.

The structure of the document storage facility is organized by patient, with PDF standard navigation panes according to the documents’ originating medical departments. Other relevant EHR information can be stored in PDF metadata, depending on capabilities of source HIS system.

This structure enables the productive part of the off-HIS operation. Thus, when outage occurs, and it is likely to last longer than an acceptable period of time, HIS users can manually switch to the off-HIS operation and perform the following four main functions:

1. Identification of the patient and her/his documents container in the Ark;

2. Retrieval of already stored medical documentation for a selected patient;

3. Entering of medical documentation newly produced during the HIS outage;

4. Communication with other off-HIS users related to the patient’s management.

The first two processes have been outlined in the previous considerations. New documents are entered in locally stored templates that are similar to the corresponding online documents, but less sophisticated and specific to different medical specialties (e.g. automatic pasting of selected lab results in the discharge letter will not be possible).

These are simple text documents requiring more effort from the users, however this is necessary trade-off to keep the off-HIS system as simple as possible. These documents are stored in the patient document container after having been completed locally. They are accessible to other off-HIS users immediately.

Additionally, rudimentary clinical procedures order entry (CPOE) can also be performed. Similar to the new document entry, these orders can also be entered in the Ark and viewed by different service departments within the hospital (radiology, laboratory, pharmacy, kitchen, etc).

This part can be equipped with an order repository, which means that these orders are not stored solely in the patient PDF container, but also in service departments folders. This represents in fact the department’s simple work-list, and the user can deposit their findings in patients’ containers, from which the healthcare providers can view and retrieve the information required.

Once HIS service operation is restored, interim stored documents are to be integrated in the HIS database. This is a semi-automated procedure, which consists of taking text based documents from the Ark and storing them in the standard manner into the HIS documents repository. Criteria include the patient visit category (out-patient or hospital admission), as well as the coding of structured data (HIS documents templates, drugs, diagnoses etc). This task can be performed by hospital clerks or residents. This proposed HIS outage solution provides for a separate part of case-related document storage called “External findings”, where the documents from the HIS-Ark can be automatically stored.

More advanced re-synchronisation methods can include automatic data retrieval from the Ark according to the timestamp at which the system failure occurred, in conjunction with the Ark document generation timestamps. Those documents that were generated during the downtime are sent to the HIS, while all others are ignored. The system can utilise the metadata to automatically return the documents to HIS system with all or the most relevant data intact, bypassing the need of manual data retrieval or additional data inputs.

In case of computer network outage, off-HIS operation can be conducted via smartphones, tablets and/or laptops connected to an independent public mobile network.

Special considerations are to be made to business-critical subsystems, such as lab or x-ray departments, in the following three issues:

Retrieval of previous findings: this has to be solved through regular findings’ communication in on-HIS functioning (subsystem HIS);

Subsystem back-office operation during outage: in modern integrated systems this is a tricky issue that has to be analysed separately for each and every subsystem;

Off-HIS operation: if autonomous functioning can be continued, the communication with HIS is to be taken into consideration (CPOE and direct findings transmission to PDF Ark).

Conclusion

University Hospital Centre Zagreb strives to mitigate the residual risk of HIS outage by means of off-HIS operation. This system has minimal functionalities, but has to provide functionality that is:

• Safe for patients;

• Feasible for HIS users and

• Acceptable in terms of costs.

The basis for off-HIS operation is an external cloud-residing document storage facility, which contains all HIS documents related to the patients. During HIS outage the users can retrieve and enter patient documentation, as well as communicate with each other. Off-HIS is intended solely for the time period of an HIS outage. After HIS operation restoration, these documents are entered manually in the HIS document repository.