Comparison of a commercially available clinical information system with other methods of measuring critical care outcomes data

doi:10.1016/j.jcrc.2004.02.003

Journal of Critical Care

Volume 19, Issue 1, March 2004, Pages 10-15

https://doi.org/10.1016/j.jcrc.2004.02.003 Get rights and content

Abstract

$Purpose:$ To compare the quality of data recorded by a commercially available clinical information system (CIS) to other commonly used methods for obtaining large amounts of patient data.

$Materials and methods:$ Five sets of clinical patient data were chosen as a cross-section of all the data collected by a CIS in our intensive care unit (ICU): 1) Length of stay in the ICU, 2) Vital signs, 3) Days of mechanical ventilation, 4) medications, and 5) diagnoses. Data generated by our ICU CIS was compared with other parallel data sets commonly used to obtain the same data for clinical research.

$Results:$ When compared with our CIS, the hospital database recorded a length of stay at least 1 day longer than the actual length of stay 53% of the time. A search of 139,387 sets of vital signs showed less than 0.1% rate of suspected artifact. When compared to direct observation, our CIS correctly recorded days of mechanical ventilation in 23 of 26 patients (88%). Two other data sets, medical diagnoses and medications given showed significant differences with other commonly used databases of the same information collected outside the ICU (billing codes and pharmacy records respectively

$Conclusions:$ Compared to other commonly used data sources for clinical research, a commercially available CIS is an acceptable source of ICU patient data.

Section snippets

Materials and methods

Given the vast amounts of data are recorded with our system the analysis was limited to a cross-section of various data fields that we thought were important. Ideally, accuracy is determined by comparing the CIS data with some gold standard and is defined by 2 elements—correctness and completeness.8 However, in many cases no other such record of that data exists with which to compare the data. In some cases where another parallel records existed, it was believed that the comparison data set was

Set 1. length of stay in ICU

Length of stay (LOS) in the ICU is often used as a measurement of outcome in critical care research. Unfortunately, in many ICUs today, the lack of bed availability outside the ICU can often cause patients to remain physically in the ICU despite being medically ready for transfer. Error can therefore be introduced into this outcome measurement by using either standard hospital census records or by the printed medical record. Nevertheless, this marker of outcome is commonly used in ICU outcomes

Discussion

In 1997, Hogan and Wagner wrote the only comprehensive review of the literature on the accuracy of computer-based records.8 Their analysis showed that good accuracy data is clearly lacking. They reviewed the data from 20 previous studies that sought to measure, in some way, the accuracy of a CIS, which they defined as including both correctness and completeness of information. Unfortunately, the systems studied and methodology of the studies were so different that no clear conclusions could be

Conclusions

An important question that needs to be addressed is the degree to which these data from a single ICU with a specific CIS can be extrapolated to other care units with other systems. To some degree, most of these data can not be automatically extrapolated to other systems in other units. One of the queries in our study (set 3/DOV) was specifically programmed into our system by the vendor and is not automatically available even to those with our same system. This lack of generalizability has

References (17)

J. Hammond et al.
A qualitative comparison of paper flowsheets vs a computer-based clinical information system
Chest
(1991)
G.T. Kennedy et al.
Miscoding of hospital discharges as acute myocardial infarctionimplications for surveillance programs aimed at elucidating trends in coronary artery disease
Am J Cardiol
(1984)
M.R. Dawdy et al.
Correlation of patient entry rates and physician documentation errors in dictated and handwritten emergency treatment records
Am J Emerg Med
(1997)
B. Goldstein
Decomplexification in critical illness and injuryrelationship between heart rate variability, severity of illness, and outcome
Crit Care Med
(1998)
R.S. Evans
A decision support tool for antibiotic therapy
Proc Annu Symp Comput Appl Med Care
(1995)
R.S. Evans
A computer-assisted management program for antibiotics and other antiinfective agents
N Engl J Med
(1998)
W.G. Cole et al.
Human performance evaluation of a metaphor graphic display for respiratory data
Methods Inf Med
(1994)
M.M. Wagner et al.
The accuracy of medication data in an outpatient electronic medical record
J Am Med Inform Assoc
(1996)

There are more references available in the full text version of this article.

Cited by (14)

Impact of computerized information systems on workload in operating room and intensive care unit
2009, Best Practice and Research: Clinical Anaesthesiology
Citation Excerpt :
For instance, most ICU's are required to collect information on the severity of illness and intensity of nursing activity of their patients using scoring systems like the TISS, APACHE or SOFA score.12,34–37 The raw data for these scoring systems can be extracted automatically from the CIS, not only reducing the workload, but probably improving the quality of the data.9,13,38–46 Another aspect is billing: manual collection of chargeable items is no longer necessary, the accumulated items per patient can be sent automatically to the financial department of the hospital.16,21,46,47
The number of operating rooms and intensive care departments equipped with a clinical information system (CIS) is rapidly expanding. Amongst the putative advantages of such an installation, reduction in workload for the clinician is one of the most appealing. The scarce studies looking at workload variations associated with the implementation of a CIS, only focus on direct workload discarding indirect changes in workload. Descriptions of the various methods to quantify workload are provided.
The hypothesis that a third generation CIS can reduce documentation time for ICU nurses and increase time they spend on patient care, is supported by recent literature. Though it seems obvious to extrapolate these advantages of a CIS to the anesthesiology department or physicians in the intensive care, studies examining this assumption are scarce.
Validation of Surgical Intensive Care-Infection Registry: A Medical Informatics System for Intensive Care Unit Research, Quality of Care Improvement, and Daily Patient Care
2008, Journal of the American College of Surgeons
We developed a prototype electronic clinical information system called the Surgical Intensive Care-Infection Registry (SIC-IR) to prospectively study infectious complications and monitor quality of care improvement programs in the surgical and trauma intensive care unit. The objective of this study was to validate SIC-IR as a successful health information technology with an accurate clinical data repository.
Using the DeLone and McLean Model of Information Systems Success as a framework, we evaluated SIC-IR in a 3-month prospective crossover study of physician use in one of our two surgical and trauma intensive care units (SIC-IR unit versus non SIC-IR unit). Three simultaneous research methodologies were used: a user survey study, a pair of time-motion studies, and an accuracy study of SIC-IR's clinical data repository.
The SIC-IR user survey results were positive for system reliability, graphic user interface, efficiency, and overall benefit to patient care. There was a significant decrease in prerounding time of nearly 4 minutes per patient on the SIC-IR unit compared with the non SIC-IR unit. The SIC-IR documentation and data archiving was accurate 74% to 100% of the time depending on the data entry method used. This accuracy was significantly improved compared with normal hand-written documentation on the non SIC-IR unit.
SIC-IR proved to be a useful application both at individual user and organizational levels and will serve as an accurate tool to conduct prospective research and monitor quality of care improvement programs.
Towards automated classification of intensive care nursing narratives
2007, International Journal of Medical Informatics
Nursing narratives are an important part of patient documentation, but the possibilities to utilize them in the direct care process are limited due to the lack of proper tools. One solution to facilitate the utilization of narrative data could be to classify them according to their content.
Our objective is to address two issues related to designing an automated classifier: domain experts’ agreement on the content of classes Breathing, Blood Circulation and Pain, as well as the ability of a machine-learning-based classifier to learn the classification patterns of the nurses.
The data we used were a set of Finnish intensive care nursing narratives, and we used the regularized least-squares (RLS) algorithm for the automatic classification. The agreement of the nurses was assessed by using Cohen's κ, and the performance of the algorithm was measured using area under ROC curve (AUC).
On average, the values of κ were around 0.8. The agreement was highest in the class Blood Circulation, and lowest in the class Breathing. The RLS algorithm was able to learn the classification patterns of the three nurses on an acceptable level; the values of AUC were generally around 0.85.
Our results indicate that the free text in nursing documentation can be automatically classified and this can offer a way to develop electronic patient records.
Implementing Single Source: The STARBRITE Proof-of-Concept Study
2007, Journal of the American Medical Informatics Association
Citation Excerpt :
Thus, data that may already be available electronically are keyed twice in the healthcare setting, transcribed to a paper case report form (CRF), and then keyed twice in the research setting (Figure 1). Large-scale direct use of healthcare data for research, although advocated by many,1–18 has thus far eluded researchers.9 Successful implementations described in the literature6,26 cite workflow incompatibility, additional research data requirements, and regulatory differences as challenges.7
Inefficiencies in clinical trial data collection cause delays, increase costs, and may reduce clinician participation in medical research. In this proof-of-concept study, we examine the feasibility of using point-of-care data capture for both the medical record and clinical research in the setting of a working clinical trial. We hypothesized that by doing so, we could increase reuse of patient data, eliminate redundant data entry, and minimize disruption to clinic workflow.
We developed and used a point-of-care electronic data capture system to record data during patient visits. The standards-based system was used for clinical research and to generate the clinic note for the medical record. The system worked in parallel with data collection procedures already in place for an ongoing multicenter clinical trial. Our system was iteratively designed after analyzing case report forms and clinic notes, and observing clinic workflow patterns and business procedures. Existing data standards from CDISC and HL7 were used for database insertion and clinical document exchange.
Our system was successfully integrated into the clinic environment and used in two live test cases without disrupting existing workflow. Analyses performed during system design yielded detailed information on practical issues affecting implementation of systems that automatically extract, store, and reuse healthcare data.
Although subject to the limitations of a small feasibility study, our study demonstrates that electronic patient data can be reused for prospective multicenter clinical research and patient care, and demonstrates a need for further development of therapeutic area standards that can facilitate researcher use of healthcare data.
The accuracy of clinical information systems
2004, Journal of Critical Care
The clinical information system (CIS) is becoming more common in intensive care units. These systems have the ability to record, store, and retrieve large amounts of clinical patient data with great ease. This should greatly facilitate outcomes research and quality assurance. Unfortunately, there is not much information available about the accuracy of the data coming from these systems. True accuracy of a patient record requires both completeness of data and correctness of data as well as legibility. Automated systems are clearly superior to human entered data in terms of completeness and legibility but the correctness of entered data remains unclear. There are aspects of automated data entry that facilitate erroneous data entry. This article reviews the existing literature on accuracy of CISs with special attention to the qualities of automated data entry that can lead to false data. Additionally, data are presented from a newly published study by the author evaluating the validity of data from a commercially available CIS.
Computers in the intensive care unit
2004, Journal of Critical Care

View all citing articles on Scopus

View full text

Comparison of a commercially available clinical information system with other methods of measuring critical care outcomes data

Abstract

Section snippets

Materials and methods

Set 1. length of stay in ICU

Discussion

Conclusions

Chest

Am J Cardiol

Am J Emerg Med

Decomplexification in critical illness and injuryrelationship between heart rate variability, severity of illness, and outcome

Crit Care Med

A decision support tool for antibiotic therapy

Proc Annu Symp Comput Appl Med Care

A computer-assisted management program for antibiotics and other antiinfective agents

N Engl J Med

Human performance evaluation of a metaphor graphic display for respiratory data

Methods Inf Med

The accuracy of medication data in an outpatient electronic medical record

J Am Med Inform Assoc