healthy4u: JAMIA

JAMIA: Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems

A new article appeared online 20 February 2013 in the Journal of the American Medical Informatics Association entitled "Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems" (<a href="http://jamia.bmj.com/content/early/2013/01/27/amiajnl-2012-001241.full" target="_blank">link to fulltext</a>) by David C Radley, Melanie R Wasserman, Lauren EW Olsho, Sarah J Shoemaker, Mark D Spranca and Bethany Bradshaw. The authors performed a meta-analysis of the literature on Computerized Practitioner Order Entry (CPOE) systems in inpatient settings and concluded: <blockquote class="tr_bq">"Processing a prescription drug order through a CPOE system decreases the likelihood of error on that order by 48% (or in  a range of 41% to 55% with ninety five percent confidence).  Given this effect size, and the degree of CPOE adoption and use in hospitals in 2008, we estimate a 12.5% reduction in medication errors, or ∼17.4 million medication errors averted in the USA in one year." </blockquote> It is important to know the potential benefits of CPOE, as the government has been pushing this technology since the foundation of the <a href="http://www.healthit.gov/newsroom/about-onc" target="_blank">Office of the National Coordinator</a> (for health IT) within HHS since 2004.  Indeed, reimbursement penalties on Medicare will start in 2015 for non-adopters of government <a href="http://hcrenewal.blogspot.com/2010/03/on-oncs-proposed-establishment-of.html" target="_blank">certified</a> health IT. It is especially important to get to the truth about CPOE specifically, and health IT in general, in terms of risks, benefits, return on investment, improvements, and alternatives. Not long before this new JAMIA article appeared, an active study of EHR problems with voluntary reporting by members of the <a href="https://www.ecri.org/Pages/default.aspx" target="_blank">ECRI Institute's</a> Patient Safety Organization (PSO) <a href="http://hcrenewal.blogspot.com/2013/02/peering-underneath-icebergs-water-level.html" target="_blank">produced some concerning data</a>.  Namely, that over a 9-week period starting April 16, 2012, and ending June 19, 2012, 171 health information technology-related problems were reported from just 36 healthcare facilities, primarily hospitals. Eight of the incidents reported involved patient harm, and three may have contributed to patient deaths. Obviously, extrapolating those number to:  1)  a much higher number of hospitals, of which the U.S. alone has <a href="http://www.aha.org/research/rc/stat-studies/fast-facts.shtml" target="_blank">approximately 5,700</a> plus other facilities such as long-term care, and private physician offices; 2)  over a full year, not just 9 weeks;  3) accounting for the perhaps 5% voluntary reporting level (per Koppel) of issues such as medication errors; 4) plus accounting for (per FDA) the issue of lack of recognition of IT as contributing to medical incidents (this list is not all-inclusive) - the results are of concern. Thus, work such as in this new JAMIA article on CPOE is important.  The article can be downloaded in its entirety as of this writing from the link above.  The article describes a methodology that is quite complex, and obviously a great deal of time and effort was put into it.  It appears to be a valiant effort to get us one step closer to the truth.  This should be applauded. I was impressed on first reading of this literature meta-analysis and its statistical calculations. (Actually I needed to read it several times to fully grasp the methodologies involved.) The question arose in my mind, however: can this article's conclusions be true, and the ECRI PSO Deep Dive study be true, at the same time? Prior to going into an analysis and perhaps detailed critique of the methodology, and knowing the difficulties and contradictions the literature on this topic presents, I decided first to look at the source articles selected from the literature for inclusion in the JAMIA meta-analysis. In doing so, issues became apparent that shed light on the difficulties of meta-analyses on topics such as this. The only methodological issue I will mention at this time is that the study used a <a href="http://hcrenewal.blogspot.com/2008/06/criticism-of-surrogate-endpoints-in.html" target="_blank">surrogate endpoint</a> - medication "error" rates before, and after, implementation of CPOE, rather than patient outcomes.  (The reason I put "error" in quotes is that, as the authors describe regarding study limitations, the exact definition varies from site to site and study to study.  They also acknowledge the limitations of using such an endpoint.)  Surrogate endpoints, however, may or may not reflect the actual information being sought regarding outcomes. As Roy Poses noted in a June 2008 post "<a href="http://hcrenewal.blogspot.com/2008/06/criticism-of-surrogate-endpoints-in.html" target="_blank">Criticism of Surrogate Endpoints in Whose Interests?</a>": <blockquote class="tr_bq">... The problem with surrogate endpoints is that they are surrogates for the real thing. In many cases, a treatment may appear beneficial when measured by its affect on such endpoints, but not turn out to be beneficial when measured by its affect on real clinical outcomes, e.g., alleviation of symptoms, improvement of function, and prolongation of survival. There are many reasons why this may be the case. </blockquote>This weakens the present study as a basis for social re-engineering.  The authors responsibly acknowledge that via the statement in the conclusion that: <blockquote class="tr_bq">Future research in this area will be critically important to inform policy and funding decisions regarding the development and implementation of CPOE in care delivery. </blockquote> When I reviewed the studies that were used for the meta-analysis, however, my enthusiasm for the results was diminished. The authors write: <blockquote class="tr_bq">Using the search terms of Ammenwerth et al, we updated the search using PubMed in February 2009, identifying 390 studies. Each was reviewed by two study authors (MRW and DCR). After applying the a priori inclusion/exclusion criteria, 10 studies were retained. [Listed in footnotes 10–19 - ed.]</blockquote><div id=":1cb"> Here are the 10 studies retained, as per footnotes #10 - 19.  Short excerpts (I am trying to keep this post relatively short) and my very brief comments about each of them are as follows.  Hyperlinks to the summaries and in some cases to fulltext are present in the online study itself at the full text link at top of this post. First, I note no randomized, controlled clinical trials, the gold standard of medical research.  That lack is not the fault of the authors; it is a general feature in the domain of healthcare information technology. That said:  Included study #1 (footnote 10): </div><div id=":1cb">Bates DW, Teich JM, et al, The impact of computerized physician order entry on medication error prevention. Brigham and Women's Hospital, J Am Med Inform Assoc 1999;6:313–21.</div><div id=":1cb"> <blockquote class="tr_bq">... During the study, the non-missed-dose medication error rate fell 81 percent, from 142 per 1,000 patient-days in the baseline period to 26.6 per 1,000 patient-days in the final period (P < 0.0001). Non-intercepted serious medication errors (those with the potential to cause injury) fell 86 percent from baseline to period 3, the final period (P = 0.0003). Large differences were seen for all main types of medication errors: dose errors, frequency errors, route errors, substitution errors, and allergies. For example, in the baseline period there were ten allergy errors, but only two in the following three periods combined (P < 0.0001).  The study periods were as follows: baseline, 51 days, Oct-Nov 1992; period 1, 68 days, Oct-Dec 1993; period 2, 49 days, Nov-Dec 1995; and period 3, 52 days, Mar-Apr 1997. </blockquote> I note that this was a highly advanced setting with long-standing Medical Informatics expertise, performed by Medical Informatics experts of the highest caliber.  This was an ideal environment for the implementation of good health IT.  The results may thus not be generalizable to facilities without that level of experience.  </div><div id=":1cb"></div><div id=":1cb">Also, the study was a considerable number of years ago, some of it two decades ago.  While one might assume the technology has improved, the increased commercial sector involvement since the 1990's, and especially after the HITECH incentives of 2009, may be creating an increased occurrence of bad health IT, and/or implementation in facilities with far less (if any) informatics expertise. Thus, in my view the study's applicability to current times and to all medical organizations is not extremely strong.  Included study #2 (footnote 11): Medication Administration Variances Before and After Implementation of Computerized Physician Order Entry in a Neonatal Intensive Care Unit, Pediatrics 2008;121:123–8 <blockquote class="tr_bq">... Data on 526 medication administrations, including 254 during the pre-computerized physician order entry period and 272 after implementation of computerized physician order entry, were collected. Medication variances were detected for 19.8% of administrations during the pre-computerized physician order entry period, compared with 11.6% with computerized physician order entry (rate ratio: 0.53). Overall, administration mistakes, prescribing problems, and pharmacy problems accounted for 74% of medication variances; there were no statistically significant differences in rates for any of these specific reasons before versus after introduction of computerized physician order entry.</blockquote> Here, 'n' is very small, and there is a finding that the CPOE had no effect on administration mistakes, prescribing problems, and pharmacy problems.  Thus, a ringing endorsement for national CPOE implementation this study is (unfortunately) not.  Included study #3 (footnote 12): The effect of computer-assisted prescription writing on emergency department prescription errors, Acad Emerg Med 2002;9:1168–75.<cite></cite> Without even a summary, my concern here is that ePrescribing and CPOE are different entities.   Inclusion of ePrescibing in a study of CPOE is not entirely without some risk of conflation of results of one with the other.  Included study #4 (footnote 13): Impact of computerized physician order entry on clinical practice in a newborn intensive care unit, J Perinatol. 2004 Feb;24(2):88-93. This article studies gentamicin dosing and turn around times and found that: <blockquote class="tr_bq">"...the accuracy of gentamicin dose at the time of admission for 105 (pre-CPOE) and 92 (post-CPOE) VLBW infants was determined. In the pre-CPOE period, 5% overdosages, 8% underdosages, and 87% correct dosages were identified. In the post-CPOE, no medication errors occurred. Accuracy of gentamicin dosages during hospitalization at the time of suspected late-onset sepsis for 31 pre- and 28 post-CPOE VLBW infants was studied. Gentamicin dose was calculated incorrectly in two of 31 (6%) pre-CPOE infants. No such errors were noted in the post-CPOE period. </blockquote> My comments are that a NICU is a specialized environment with a high ratio of clinicians/staff to patients.  Findings in such an environment again may not be generalizable.  Also, one should ask if complex CPOE systems are really needed for dosing calculations and turn around time improvements.  Simpler and cheaper human/technological solutions might have achieved similar or better results.  Thus, again, while not demeaning the results achieved by this study's interventions in 2004, I have my concerns that this study is not strong evidence of generalizability of even the CPOE surrogate measurement, namely decrease of med "errors."  Included study #5 (footnote 14): A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338:232–8. <blockquote class="tr_bq">We have developed a computerized decision-support program linked to computer-based patient records that can assist physicians in the use of antiinfective agents and improve the quality of care. This program presents epidemiologic information, along with detailed recommendations and warnings. The program recommends antiinfective regimens and courses of therapy for particular patients and provides immediate feedback. We prospectively studied the use of the computerized antiinfectives-management program for one year in a 12-bed intensive care unit. RESULTS: During the intervention period, all 545 patients admitted were cared for with the aid of the antiinfectives-management program. Measures of processes and outcomes were compared with those for the 1136 patients admitted to the same unit during the two years before the intervention period. The use of the program led to significant reductions in orders for drugs to which the patients had reported allergies (35, vs. 146 during the preintervention period; P less than 0.01), excess drug dosages (87 vs. 405, P less than 0.01), and antibiotic-susceptibility mismatches (12 vs. 206, P less than 0.01). There were also marked reductions in the mean number of days of excessive drug dosage (2.7 vs. 5.9, P less than 0.002) and in adverse events caused by antiinfective agents (4 vs. 28, P less than 0.02).  [Several other benefits omitted for brevity - they can be seen at the JAMIA footnote hyperlink -  ed.]</blockquote></div> My thoughts here are that, while the results were commendable, once again this study took place 15 years ago, and was in a high-staff-to-patient specialized ICU environment.  I also wonder if complex, expensive CPOE is needed to accomplish these tasks as opposed to, say, an online DSS and appropriate workflows and process.  Included study #6 (footnote 15): Impact of computerized prescriber order entry on medication errors at an acute tertiary care hospital. Hosp Pharm 2003;38:227–31. <blockquote class="tr_bq">The authors analyzed medication errors documented in a hospital's database of clinical interventions as a continuous quality improvement activity. They compared the number of errors reported prior to and after computerized prescriber order entry (CPOE) was implemented in the hospital. Results indicated that in the first 12 months of CPOE, overall medication errors were reduced by more than 40%, incomplete orders declined by more than 70%, and incorrect orders decreased by at least 45%. Illegible orders were virtually eliminated but the level of medication errors categorized by drug therapy problems remained significantly unchanged. The study underscores the positive impact of CPOE on medication safety and reemphasizes the need for proactive clinical interventions by pharmacists.</blockquote> This study appears reasonable for inclusion in a meta-analysis, although ideally there might have been accounting for possible influence of non-intervention (computer)-related pre-post interval changes.  The transition to CPOE, training, increased awareness, etc. can influence results, especially short term.  Included study #7 (footnote 16): Error reduction in pediatric chemotherapy: computerized order entry and failure modes and effects analysis. Arch Pediatr Adolesc Med 2006;160:495–8. <blockquote class="tr_bq">Before-and-after study from 2001 to 2004. After CPOE deployment, daily chemotherapy orders were less likely to have improper dosing (relative risk [RR], 0.26; 95% confidence interval [CI], 0.11-0.61), incorrect dosing calculations (RR, 0.09; 95% CI, 0.03-0.34), missing cumulative dose calculations (RR, 0.32; 95% CI, 0.14-0.77), and incomplete nursing checklists (RR, 0.51; 95% CI, 0.33-0.80). There was no difference in the likelihood of improper dosing on treatment plans and a higher likelihood of not matching medication orders to treatment plans (RR, 5.4; 95% CI, 3.1-9.5).</blockquote> Again, the results appear commendable.  However:  there was no difference in the likelihood of improper dosing on treatment plans, and worse, there was found a higher likelihood of not matching medication orders to treatment plans. In fact, this article was accompanied by a letter in response entitled "Primum non nocere", David Dickens, MD; Dianne Sinsabaugh, RPh; Brenda Winger, PharmD, Arch Pediatr Adolesc Med. 2006;160(11):1185-1186 (after some digging, text found at <a href="http://archpedi.jamanetwork.com/article.aspx?articleid=486317" target="_blank">http://archpedi.jamanetwork.com/article.aspx?articleid=486317</a>): <blockquote class="tr_bq">Kim et al. demonstrated that in the practice of pediatric oncology, computerized physician order entry (CPOE) reduced improper dosing, missing cumulative doses, and incomplete nursing checklists.  In contrast to these benefits, however, CPOE also resulted in a 5-fold increase in “not matching medication orders to treatment plans.” Although little detail was provided on the nature of these medication order/treatment plan “mismatches,” it implies that chemotherapy ordered through CPOE deviated more often from intended protocol therapy as compared with paper-ordered chemotherapy. While CPOE ostensibly led to more precise chemotherapy dosing, it increased the risk of that chemotherapy being the wrong chemotherapy.</blockquote> The author did respond: "Mismatches between treatment plan and orders at point of therapy increased, but were intercepted and clarified at POC. [By people - ed.].  No incorrect meds were given. On its face, this is not an entirely dispositive proof of CPOE beneficence and raises significant concern that, sooner or later, a person might miss the discrepanc(ies) resulting in unintended adverse consequences.  Included study #8 (footnote 17): Effects of an integrated clinical information system on medication safety in a multi-hospital setting. Am J Health Syst Pharm 2007;64:1969–77. <blockquote class="tr_bq">This study took place at Lifespan health care system that includes Rhode Island Hospital (RIH), a private, 719-bed, not-for-profit, acute care hospital and academic medical center that has a pediatric division, the Hasbro Children’s Hospital; and The Miriam Hospital (TMH), a 247-bed, not-for-profit, acute care general hospital. Methods. The integrated systems selected for implementation included computerized physician order entry, pharmacy and laboratory information systems, clinical decision-support systems (CDSSs), electronic drug dispensing systems (EDDSs), and a bar-code point-of-care medication administration system. The indicators for CPOE with inherent CDSSs demonstrated a significant effect of this functionality on reducing prescribing error rates for three of the four indicators measured: drug allergy detection, excessive dose, and incomplete or unclear order. The fourth indicator measured, therapeutic duplication, did not show a significant effect on prescribing error rates. For the rules engine software CDSS, the colchicine indicator did not show a statistically significant effect on prescribing error rate, but a significant decrease in prescribing errors related to metformin use in renal insufficiency was observed after implementation of the rules engine software and integration with CPOE. </blockquote> Again, these are commendable results, but on its face the technologies involved went far beyond just CPOE, and the results were not uniform.  The actual reduction figures for seven categories were mostly in 50% range, one at 86% (allergy), but the duplicates issue at 8%, not felt statistically significant. Of more concern, there was this in the news in 2011.  A software bug at this organization led to thousands and perhaps tens of thousands of prescription errors that could have (and without definite proof, despite organization denials I would be concerned did) led to injury and death.  I wrote about the malfunction at <a href="http://hcrenewal.blogspot.com/2011/11/lifespan-rhode-island-yet-another.html">http://hcrenewal.blogspot.com/2011/11/lifespan-rhode-island-yet-another.html</a>.  The bug was not discovered for about a year. Other organizations, especially those new to CPOE and/or health IT, face similar risks. Again, this is not a caveat-free endorsement of national CPOE rollout in 2013. Included study #9 (footnote 18): Effect of computer order entry on prevention of serious medication errors in hospitalized children. Pediatrics 2008;121:e421–7. 627 pediatric admissions, with 12 672 medication orders written over 3234 patient-days.  The rate of non-intercepted serious medication errors in this pediatric population was reduced by 7% after the introduction of a commercial computerized physician order entry system, much less than previously reported for adults, and there was no change in the rate of injuries as a result of error. Several human-machine interface problems, particularly surrounding selection and dosing of pediatric medications, were identified. The issues of concern here are bolded and underlined above and need not be restated. Lastly: Included study #10 (footnote 19): Evaluation of reported medication errors before and after implementation of computerized practitioner order entry. J Health Inf Manag 2006;20:46–53. <blockquote class="tr_bq">While a major objective of CPOE is to reduce medication errors, its introduction is a major system change that may result in unintended outcomes. Monitoring voluntarily-reported medication errors in a university setting was used to identify the impact of initial CPOE implementation on medical-surgical and intensive care units. A retrospective trend analysis was used to compare errors one year before and six months after implementation. Total error reports increased post-CPOE but the level of patient harm related to those errors decreased. Numerous modifications were made to the system and the implementation process. The study supports the notion that CPOE configuration and implementation influences the risk of medication errors. Implementation teams should incorporate monitoring medication errors into project plans and expect to make ongoing changes to continually support the design of a safer care delivery environment.</blockquote> This study appeared more a study reporting unintended outcomes than benefits. The total medication error reports increased post-CPOE but the level of patient harm related to those errors decreased.   (That decrease might have been due to human factors, or to serendipity, both of which cannot be expected to protect forever.) The reasons for the changes were described this way: <blockquote class="tr_bq">... Contributing causes. To assist in the development of safety interventions, contributing causes were identified for reported errors. The most common contributing cause was noncompliance to policy and procedure, identified in 40 percent of errors. For example, a previous order may not have been discontinued when a new dose change was entered, resulting in two active orders for the same medication with different dosages  The next most common contributing cause was computer entry errors, seen in 25 percent of mistakes. One example was if a medication order was placed on the wrong patient.  ["Use error" due to confusing user interfaces as recently defined by NIST - as opposed to "user error" - was likely to have contributed to at least some of these errors - ed.]  The next most common error was initial load errors (19 percent). During entry of all current medications on the day of activation, multiple category B errors were made. An example was a written order for sliding scale calcium gluconate “PRN,” which was entered into the CPOE system as “scheduled.” There were also computer design issues that contributed to 10 percent of errors. An example was when the pharmacist received two printouts for methylprednisolone 500 mg IV. He assumed it was a duplicate order, but when he reviewed the CPOE system, he saw that one order was for today and the other was for tomorrow. The dates for these orders were not visible on the order printout from the CPOE system. [This again seems to be 'use error' - ed.]</blockquote><div><div> <div><div> These issues can and will occur anywhere.  Once again, this is not entirely an article, either by itself or in a meta-analysis, that I find ideal in attempted proof of CPOE effectiveness and beneficence. In fact, after I reviewed this source, I noted that this study was eliminated from the inclusion set: <blockquote class="tr_bq">... Based on later expert reviewer feedback, we eliminated one additional study that solely used a voluntary reporting method for error detection, leaving nine studies for our final pooled analysis. </blockquote></div></div><div><div> It would probably not be hard to convince critical thinkers of the possibility this study was removed for reasons other than stated. In summary, while one should not and cannot expect perfection in the studies available in a meta-analysis, due to the difficulties in this domain the literature resources utilized were not ideal, and the surrogate endpoint also raises concern.  (I have not reviewed the entire corpus of potential literature myself.)  The authors conducted a difficult and rigorous study, but one cannot turn data "lead" (as in Pb) to gold (as in Au), no matter how hard one works or however good one's intentions are. The authors did note the limitations of the study, although their conclusion will likely be taken by the industry as a "full steam ahead" signal.   (I do note with some irony the proximity of this article's release to the soon-to-start massive <a href="http://www.himssconference.org/" target="_blank">HIMSS 2013 Annual Conference & Exhibition trade show</a>, March 3-7, 2013 in New Orleans, LA.)  While likely a coincidence, my concern is that CPOE vendors will be talking nonstop about these results. Thus, I agree with the author's conclusion (especially in view of the recent voluntary reporting-based ECRI PSO study) that "future research in this area will be critically important to inform policy and funding decisions regarding the development and implementation of CPOE in care delivery." From a clinical perspective, "primum non nocere" and the avoidance of gambling billions of dollars applies, at least until a better understanding of the technology's risk/benefit ratio and how to improve it occurs. A fraction of those billions would pay for more robust, current studies on the scale needed to get closer to the truth, such as formal post-market, mandatory surveillance that measures not surrogate but primary variables - such as outcomes both positive and negative.  As noted by the authors, voluntary reporting has the least sensitivity towards uncovering error:  <blockquote class="tr_bq">... Reviewed studies used various medication error detection methods. Research suggests that the highest error rates are found through direct observation, followed by chart review, then automated surveillance, and voluntary reporting.  [Citations were made to papers by Flynn and Jha regarding these points - ed.] </blockquote> Formal studies are essential from the basis of medical (e.g., safety and public health), business (e.g., ROI and liability), and social policy perspectives (e.g., are we spending the billions of dollars this technology costs wisely). -- SS Addendum:    I have often been the <a href="http://hcrenewal.blogspot.com/2013/02/modern-healthcare-health-it-iconoclasts.html" target="_blank">fire-breathing and </a><a href="http://hcrenewal.blogspot.com/2013/02/modern-healthcare-health-it-iconoclasts.html" target="_blank">über-skeptic iconoclast</a> on matters such as this.  However, I will allow a true international expert to take on that role this time, Dr. Richard Cook, who had a guest post here yesterday (<a href="http://hcrenewal.blogspot.com/2013/03/dr-richard-cook-on-health-it-sectors.html" target="_blank">link</a>).  Dr. Cook's opinion on this JAMIA study (again, reproduced here with permission) was this: <blockquote class="tr_bq"><div>A meta-analysis of the literature on the nature of the universe in the mid 1500's would have concluded that the sun revolves around the earth. The data isn't fake, just worthless.  </div></blockquote> -- SS </div></div></div></div>

JAMIA: Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems

A new article appeared online 20 February 2013 in the Journal of the American Medical Informatics Association entitled "Reduction in medication errors in hospitals due…

Đọc thêm »

02 Mar 2013

Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop

A preprint article "Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop" by Bonnie Kaplan and Kimberly D. Harris-Salamone (doi:10.1197/jamia.M2997) has just been published and made available in the Journal of the American Medical Informatics Association (<a href="http://www.jamia.org/">JAMIA</a>). It will appear in the May/June 2009 issue of JAMIA. [Addendum 1/2010: a link to a free PDF of the published article is <a href="http://jamia.bmj.com/content/16/3/291.full.pdf">here</a>.] There is a history to this publication, that summarizes the findings and recommendations of a surprisingly well attended workshop ironically named "<a href="http://www.ischool.drexel.edu/faculty/ssilverstein/failure_workshop_final_revised.doc">Avoiding The F-Word: IT Project Morbidity, Mortality, and Immortality</a>." The workshop was held at the 2006 national meeting of AMIA. Ten years ago, in early 1999 I started a website hosted on AOL, and called at that time "Medical Informatics and Leadership of Clinical Computing: Common Examples of Healthcare IT Failure." A historical version of the site (with a "politically corrected" title) is <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/medinformaticsmd/failurecases.htm">here</a>; the modern site is now <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/cases/">here</a>. The website was unique at the time and today, remarkably, remains nearly so (see <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/cases/content/documents/FINAL_Healthcare%20IT%20Failure_Oct_19_2006.ppt">this pdf poster</a> from AMIA's 2006 annual meeting). Yet I was to soon learn through correspondence about the website that these problems were common as well as international in scope. Healthcare IT failure was (and is) a somewhat "taboo" subject, in no small part because of contractual gag clauses on users and vendor lack of legal accountability for defects, as well as a striking overconfidence in computing and timidity about discussing organizational failure and IT mismanagement. Its open discussion has therefore been severely impaired and minimized. <div> </div><div>I pushed this topic hard among the Medical Informatics community, yet for years there was resistance to bringing this issue to the fore. I took significant "heat" for my views as well, making me somewhat of an outcast in my own professional community and <a href="http://hcrenewal.blogspot.com/2007/03/problems-found-in-uc-davis-emr-project.html">even to one of my former mentees.</a> This phenomenon may have been a combination of academic concerns about a topic that might dilute the field, preferential and understandable focus on successes, conflicts of interest, and other issues. <div> </div><div>[May 2009 addendum: the "taboo" nature of the subject of HIT difficulty and failure might in fact have been engineered by the HIT industry; see the article "<a href="http://www.washingtonpost.com/wp-dyn/content/article/2009/05/15/AR2009051503667.html?hpid=topnews">The Machinery Behind Healthcare Reform: How an Industry Lobby Scored a Swift, Unexpected Victory by Channeling Billions to Electronic Records</a>" by Robert O'Harrow, Jr., May 16, 2009, Washington Post -- ed.]</div><div> I, on the other hand, having been (pre-informatics fellowship) a Medical Review Officer in the medical department of a regional mass transit authority, and observing a fatal subway elevated accident related to drug abuse, was not going to back down. The accident was the worst in perhaps sixty years in Philadelphia, with hundreds of injuries and several deaths in part due to overriding of medical judgment by non-medical personnel on drug testing at time of union-demanded reinstatement of persons known to have problems. This was as a quid pro quo for labor contract approval. Worse, there were IT errors as well -- again with overriding of medical personnel, namely, me. I'd written a program on our departmental PC for random drug test date selection, but by MIS edict, use of that program was overridden in favor of some slop they authored for their machines - which "forgot" to enroll the operator behind the accident in random testing altogether. Months went by with no tests, and then at the time of the accident the operator's cocaine metabolite levels "blew the lid off the GCMS machine", according to the toxicologist performing the test. Having been informed by my earlier transit authority experience (and gaining some lessons in fortitude from the militant mass transit labor unions I had dealings with), I started the HIT difficulties website in late 1998-early 1999 after <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=clinical%20computing%20problems%20in%20ICU">watching ICU patients needlessly being put at risk</a> and the high risk <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=Cardiology%20story">Invasive Cardiology cath lab being disrupted</a> as CMIO at Christiana Care in Mr. Biden's home state. These problems were once again due to the ill-informed, capricious edicts of overempowered MIS leaders, sanctioned by equally ill-informed executive leadership. In fact, I quit that CMIO role, no longer being able to tolerate being a marginalized "Director of Workarounds to Dangerous Health IT Mismanagement" (as are many of today's CMIOs), unable by management caprice to make even the most basic of corrections to obvious and risky - and grossly negligent, perhaps even criminally negligent - design and implementation errors. With the exception of a period of time in pharmaceutical research IT from 2000-2003, I periodically sought additional case information on healthcare IT difficulties from colleagues via email and AMIA message board postings, but the results were lukewarm at best. I'd hoped upon returning to a focus on the healthcare provider IT sector in late 2003 that conditions in that sector would have improved. I was wrong. I continued to comment on these issues but with results similar to the past. However, as more informaticists became familiar with the organizational and clinician chaos that ill-conceived clinical IT and/or suboptimally managed electronic medical records projects were causing, a "critical mass" of interest and determination appeared to have been reached by early 2006. At my request my assigned research assistant at Drexel sent out the following message to a number of AMIA message boards, the clinical information systems (cis), ethical, legal and social (els), evaluation special interests (eval-sig) and the people and organizational issues (poi) workgroups. <blockquote style="font-family: arial;">From: poi-wg-bounces@mailman.amia.org [mailto:poi-wg-bounces@mailman.amia.org] On Behalf Of Yunan Chen Sent: February 27, 2006 8:46 AM To: cis-wg@mailman.amia.org; els-wg@mailman.amia.org; eval-sig@mailman.amia.org; poi-wg@mailman.amia.org Subject: [poi-wg] Healthcare IT failure cases Hello: My name is Yunan Chen and I am a research assistant for Dr. Scot Sliverstein in the Institute of Healthcare Informatics , College of Information Science & Technology at Drexel University . Currently, we are working on a project regarding Healthcare IT infrastructure failures. The purpose of our project is to investigate why Healthcare IT infrastructure may fail from a social- technologic perspective. We will collect cases published in scientific papers and newspapers, as well as unpublished stories by the Healthcare IT practitioners. Then, we will categorize these cases and build a casebase for future implementation guideline. We have already collected some cases which are listed in: http://home.aol.com/medinformaticsmd/failurecases.htm . The cases listed here were collected from 1998-2001. Now we hope to gather newly happened cases to enrich our collection. If you experienced or heard of any interesting story, please do not hesitate to write it down and send it to me. We hope to hear more from healthcare IT practitioners' hand-on experience. Any story is valuable to us. Any posting of the cases will be keep anonymous and we won't reveal identities of people or organizations . We will move the Healthcare IT failure cases webpage to Drexel University server later, so all your stories will be listed on that site. We will inform you once the new website is ready. If interested, please write back to me at: yunan.chen@ischool.drexel.edu . Comments and suggestions are welcome. I am looking forward to hearing back from you. Thanks very much. Yunan Chen Doctoral student & Research assistant Institute of Healthcare Informatics College of Information Science & Technology Drexel University</blockquote> It soon became apparent that a critical mass had been reached. A torrent of shared experiences of HIT difficulty in the various message boards appeared from all over the country and from other countries as well. Calls for a formal AMIA panel or workshop on this issue arose. This call was energetically supported by a heterogeneous and widespread group of informatics professionals. This momentum prompted the "<a href="http://www.ischool.drexel.edu/faculty/ssilverstein/failure_workshop_final_revised.doc">Avoiding The F-Word</a>" workshop to be collaboratively formulated in a wonderful volunteer effort led by former colleague Bonnie Kaplan at the Yale Center for Medical Informatics (where I'd completed my postdoc in medical informatics), and Kim Harris-Salamone, along with the AMIA workgroup members. Ten <a href="http://www.amia.org/mbrcenter/wg">AMIA workgroups</a> and over fifty HIT experts ultimately participated in the workshop. In Sept. 2006, before the workshop occurred, I wrote at "<a href="http://hcrenewal.blogspot.com/2006/09/holes-in-quest-to-enable-electronic_21.html">The holes in the quest to enable the electronic clinical trial ...</a>": <blockquote style="font-family: arial;">... I also think some answers to the question "Where are the holes in the quest to enable the electronic clinical trial?" will be found in an upcoming Nov. 2006 American Medical Informatics Association Annual Conference workshop on healthcare IT failure entitled "<a href="http://www.ischool.drexel.edu/faculty/ssilverstein/failure_workshop_final_revised.doc">Avoiding The F-Word: IT Project Morbidity, Mortality, and Immortality</a>", the first of its kind: <o:p></o:p> <blockquote><div class="MsoNormal">SESSION DESCRIPTION Recent studies of health care computer applications and the reported failures of well-known systems surprised the medical informatics community, leading to questions of how to increase the chances of IT systems success and the reduction of errors. Similar problems plague a variety of different systems, whether for institutions as a whole, for ancillary services, or for consumer health, and have done so for many years. Despite an accumulation of best practices research that has identified a series of success factors, some 40% of information technology developments in a variety of sectors are either abandoned or fail, while fewer than 40% of large systems purchased from vendors meet their goals. According to the recent CHAOS Report by The Standish Group, which surveyed failures of IT in general (not just in health care), only 34% of IT projects were considered truly successful. Similar numbers have been estimated for health care, and the number has unfortunately remained approximately the same for at least the last 25 years. While there have been some published reports of failures, removals, sabotage of systems, or how failures became successes or were otherwise redefined, there has been too little opportunity to learn from studies in which technology interventions resulted in null, negative, or disappointing results. The purpose of this session is to examine why this happens and what might be done to improve the situation, and to collaboratively develop a series of frameworks for various types of systems and healthcare settings to aid in implementation and evaluation. The session builds on a lively exchange by numerous members of a number of AMIA Working Groups concerning success and failure in medical informatics. The session will be devoted to better defining or characterizing "success" and "failure." From there, participants will break out into smaller groups to continue the discussion, develop a set of important issues, action items, and recommendations.<o:p></o:p></div></blockquote></blockquote> The report, now published, is filled with pearls such as these: <ul><li>With the US joining other countries in national efforts towards the many benefits health information technology use can bring for health care quality and savings, recent sobering reports recall the complexity and difficulties of implementing even smaller-scale systems. Despite best practice research identifying success factors for health information technology projects, a majority, in some sense, still fail. </li></ul><ul style="font-family: arial;"><li>With the United States Congress appropriating more than $20 billion for health information technology as part of the February, 2009 economic stimulus package, the US joined other countries in national efforts towards the many proven benefits such technology use can bring for health care quality and savings. Moreover, Medicare, along with private and commercial health plans, is implementing a new paradigm for paying for health care services in the US, known as Value-Based Purchasing (VBP), or pay for performance initiatives (P4P). These initiatives rely heavily on using electronic health records to provide clinical documentation that proves the value of their services. Tempering the fervor, though, are recent sobering reports that raise concerns about how the technology is designed and deployed.</li></ul><ul><li>Similar failure rates [to other types of IT] have been reported for health IT.(14, 15) Hospitals are among those organizations where delays and cancellations of software projects are endemic.(16) For years, problems have plagued the implementation of health IT applications, whether for ancillary services, for whole institutions, for regional or national systems, or for consumers. Today's problems are reminiscent of those analyzed since at least the 1970s in classic studies of hospital information and patient record systems.(17-19) In 1990, Dowling estimated that staff interfere or sabotage with "nearly half" of projects(20), while Heeks noted in 2006 that it is his "best estimate...that most HIS [health information systems] fail in some way."(15) Recent studies and newspaper accounts cite difficulties in a variety of health information technology applications. Over the years, in many countries, patterns of severe problems repeatedly have beset a variety of efforts: hospital information systems and electronic records(21-26); ambulance services(27, 28); community, regional, and national health information networks (28-33); public health systems (34, 35); patient education (36); and physician order entry.(18, 19, 37-41) </li></ul><ul style="font-family: arial;"><li>Yet, while there have been some published research reports of health care IT failures, unfortunately, there has been an absence of systematic and thoughtful publication of lessons learned from IT interventions where there have been null, negative, or disappointing outcomes.(27, 53) Despite calls for increased research, there are insufficient numbers of published research reports of health care IT failures, removals, sabotage of systems, or how failures became successes or were otherwise redefined. As in other sectors (69), IT-related failures in health care often are covered up, ignored, or rationalized, so mistakes are repeated.</li></ul><ul style="font-family: arial;"><li>Participants emphasized that communication and workflow issues add to project complexity. Health care requires collaboration, as does system implementation, yet there is difficulty in translating among specialties, stakeholders, clinicians, and implementers, sometimes to the point of a seeming "culture clash."</li></ul><ul style="font-family: arial;"><li>The workshop concluded with reports from break-out groups charged with discussing ideas for how AMIA could address health informatics failure. Break-out groups made suggestions concerning: research and publication, best practices, advocacy, education, certification, and databases and knowledge integration.</li></ul> The workshop findings concluded with these words of wisdom: <blockquote style="font-family: georgia;">Much has been learned about success and failure in IT implementation, but we need to understand more. There are legal issues when a system "fails," including just what constitutes "failure." There are social issues, ranging from how such failures affect various groups and health informatics as a whole (including possible policy and regulatory reactions), to the social aspects of what makes for a "successful" implementation. Finally there are ethical issues involved in evaluating system "success" or not sufficiently attending to previously-identified success factors and best practices.(24) Most "failures" are failures to properly apply managerial wisdom that has been substantiated by research and experience. Perhaps the worst aspect of failure is failure to learn from past experiences so that the same issues and problems are not perpetuated. </blockquote> While the workshop was a culmination of the kindling on this crucial topic I've been involved in since being overruled on <a href="http://www.ischool.drexel.edu/faculty/ssilverstein/cases/?loc=cases&sloc=clinical%20computing%20problems%20in%20ICU">patient-endangering healthcare IT mismanagement</a>, I believe we as a society, and the HIT sector as a for-profit, entirely unregulated industry lacking accountability (the accountability rests on clinical end users, as in, in the courtroom in malpractice suits), are still very far from solutions to that final observation on failure to learn from history. The observations of this workshop are only made more acutely important by what I believe are ill advised, rushed plans to coerce US healthcare organizations and practitioners to adopt these evolving, arguably still-experimental virtual medical devices by 2014, or face penalties. -- SS</div></div>

Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop

A preprint article "Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop" by Bonnie Kaplan and Kimberly D. Harris-Salamone (doi:10.1197/jamia.M2997) has just b…

Đọc thêm »

05 Mar 2009

Load more posts