Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop

A preprint article "Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop" by Bonnie Kaplan and Kimberly D. Harris-Salamone (doi:10.1197/jamia.M2997) has just been published and made available in the Journal of the American Medical Informatics Association (JAMIA). It will appear in the May/June 2009 issue of JAMIA.

[Addendum 1/2010: a link to a free PDF of the published article is here.]

There is a history to this publication, that summarizes the findings and recommendations of a surprisingly well attended workshop ironically named "Avoiding The F-Word: IT Project Morbidity, Mortality, and Immortality." The workshop was held at the 2006 national meeting of AMIA.

Ten years ago, in early 1999 I started a website hosted on AOL, and called at that time "Medical Informatics and Leadership of Clinical Computing: Common Examples of Healthcare IT Failure." A historical version of the site (with a "politically corrected" title) is here; the modern site is now here. The website was unique at the time and today, remarkably, remains nearly so (see this pdf poster from AMIA's 2006 annual meeting).

Yet I was to soon learn through correspondence about the website that these problems were common as well as international in scope.

Healthcare IT failure was (and is) a somewhat "taboo" subject, in no small part because of contractual gag clauses on users and vendor lack of legal accountability for defects, as well as a striking overconfidence in computing and timidity about discussing organizational failure and IT mismanagement. Its open discussion has therefore been severely impaired and minimized.

I pushed this topic hard among the Medical Informatics community, yet for years there was resistance to bringing this issue to the fore. I took significant "heat" for my views as well, making me somewhat of an outcast in my own professional community and even to one of my former mentees. This phenomenon may have been a combination of academic concerns about a topic that might dilute the field, preferential and understandable focus on successes, conflicts of interest, and other issues.

[May 2009 addendum: the "taboo" nature of the subject of HIT difficulty and failure might in fact have been engineered by the HIT industry; see the article "The Machinery Behind Healthcare Reform: How an Industry Lobby Scored a Swift, Unexpected Victory by Channeling Billions to Electronic Records" by Robert O'Harrow, Jr., May 16, 2009, Washington Post -- ed.]

I, on the other hand, having been (pre-informatics fellowship) a Medical Review Officer in the medical department of a regional mass transit authority, and observing a fatal subway elevated accident related to drug abuse, was not going to back down.

The accident was the worst in perhaps sixty years in Philadelphia, with hundreds of injuries and several deaths in part due to overriding of medical judgment by non-medical personnel on drug testing at time of union-demanded reinstatement of persons known to have problems. This was as a quid pro quo for labor contract approval.

Worse, there were IT errors as well -- again with overriding of medical personnel, namely, me. I'd written a program on our departmental PC for random drug test date selection, but by MIS edict, use of that program was overridden in favor of some slop they authored for their machines - which "forgot" to enroll the operator behind the accident in random testing altogether. Months went by with no tests, and then at the time of the accident the operator's cocaine metabolite levels "blew the lid off the GCMS machine", according to the toxicologist performing the test.

Having been informed by my earlier transit authority experience (and gaining some lessons in fortitude from the militant mass transit labor unions I had dealings with), I started the HIT difficulties website in late 1998-early 1999 after watching ICU patients needlessly being put at risk and the high risk Invasive Cardiology cath lab being disrupted as CMIO at Christiana Care in Mr. Biden's home state. These problems were once again due to the ill-informed, capricious edicts of overempowered MIS leaders, sanctioned by equally ill-informed executive leadership.

In fact, I quit that CMIO role, no longer being able to tolerate being a marginalized "Director of Workarounds to Dangerous Health IT Mismanagement" (as are many of today's CMIOs), unable by management caprice to make even the most basic of corrections to obvious and risky - and grossly negligent, perhaps even criminally negligent - design and implementation errors.

With the exception of a period of time in pharmaceutical research IT from 2000-2003, I periodically sought additional case information on healthcare IT difficulties from colleagues via email and AMIA message board postings, but the results were lukewarm at best.

I'd hoped upon returning to a focus on the healthcare provider IT sector in late 2003 that conditions in that sector would have improved. I was wrong. I continued to comment on these issues but with results similar to the past.

However, as more informaticists became familiar with the organizational and clinician chaos that ill-conceived clinical IT and/or suboptimally managed electronic medical records projects were causing, a "critical mass" of interest and determination appeared to have been reached by early 2006.

At my request my assigned research assistant at Drexel sent out the following message to a number of AMIA message boards, the clinical information systems (cis), ethical, legal and social (els), evaluation special interests (eval-sig) and the people and organizational issues (poi) workgroups.

From: poi-wg-bounces@mailman.amia.org [mailto:poi-wg-bounces@mailman.amia.org] On Behalf Of Yunan Chen
Sent: February 27, 2006 8:46 AM
To: cis-wg@mailman.amia.org; els-wg@mailman.amia.org; eval-sig@mailman.amia.org; poi-wg@mailman.amia.org
Subject: [poi-wg] Healthcare IT failure cases

Hello:

My name is Yunan Chen and I am a research assistant for Dr. Scot Sliverstein in the Institute of Healthcare Informatics , College of Information Science & Technology at Drexel University . Currently, we are working on a project regarding Healthcare IT infrastructure failures.

The purpose of our project is to investigate why Healthcare IT infrastructure may fail from a social- technologic perspective. We will collect cases published in scientific papers and newspapers, as well as unpublished stories by the Healthcare IT practitioners. Then, we will categorize these cases and build a casebase for future implementation guideline. We have already collected some cases which are listed in: http://home.aol.com/medinformaticsmd/failurecases.htm . The cases listed here were collected from 1998-2001. Now we hope to gather newly happened cases to enrich our collection. If you experienced or heard of any interesting story, please do not hesitate to write it down and send it to me. We hope to hear more from healthcare IT practitioners' hand-on experience. Any story is valuable to us. Any posting of the cases will be keep anonymous and we won't reveal identities of people or organizations .

We will move the Healthcare IT failure cases webpage to Drexel University server later, so all your stories will be listed on that site. We will inform you once the new website is ready.

If interested, please write back to me at: yunan.chen@ischool.drexel.edu . Comments and suggestions are welcome. I am looking forward to hearing back from you. Thanks very much.

Yunan Chen
Doctoral student & Research assistant
Institute of Healthcare Informatics
College of Information Science & Technology
Drexel University

It soon became apparent that a critical mass had been reached. A torrent of shared experiences of HIT difficulty in the various message boards appeared from all over the country and from other countries as well.

Calls for a formal AMIA panel or workshop on this issue arose. This call was energetically supported by a heterogeneous and widespread group of informatics professionals.

This momentum prompted the "Avoiding The F-Word" workshop to be collaboratively formulated in a wonderful volunteer effort led by former colleague Bonnie Kaplan at the Yale Center for Medical Informatics (where I'd completed my postdoc in medical informatics), and Kim Harris-Salamone, along with the AMIA workgroup members.

Ten AMIA workgroups and over fifty HIT experts ultimately participated in the workshop.

In Sept. 2006, before the workshop occurred, I wrote at "The holes in the quest to enable the electronic clinical trial ...":

... I also think some answers to the question "Where are the holes in the quest to enable the electronic clinical trial?" will be found in an upcoming Nov. 2006 American Medical Informatics Association Annual Conference workshop on healthcare IT failure entitled "Avoiding The F-Word: IT Project Morbidity, Mortality, and Immortality", the first of its kind:

SESSION DESCRIPTION

Recent studies of health care computer applications and the reported failures of well-known systems surprised the medical informatics community, leading to questions of how to increase the chances of IT systems success and the reduction of errors.

Similar problems plague a variety of different systems, whether for institutions as a whole, for ancillary services, or for consumer health, and have done so for many years. Despite an accumulation of best practices research that has identified a series of success factors, some 40% of information technology developments in a variety of sectors are either abandoned or fail, while fewer than 40% of large systems purchased from vendors meet their goals. According to the recent CHAOS Report by The Standish Group, which surveyed failures of IT in general (not just in health care), only 34% of IT projects were considered truly successful. Similar numbers have been estimated for health care, and the number has unfortunately remained approximately the same for at least the last 25 years. While there have been some published reports of failures, removals, sabotage of systems, or how failures became successes or were otherwise redefined, there has been too little opportunity to learn from studies in which technology interventions resulted in null, negative, or disappointing results.

The purpose of this session is to examine why this happens and what might be done to improve the situation, and to collaboratively develop a series of frameworks for various types of systems and healthcare settings to aid in implementation and evaluation. The session builds on a lively exchange by numerous members of a number of AMIA Working Groups concerning success and failure in medical informatics.

The session will be devoted to better defining or characterizing "success" and "failure." From there, participants will break out into smaller groups to continue the discussion, develop a set of important issues, action items, and recommendations.

The report, now published, is filled with pearls such as these:

• With the US joining other countries in national efforts towards the many benefits health information technology use can bring for health care quality and savings, recent sobering reports recall the complexity and difficulties of implementing even smaller-scale systems. Despite best practice research identifying success factors for health information technology projects, a majority, in some sense, still fail.

• With the United States Congress appropriating more than $20 billion for health information technology as part of the February, 2009 economic stimulus package, the US joined other countries in national efforts towards the many proven benefits such technology use can bring for health care quality and savings. Moreover, Medicare, along with private and commercial health plans, is implementing a new paradigm for paying for health care services in the US, known as Value-Based Purchasing (VBP), or pay for performance initiatives (P4P). These initiatives rely heavily on using electronic health records to provide clinical documentation that proves the value of their services. Tempering the fervor, though, are recent sobering reports that raise concerns about how the technology is designed and deployed.

• Similar failure rates [to other types of IT] have been reported for health IT.(14, 15) Hospitals are among those organizations where delays and cancellations of software projects are endemic.(16) For years, problems have plagued the implementation of health IT applications, whether for ancillary services, for whole institutions, for regional or national systems, or for consumers. Today's problems are reminiscent of those analyzed since at least the 1970s in classic studies of hospital information and patient record systems.(17-19) In 1990, Dowling estimated that staff interfere or sabotage with "nearly half" of projects(20), while Heeks noted in 2006 that it is his "best estimate...that most HIS [health information systems] fail in some way."(15) Recent studies and newspaper accounts cite difficulties in a variety of health information technology applications. Over the years, in many countries, patterns of severe problems repeatedly have beset a variety of efforts: hospital information systems and electronic records(21-26); ambulance services(27, 28); community, regional, and national health information networks (28-33); public health systems (34, 35); patient education (36); and physician order entry.(18, 19, 37-41)

• Yet, while there have been some published research reports of health care IT failures, unfortunately, there has been an absence of systematic and thoughtful publication of lessons learned from IT interventions where there have been null, negative, or disappointing outcomes.(27, 53) Despite calls for increased research, there are insufficient numbers of published research reports of health care IT failures, removals, sabotage of systems, or how failures became successes or were otherwise redefined. As in other sectors (69), IT-related failures in health care often are covered up, ignored, or rationalized, so mistakes are repeated.

• Participants emphasized that communication and workflow issues add to project complexity. Health care requires collaboration, as does system implementation, yet there is difficulty in translating among specialties, stakeholders, clinicians, and implementers, sometimes to the point of a seeming "culture clash."

• The workshop concluded with reports from break-out groups charged with discussing ideas for how AMIA could address health informatics failure. Break-out groups made suggestions concerning: research and publication, best practices, advocacy, education, certification, and databases and knowledge integration.

The workshop findings concluded with these words of wisdom:

Much has been learned about success and failure in IT implementation, but we need to understand more. There are legal issues when a system "fails," including just what constitutes "failure." There are social issues, ranging from how such failures affect various groups and health informatics as a whole (including possible policy and regulatory reactions), to the social aspects of what makes for a "successful" implementation. Finally there are ethical issues involved in evaluating system "success" or not sufficiently attending to previously-identified success factors and best practices.(24)

Most "failures" are failures to properly apply managerial wisdom that has been substantiated by research and experience. Perhaps the worst aspect of failure is failure to learn from past experiences so that the same issues and problems are not perpetuated.

While the workshop was a culmination of the kindling on this crucial topic I've been involved in since being overruled on patient-endangering healthcare IT mismanagement, I believe we as a society, and the HIT sector as a for-profit, entirely unregulated industry lacking accountability (the accountability rests on clinical end users, as in, in the courtroom in malpractice suits), are still very far from solutions to that final observation on failure to learn from history.

The observations of this workshop are only made more acutely important by what I believe are ill advised, rushed plans to coerce US healthcare organizations and practitioners to adopt these evolving, arguably still-experimental virtual medical devices by 2014, or face penalties.

-- SS