Mobile Menu - OpenMobile Menu - Closed

Bryan D. Volpp, M.D.

Bryan D. Volpp, M.D., Veterans Affairs Northern California Healthcare System, Associate Chief of Staff, Clinical Informatics, Veterans Health Administration, U.S. Department of Veterans Affairs

Good morning Mr. Chairman and members of the Committee.  Thank you for this opportunity to discuss the impact on patient care due to the disruption to the VISTA and Computerized Patient Record System (CPRS) at the VA Northern California Healthcare System (VA NHCHCS).  The VA NCHCS is an integrated health care delivery system serving more 377,700 veterans dispersed over a wide area covering ten geographic sites.  We serve approximately 70,000 unique veterans per year and average close to 2000 visits per day.  VA NCHCS offers a comprehensive array of medical, surgical, rehabilitative, primary, mental health and extended care to veterans in Northern California.    In addition, we provide inpatient acute and critical care services at the Sacramento site (50 beds) and inpatient nursing home and subacute care (115 beds) at the Martinez site.  

Disruption to VISTA and CPRS

On August 31, 2007, at approximately 7:30 am on Friday, VA NCHCS experienced a major disruption with the logons to our VistA and CPRS.   The disruption resulted from a problem at the Sacramento Regional Data Processing Center (SRDPC) and affected 17 sites within VA NCHCS. 

Contingency Plan for Disruptions

VA NCHCS immediately implemented our local contingency plan for failure, which consists of three backup levels.  The first level backup is a switch over from the Sacramento Data Center to the Denver Data Center.  The second level backup is a read-only version of the patient data.  And the final level of backup is a set of files stored on some local PCs that contains brief summaries of a subset of the patient data for patients who are current inpatients or who have appointments in the next two days.  A key element in our contingency plan is that communication to the users on the cause and an estimate of length of the downtime are to be made on a regular basis by IRM.  This did not occur.

The contingency plans failed to stop the disruption.  The switch over to the Denver Data Center did not occur.  The read-only backup of the patient data had been made unavailable earlier in the week of August 31 in order for the Regional Data Center staff to create a new version of our test account.  Test accounts are required to be refreshed every 4-6 months at all VA sites.   With failure of the first two backup levels, we became reliant on the data stored on several local personal computers that could be printed.  The data stored on the personal computers are health summaries.  Health summaries are brief extracts of the record for patients with scheduled appointments which contain recent labs, medication lists, problem lists and recent notes along with allergies and a few other elements of the patient record.  The disruption severely interfered with our normal operation, particularly with inpatient and outpatient care, and pharmacy.

Disruption Impact Inpatient Care

The inpatient sites were immediately affected.  The residents on rounds in all the impacted facilities were not able to access patient charts to review the prior day's results, add or review orders.   Nursing reports were interrupted because some of the handoffs from one shift to the next are done by reviewing activities and progress in the electronic record.  Discharge planning for that morning was interrupted as well due to lack of electronic record availability.  On the inpatient wards, there were many delays in medication administration and in discharges.  The delays included the following:

  • The medical staff was forced to write discharge instructions and notes on paper.
  • The electronic lists of instructions and of medications were not available for the patients being discharged.
  • Patients being discharged could not be given follow-up appointments at the time of discharge.  The appointments had to be made later and the patient notified by phone.
  • There were delays in obtaining discharge medications and patients remained on the wards longer than would normally be required. 
  • The nurses administered medications to the patients and used the paper MAR to record the administration events.  Initial medication passes were interrupted and delayed until the paper copies of the Medication Administration Record (MAR) could be printed.

The use of the paper MAR continued well after the system came back up at around 4 pm.  This occurred because there was a delay in the automated updating all the medications with new orders and changes.  Until both Pharmacy and Nursing can verify that that the electronic lists have been updated and are accurate, the electronic MAR cannot be used.  One inpatient did not meet inpatient criteria but could not be transferred to the nursing home since adequate records were not available.  The patient stayed an extra four days and required an additional nurse to stay in his room as a sitter until he could be transferred.

Disruption Impact on Outpatient Care

Outpatient activities were impacted within a few minutes after the outage.  Although most clinics did not have scheduled patients until 8:00 am, many providers who were beginning to prepare for clinic were affected almost immediately.  Consent forms that had been done previously for scheduled surgery and for other procedures were not available since these are all done electronically.  The providers with patient appointments early in the morning had no medical records to use for these patients.  For many of the patients, a medication list was available on paper but the paper health summary backups had not yet been printed.  We began to instruct the users to print the paper health summaries for use in the clinics and on the wards just after 8:00 am.  These were distributed as quickly as possible but for patients with appointments at 8:00 am to 9:00 am, very few of these summaries were available in time to provide the needed information to the provider while seeing the patient.

Disruption impact on Pharmacy

The pharmacy quickly became overloaded with prescriptions that they were attempting to fill for patients.  The labeling equipment and automated dispensing equipment, both linked to VistA, were unavailable.  The pharmacy began to ask patients if they could wait to have the prescriptions mailed.  This problem was made more difficult by the fact that Monday, September 3, 2007, was Labor Day and the next transmission to the Centralized Mail Out Pharmacy (CMOP) would be on Tuesday, September 4, 2007.  In addition, the transmission to the CMOP for August 31, 2007 was scheduled for 8:00 am. This also caused a delay in patients receiving medications.  The prescriptions entries completed on August 30, 2007 by the pharmacy were not received at the CMOP for fulfillment until September 4, 2007.

Other Impacts Resulting From the Disruption

The local health summaries for patients were printed in all clinic areas and on the wards which essentially created a temporary patient record.  After 2 hours, most users began to record their documentation on paper.  For example:

  • Paper order forms were distributed and orders were being faxed to Pharmacy and Radiology for inpatients and outpatients.
  • Paper prescriptions were written for outpatients.
  • Laboratory orders were written on paper and patients sent to the lab with paper copies of orders. 
  • Multiple patients who had planned CT scans and who needed a measure of kidney function prior to the procedures had to have their blood redrawn since the prior results were not available.
  • Consent forms were done on paper.
  • Vital signs and screenings for depression, post-traumatic stress disorder (PTSD) and other interventions were recorded on paper.
  • The cardiologists could not read any of the EKGs that had been done prior to the failure since these had not been printed and are usually reviewed and interpreted online. 
  • Surgeons could not enter their operative notes in to the surgery package.
  • Consults could neither be ordered or responded to or even updated. 
  • Appointments could not be made and, if a patient cancelled, there was no way to identify other patients to fill those slots. 

Although the paper health summaries were available for patients with scheduled appointments, there were no records at all available for patients who came to Urgent Care or to the Sacramento ER or walk-in patients at any of the clinics.

Prior Computer Failures

Although we have had brief periods of scheduled and occasionally unscheduled computer failure in the past, many of these were isolated to one site or one building and none lasted as long as the disruption experienced on August 31, 2007.    Our contingency plans had been implemented successfully as drills during many of these periods.  During prior outages, the local IT staff had always been very forthcoming with information on the progress of the failure and estimated length even in the face of minimal or no knowledge of the cause.  To my knowledge, this was absent during the most recent outage. 

Disruption Recovery

Once the disruption was resolved, a tremendous amount of work was undertaken to restore the integrity of the electronic record.  Laboratory and pharmacy staff worked late that Friday night and over the weekend to update the results and orders in the electronic record and to enter all the new orders and outpatient prescriptions.  Complete recovery in the pharmacy took over a week.  Administrative staff worked for over two weeks to complete the checkouts on all the patients who were seen that day.  However, entering checkout data on all these patients many days after the fact is potentially inaccurate.  Many providers have gone back into CPRS and tried to reconstruct notes that summarize the paper notes that they wrote in order to mitigate the risk of missing information.

This work to recover the integrity of the medical record will continue for many months since so much information was recorded on paper that day.  When you consider that hundreds of screening exams for PTSD, depression, alcohol use, and smoking, and entry of educational interventions, records of outside results, discharge instructions and assessments are all now on paper and are not in a format that is easily found in the electronic record, the burden of this one failure will persist for a long time.  This adds an additional load for the staff to have to pull up the paper records from that day and presents a risk that some important facts or results collected on that day will be missed at some point in the future.  For example, consent forms done that day for future procedures will not be in the same location as our usual consent forms since these were done on paper and scanned into the record during recovery.

In summary, there were severe impacts to patient care, timeliness of care and the integrity of the medical record due to the disruption and these affects will persist for some period of time into the future.  Mr. Chairman, this concludes my statement.