Witness Testimony of Belinda J. Finn, Office of Inspector General, Assistant Inspector General for Auditing, U.S. Department of Veterans Affairs
Mr. Chairmen and Members of the Subcommittees, I am pleased to be here to address the Office of Inspector General’s (OIG) findings related to the Veterans Health Administration’s (VHA) reported waiting times for outpatient appointments. I am accompanied by Larry Reinkemeyer, Director of the Kansas City Audit Operations Division, who directed the teams responsible for the audits we performed. Our audit coverage on outpatient waiting times and waiting lists consists of two reports. I will discuss both reports today in order to provide a more complete assessment of the problems we identified and the current status of actions by VHA to improve outpatient waiting times.
In July 2005we issued Audit of the Veterans Health Administration’s Outpatient Scheduling Procedures (Report No. 04-02887) and concluded that schedulers were not following outpatient scheduling procedures, resulting in inaccurate waiting times and incomplete waiting lists. As a follow-up to the 2005 report, we issued Audit of the Veterans Health Administration’s Outpatient Waiting Times (Report No. 07-00616-199) in September 2007. We again concluded that schedulers were not following established procedures for making outpatient appointments, causing VHA’s reported performance on waiting times and waiting lists to be unreliable for Congressional and VA decision making.
OIG IDENTIFIES DATA INTEGRITY PROBLEMS IMPACTING THE RELIABILITY OF VHA’S WAITING TIMES INFORMATION
In FY 2005, at the request of the Secretary of Veterans Affairs, we audited VHA’s compliance with outpatient scheduling procedures to determine the accuracy of the reported veterans’ waiting times and facility waiting lists. Our objectives were to determine whether schedulers followed established procedures when selecting the type of appointment and entering the desired appointment date into the Veterans Health Information Systems and Technology Architecture (VistA) and to evaluate the effectiveness of the procedures used at VHA medical facilities to ensure all veterans either had appointments or were identified on electronic waiting lists.
Our 2005 audit work analyzed a statistical sample of 1,104 appointments from a universe of 38,786 appointments at 8 medical centers. We reviewed scheduling data and medical records to determine when the appointments were scheduled, how the schedulers created the appointments, and whether the schedulers used the correct desired dates when creating the appointments. We also reviewed each appointment to determine whether the veteran qualified for the electronic waiting list and if service-connected veterans received appointments within 30 days. In addition, we gathered information from 15,750 (53 percent) of the 29,818 schedulers nationwide on their training, experience, adequacy of supervision, and scheduling practices through a national survey. We also interviewed 247 schedulers at the 8 medical facilities visited during the audit.
VHA calculates outpatient waiting time for each appointment from the desired date of care recorded in the VistA scheduling software to the actual appointment date. The desired date of care is defined as the earliest date that the patient or clinician specifies the patient needs to be seen. In addition, VHA policy establishes a goal of scheduling appointments within 30 days of the desired appointment but not more than 4 months beyond the desired appointment date. When a specific appointment date is not requested, VHA policy requires the scheduler to use the next available appointment. VHA policy requires that all appointment requests, including consult referrals to a specialist, must be acted on by the medical facility within 7 days. Acting on the request involves either scheduling the requested care or placing the patient on the electronic waiting list. The electronic waiting list is a standard tool that VHA implemented in December 2002 to capture and track information about veterans waiting for clinic appointments in VHA medical facilities.
Our 2005 results showed that outpatient scheduling procedures needed to be improved to ensure accurate and reliable reporting of veterans’ waiting times and facility waiting lists. Because schedulers did not follow established procedures, medical facility directors were unaware that 18 percent of the service-connected veterans in our sample waited more than 30 days for their appointment. We projected that over 2,000 service-connected veterans waited longer than 30 days from their desired date to see a physician at these 8 medical facilities. Nationwide, as many as 24,463 service-connected veterans could have been similarly impacted. Inaccurate waiting time data and waiting lists can compromise VHA’s ability to assess and manage demand for medical care. VHA managers plan budget priorities, measure organizational and individual medical center directors’ performance, and determine whether strategic goals are met, in part, by reviewing data on waiting times and lists.
We found that schedulers created appointments using the wrong appointment type for 380 (34 percent) of the 1,104 appointments and the wrong desired date for 457 (41 percent) of the 1,104 appointments in our sample. When scheduling an outpatient appointment, schedulers are asked if the appointment should be considered as “next available.” If the scheduler answers yes to this question, then the system enters that date as the desired date of care by default. If the scheduler answers no to the question, then the scheduler must input a desired date of care. In 2005, VHA strived to schedule at least 90 percent of all next available appointments for veterans within 30 days. Our results showed, however, that 65 percent of the next available appointments were scheduled within 30 days—well below the VHA goal of 90 percent and the medical facilities directors’ reported accomplishment of 81 percent.
VHA medical facilities did not have effective procedures to ensure all veterans were identified on the electronic waiting lists. In fact, our testing showed that 5 medical facilities understated their waiting list by 856 veterans. Nationwide, the electronic waiting lists could be understated by as many as 10,301 veterans. We also identified clinics with substantial backlogs of consult referrals where veterans did not have appointments within 7 business days, and those veterans were not included on the electronic waiting lists. Further, 17 percent of the 247 schedulers interviewed told us they maintained informal waiting lists of veterans who needed appointments.
VHA did not have a standardized training program for schedulers so schedulers were receiving most of their training as on-the-job training. This led to inconsistencies implementing the scheduling procedures and directly contributed to the errors we identified. Forty-five percent of schedulers responding to our survey reported that they had received no formal training on the use of the VistA scheduling module, and 81 percent responded that they had received no training on the use of the electronic waiting list. Further, 2,246 (68 percent) of the 3,298 schedulers who identified themselves as trainers in our nationwide survey, did not know how to correctly create an appointment for a veteran who wanted an appointment as soon as possible but who did not need urgent or emergency care. Seven percent of schedulers said that managers or supervisors directed or encouraged them to schedule appointments contrary to established procedures. Sixteen percent of the schedulers reported that they maintained informal waiting lists.
We recommended that the Under Secretary for Health take the following actions to improve outpatient scheduling procedures and the data integrity of waiting time information:
- Ensure that medical facility managers require schedulers to create appointments following established procedures.
- Monitor the schedulers’ use of correct procedures when creating appointments.
- Monitor consult referrals to ensure that all veterans with referrals either have scheduled appointments within 7 business days or are included on electronic waiting lists.
- Establish an automated link from the Computerized Patient Record System consult package to the Vista scheduling module.
- Ensure medical facilities prohibit the use of informal waiting lists.
- Develop a standard training package for medical facilities to train schedulers on electronic waiting list and VistA scheduling modules.
- Ensure all schedulers view the video training titled “Vista Scheduling Software: Making a Difference.”
- Require all schedulers to receive annual training on the electronic waiting list and VistA scheduling module.
The Under Secretary for Health agreed with the findings and recommendations to make the needed improvements in outpatient scheduling. According to the Under Secretary, VHA was vigorously addressing problems with waiting times and scheduling delays, and they had taken steps to accurately quantify the numbers of patients on waiting lists, lengths of waits, and the reasons for the scheduling delays. The Under Secretary also stated that VHA’s Advanced Clinic Access (ACA) initiative, in conjunction with other planned and ongoing improvements, was expected to result in needed scheduling enhancements that are consistently applied to all VHA medical facilities. While we did not evaluate the implementation of the ACA initiative, our 2005 results showed that the schedulers’ use of incorrect procedures distorted the reported measurement of veterans’ waiting times and facility waiting lists, regardless of whether the clinic had implemented ACA.
In response to our 2005 report, VHA issued new policy, Directive 2006-055, on October 11, 2006, for implementing processes and procedures for the scheduling of outpatient appointments and for ensuring the competency of staff involved in any or all components of the scheduling process. VHA’s directive also addressed the VA medical facilities’ responsibilities relating to recall, reminder systems, and other forms of patient-driven scheduling, noting that facilities must ensure that the patient entitled to priority access is given an appointment in 30 days and all others within 120 days. The facility retains principal responsibility for providing the patient an appointment to be seen within the appropriate timeframes. VHA policy further extends the facility’s responsibility to call and/or send a reminder letter and to make available a scheduled appointment for the patient to be seen within 30 days of the originally specified desired date for patients entitled to priority access or 120 days for all others.
OIG FOLLOW-UP REVIEW SHOWS VHA’S OUTPATIENT WAITING TIMES INFORMATION STILL HAS DATA INTEGRITY PROBLEMS
In November 2006, we received a Congressional request to follow up on the patient waiting times issue to determine if VHA had improved their practices and procedures related to outpatient scheduling. The objectives of our review, completed in 2007, were essentially the same as our 2005 review, except that we also assessed whether the 2005 audit report recommendations were fully implemented.
During our follow-up review, we determined whether established scheduling procedures were followed, outpatient waiting times reported by VHA were accurate, and electronic waiting lists were complete. We visited 10 medical facilities, testing 700 appointments. A key point of our methodology was that we reviewed appointments that VHA had reported as being completed in 30 days. We also tested 300 consult referrals to assess the accuracy of the consult tracking report because medical facility personnel said that clinic personnel did not always update the report after action was taken. VHA includes and relies upon this same information in its performance and accountability reporting measure. At the time of our review, these 10 facilities listed over 70,000 consult referrals that were over 7 days old on the consult tracking reports.
The review showed that many of the data integrity weaknesses reported in 2005 were still impacting the reliability of patient waiting times and that schedulers were not following established procedures for making or recording medical appointments. We concluded that the accuracy of VHA’s reported waiting times could not be relied on and the electronic waiting lists at the 10 medical facilities were not complete.
In reviewing each appointment, we researched the medical records to find out when the referring medical provider had recommended that the patient receive an appointment and compared the recommended date of care to the actual appointment. We found unexplained differences between the desired dates as shown in the scheduling system and used by VHA to calculate and report waiting times, as compared to the desired dates recommended by the medical provider and indicated in the patients’ related medical records. In a few appointments, schedulers had annotated the appointment records to indicate when a patient requested a specific date and we used that date to calculate the waiting time. Our review also found instances where medical providers had suggested a range of time, such as 4 to 6 months, instead of a specific date for care. In those cases, we followed the guidance in VHA’s scheduling directives and used the earliest point of the time range as the desired date of care.
Our review of 700 appointments provided us with reasonable assurance to conclude that schedulers were not recording either accurate, complete, or in some cases any information to support the desired date of care used to compute the reported waiting time. Overall, we found evidence to support that only 524 (75 percent) of the 700 appointments that VHA reported as having completed within 30 days actually were. The 700 appointments that had occurred within 30 days included 78 percent of the primary care appointments and 73 percent of specialty care appointments. As a result, VHA’s reported waiting times are not based on accurate and complete information. For example, on December 20, 2005, a veteran who was 50 percent service-connected was seen in an Eye Clinic. The provider wrote in the progress note that the veteran should return to the clinic in 6 weeks (January 31, 2006). On September 6, the scheduler created an appointment for the veteran on October 17. The scheduler entered a desired date of October 2, which resulted in a reported waiting time of 15 days. However, based on the provider’s desired date of January 31, the veteran actually waited 259 days for his appointment. The scheduling records did not contain any explanation for the delay. Medical facility personnel told us the reason this appointment took so long to schedule was because it “fell through the cracks.”
In order to validate our results at each medical facility, we provided our case review findings to the local medical facility personnel responsible for scheduling. In response, our findings were validated as being accurate and supportable. Concern over our methodology did not become an issue until the draft report was presented to VHA senior management. VHA non-concurred with this finding and told us that even though schedulers did not document it, the unexplained differences between the date recommended by the medical provider and the date shown in the scheduling system can be attributed to patient preferences for a specific appointment date. VHA directives require schedulers to annotate appointment records to indicate patient requests for specific appointment dates. VHA personnel told us that schedulers often do not document patient preferences due to high workload; and that this documentation only serves to support audit requirements. We contend that this basic annotation is critical to the integrity of reported waiting times information. To accept an assumption that every patient requested a desired date different than the documented desired date shown in the medical records would be irresponsible and contrary to VHA’s own directives. We would agree that some of the date differences we identified in appointment information could possibly be due to patient preferences that were not documented by schedulers. However, in the absence of specific information, neither we nor VHA can be sure whether patient preference or the scheduler’s use of inappropriate scheduling procedures contributed to the 25 percent error rate we found.
VHA’S ELECTRONIC WAITING LISTS CONTINUE TO BE INCOMPLETE
VHA’s policy prohibits schedulers from making appointments for veterans that exceed the 30- or 120-day requirement and the policy requires that those veterans be placed on the electronic waiting list immediately. Our review identified 64 veterans (9 percent of the total appointments reviewed) who should have been on the 30-day electronic waiting list and were not.
Additionally, VHA’s consult tracking report identified over 70,000 veterans with consult referrals over 7 days old that—in accordance with VHA policy—should have been on the waiting list of the 10 facilities we reviewed. Our review of 300 consult referrals found that 183 (61 percent) of the associated veterans should have been on the waiting list and more than half of those had been waiting more than 30 days. The remaining referrals had already been acted on, but facility personnel had not updated the records to reflect the true status (for example, completed or discontinued). The lack of action on consults may lead to situations such as the following one highlighted in our 2007 report:
· On April 18, 2006, a veteran who was 80 percent service-connected, including service- connected for hearing impairment, was referred to the Audiology Clinic. Because this was a consult referral, the veteran should have received the next available appointment. On September 20 (155 days after the referral), the scheduler created an appointment for the veteran for October 20 and entered the desired date of September 20, which resulted in a reported waiting time of 30 days. However, based on the provider’s desired date of April 18, the veteran actually waited 185 days for his appointment. The scheduling records did not contain any explanation for the delay. Medical facility personnel agreed with our recalculated waiting time.
At the time of our review, the 10 facilities had reported only 2,600 veterans on the waiting list. In 2007, we found that schedulers at some facilities interpreted guidance from their managers to reduce waiting times as instructions never to put patients on the electronic waiting list. This seems to have resulted in some “gaming” of the scheduling process. Medical center directors told us their guidance was intended to ensure patients received their appointments timely and did not need to be on the electronic waiting lists.
Low priority for training schedulers continues to affect the accuracy of waiting times and completeness of waiting lists. Schedulers and managers told us that, although training is readily available, they were short of staff and did not have time to take the training. In 2007, 47 percent of the schedulers we interviewed reported they had no training on consults within the last year, and 53 percent had no training on the electronic waiting list within the last year.
PRIOR OIG RECOMMENDATIONS REMAIN OPEN
Outpatient waiting times continue to be inaccurate and waiting lists continue to be incomplete because management has not yet effectively implemented our recommendations. Almost 3 years later, five of the eight recommendations remain open, which included one recommendation that was reopened based on the findings in our 2007 report. Specifically, actions taken by VHA with respect to one of the previously closed recommendations proved ineffective in monitoring schedulers’ use of correct procedures when making appointments.
We believe that the most important recommendations from our two reports concern the need for VHA management to monitor how schedulers perform and routinely test the accuracy of reported waiting times and completeness of electronic waiting lists. In our opinion, these are critical quality assurance steps that are necessary to ensure that the VistA system contains complete and accurate information on waiting times.
In addition to monitoring the accuracy of information, management needs to take corrective action when testing shows questionable differences between the desired dates of care shown in medical records and those documented in the VistA scheduling package. To date, VHA has not implemented an effective process to routinely test the accuracy of reported waiting times and the completeness of electronic waiting lists.
The findings in our 2005 and 2007 reports demonstrate that the data recorded in VistA and used to calculate veteran outpatient waiting times is not reliable. It is our position that until VHA establishes procedures to ensure that schedulers comply with policy and document the correct desired dates of care, whether recommended by medical providers or requested by veterans, that calculations of waiting times using VHA’s current system will remain inaccurate.
2007 FOLLOW-UP REVIEW LEADS TO FIVE OIG RECOMMENDATIONS
Based upon our follow-up work, we recommended that the Under Secretary implement the following recommendations:
- Establish procedures to routinely test the accuracy of reported waiting times and completeness of electronic waiting lists, and take corrective action when testing shows questionable differences between the desired dates of care shown in medical records and documented in the VistA scheduling package.
- Ensure schedulers comply with the policy to create appointments within 7 days or revert back to calculating the waiting times of new patients based on the desired date of care.
- Amend VHA Directive 2006-055 to clarify specialty clinic procedures and requirements for receiving and processing pending and active consults to ensure they are acted on in a timely manner and, if not, are placed on the electronic waiting lists.
- Ensure all schedulers receive required annual training.
- Identify and assess the alternatives to the current process of scheduling appointments and recording and reporting waiting times, and develop a plan to implement changes to the current process.
VHA RESPONSE TO LATEST OIG REPORT AND OIG REBUTTAL
The Under Secretary for Health agreed that our report correctly identifies areas VHA needs to address to improve outpatient waiting times accuracy. The Under Secretary acknowledged that our report highlights many of the roadblocks VHA faces making improvements in wait times. However, VHA took exception to the findings on the wait times because of their perceived limitations of our review methodology.
The Under Secretary stated that one of the most valid measurements VHA has relating to access efficiency is generated directly from a patient satisfaction survey of the veterans seeking health care services and noted that 85 percent of the veterans who completed the survey reported that they had access to primary care appointments when they needed them, and 81 percent of these same veterans also reported satisfaction with timely access to specialty care.
We see no valid basis for comparison between the results of the patient satisfaction survey and the results of our audit. Further, there is no basis for comparing overall patient satisfaction and VA’s compliance with specific policy requirements, or the accuracy of waiting time information reported by VHA. We also noted that waiting time information reported by VHA was obtained by the same data system that the OIG used to conduct the audit, not from patient satisfaction survey. To support any level of comparison, the patient satisfaction survey would have to ask veterans whether they were seen in the 30-day requirement.
In addition, the patient satisfaction results do not support the results VHA reported to Congress in November 2006. VHA reported that 96 percent of all veterans seeking primary care and 95 percent seeking specialty care were seen within the 30-day standard. Only 85 percent of the veterans who responded to the survey reported satisfaction with access to primary care and only 81 percent were satisfied with timely access to specialty care. These percentages are closer to the results in our audit, which were 78 percent and 73 percent, respectively. Our results are accurate, well-documented, and based on all available VA information.
We also disagree with the Under Secretary’s statement that during our review we did not consider a patient’s preference for a specific date other than what the medical provider requested. We accepted schedulers’ comments on specific date requests as evidence of patient preference, but we cannot accept a blanket statement that all differences are due to patient preference. Although the Under Secretary stated that it is unrealistic to expect schedulers to document patient requests due to workload demands, we noted that scheduling directives contain numerous requirements for documentation of patient requests and actions.
While we recognize that ensuring scheduling information nationwide has its challenges, both the 2005 and 2007 OIG reviews showed that schedulers were not following VHA’s policies and procedures to record the correct desired date of care. Further, the findings in our reports do support the fact that data recorded in VistA and used to calculate veteran outpatient waiting times is not reliable. Until VHA establishes procedures to ensure that schedulers comply with policy and document the correct desired dates of care, whether recommended by medical providers or requested by veterans, calculations of waiting times from the current system will remain inaccurate.
Our follow-up review results showed that VHA has not taken timely action to implement five of the eight recommendations in our 2005 report, and the Under Secretary for Health, by his own admission said the system information is inaccurate in that it does not always document patient preference for a specific date. We find it contrary for VHA to state their agreement with the findings and recommendations in our 2005 report and then to disagree with our follow-up report that found a continuation of the same problems — problems that could have been resolved had VHA implemented the recommendations in our 2005 report. In fact, VHA’s response to our 2007 report concedes the failure of scheduling clerks to adequately document patient preferences in appointment dates. Both reports demonstrated and supported the fact that the system is not accurate and therefore should not be relied on as an accurate source for reporting waiting times to Congress.
PERFORMANCE AND ACCOUNTABILITY REPORTING
VA’s FY 2006 Performance and Accountability Report prominently reported that 96 percent of primary care outpatient appointments and 95 percent of specialty care outpatient appointments were scheduled within 30 days. We cannot compare this performance measure to the results of our latest audit because we selected appointments from a different timeframe. In FY 2007, VA reported that 97.2 percent of primary care appointments and 95 percent of specialty care appointments were scheduled within 30 days. We took great exception to VA’s reporting of this performance measure because our audit clearly showed significant issues with the integrity of data being used to formulate these performance measures. Although VA has continued to report these measures, they added a footnote acknowledging our reports.
Long-term fixes and changes to the scheduling system may take years to implement; however, in the meantime VHA needs to address the data integrity issues associated with its scheduling system and ensure accuracy in its current system. In addition, VHA needs to ensure scheduling procedures are followed and implemented consistent with its own policies. It is problematic when VHA continues to report waiting time information to Congress that was knowingly derived from a system that contains inaccurate and incomplete data. Debating the differences between our reported error rate and VHA’s reported waiting times would only serve to overshadow the primary point of both audit reports, which is that the data in VHA’s scheduling system is inaccurate. Our concern is that VA and Congress not only have accurate and reliable information for budgeting, assessing, and managing the demand for care but, more importantly, for ensuring no veteran falls through the cracks and fails to receive timely medical care.
Mr. Chairmen, that concludes my remarks and thank you once again for the opportunity to discuss this important issue. Mr. Reinkemeyer and I would be pleased to answer any questions.