Good afternoon.  I am Patricia Keenan, a Program Manager at the Human Resources Research Organization, known less formally as HumRRO.  HumRRO is a non-profit, 501(c)3 research and development organization, established in 1951, that works with government agencies and other organizations to improve their effectiveness through improved human capital development and management.  

I will comment today about the Compensation and Pension (C&P) Service’s training program, as well as on methods to increase accountability and reduce rating variance.  I am the project leader for HumRRO’s work with VBA’s Skills Certification program.

Skills Certification Testing Program

HumRRO’s has worked closely with C&P Service on Skills Certification program for Veterans Service Representatives (VSRs) and Rating VSRs (RVSRs).  We began assisting VBA with the program in 2001, by developing the multiple-choice VSR test that is administered to GS-10s seeking promotion to GS-11.  In 2006 our work expanded to include VSRs at Pension Maintenance Centers (PMC) and RVSRs who have just completed training. Last year we began developing the Skills Certification test for journey-level RVSRs. 

The VSR test is completely operational with two administrations planned per year.  Development of the RVSR end-of-training test is complete and the test is expected to become operational in December of this year.  Development of the PMC VSR test is also complete and is expected to become operational next spring.  In the future, all Skills Certification tests will be administered via the internet.  This has the advantage of allowing us to expand the types of items on the test to include completion and short essay items. 

The tests were developed using a content-validation strategy, which requires that the tests reflect important job-related content.  The first step in creating that link was to conduct job analyses for each of the four target positions.  Subject matter experts linked important tasks to the knowledges, skills and abilities required to perform them.  The number of tasks linked to a knowledge area is reflected in the weight given to that knowledge area of the test blueprint and to the number of items in that area that are on the test.  To maintain the content validity of a test, it is necessary that job analyses be repeated periodically to ensure that the test still captures the important job requirements.  Because the original VSR job analysis was conducted in 2001 and many characteristics of the job have changed since then, we conducted a new job analysis this year. We are in the final stages of revising the test blueprint for the VSR test.  

C&P Training Programs

Since the Skills Certification program began in 2001, C&P Service has initiated several training programs.  Newly hired or promoted VSRs and RVSRs are required to take centralized Challenge training.  There is an RVSR Challenge course and separate VSR Challenge courses for Pre-Determination and Post-Determination.  All three Challenge courses include several Training Performance Support System (TPSS) modules.  These areweb-based training modules and electronic job aids with accompanying case-based performance practice and performance testing.  The content of the modules are content tailored to the specific position.  Experienced decision makers also use the modules for refresher training. 

VSRs and RVSRs are also required to take 80 hours of refresher training per year.  Central Office decides the appropriate content for 60% of this training (48 hours); the other 40% (32 hours) is determined by each Regional Office on an “ad hoc” basis.  This mix ensures that training addresses areas recognized as requiring additional training both nationally and locally.  Additional training may be required for several reasons – to disseminate information about new regulations or court decisions, to address areas that are commonly appealed, or to correct problems identified during Systematic Technical Accuracy Review (STAR) review.  This refresher training is provided in several ways – via classroom training, satellite broadcasts, net meetings, or online courses.  VSRs who are preparing to take the Skills Certification Test also receive additional training time to prepare for the test.  Please note that HumRRO has no role in preparing or delivering these training courses; our job is to help VBA by developing an independent assessment of knowledge necessary for the job itself.

Training and Test Results

As of August, 2008, 1,227 participants have passed the VSR Skill Certification test.  The passing rates have risen steadily from 25% in the 2003 test to 49.58% in August of this year.  These rising passing rate for the VSR test indicates to us that training is having a positive effect on test performance. 

We asked participants in the May test which TPSS modules they had completed or used as reference.  The three modules with the highest percentage of completion were Original Claim for Compensation, which157 VSRs reported completing (33.12%), Original Claim for Pension, which 127 respondents (26.7%), reported completing, and Dependency Benefits, which 118 VSRs (24.9%) reported completing.  The other modules were completed by 21% of test takers or less.  The average number of modules completed was three.  The general trend in the data shows that those who completed the TPSS modules had the highest scores, followed by those who did not use them at all (i.e., neither completed them nor used them as a reference), followed by those who used them only as a reference.  Newer employees were more likely to have completed the TPSS modules whereas employees with longer tenure were more likely to have used the TPSS modules as references.  There was no correlation between the number of TPSS modules completed and total test score.

Similarly, we asked participants in this summer’s RVSR end-of-training test which TPSS modules they completed.  Candidates reported completing an average of 6.7 out of 10 modules.  Eighty-seven percent of respondents (n=334) said they had completed Rate an Original Claim for Disability Compensation; 72.4% (n=278) reported completing Rate an Original Claim for Disability Pension.  Other modules were completed by 71% of respondents or fewer.  There correlation between the number of TPSS modules completed and test score was significant (r(384)=.17, p=.001). Slightly more than 92% of respondents (n=354) reported completing Challenge Training. 

The field test for the PMC VSRs was conducted last week.  The two TPSS modules completed by the largest percentage of the 60 participants were: Original Claim for Pension (n=24; 40%) and Income Adjustments (n=19; 31.7%). We will not be able to further analyze the data until scoring is completed in early October.   

Impediments to Rating

As part of the job analyses required for test development, HumRRO staff conducted a series of visits to regional offices to learn more about the VSR and RVSR jobs.  The main purpose of these site visits was to identify the critical job elements for incumbents.  However, participants in focus groups and interviews also provided information about other aspects of their jobs.  We talked with incumbent RVSRs, Decision Review Officers, coaches and Service Center Managers. 

The rating process is cognitively complex, requiring the RVSR to compare the facts and medical evidence presented in the claim folder to the descriptions of level of disability found in 38 CFR, encompassing two widely differing bodies of knowledge.  When the RVSR has completely reviewed the file, established that the veteran has a service-connected condition and all pertinent evidence has been included, the rater begins evaluation process.  This is done by comparing the relevant facts presented in the claims folder to the rating schedule, which is organized around 15 body systems (e.g., endocrine, musculoskeletal, respiratory, mental disorders).  Participants in the job analysis site visits cited being able to do this with relative ease is the major difference between trainee RVSRs and their journey-level counterparts.  They indicated that trainee RVSRs often struggle with anatomy and understanding medical terms at first, then realize that applying the regulations, via the rating schedule, is actually the more challenging part of the job.

The RVSR reviews the medical evidence for each separate condition being claimed and matches the condition to a diagnostic code in the rating schedule.  These diagnostic codes, in turn, are associated with descriptions for varying levels of severity of impairment.  These levels are either assigned percentages in increments of 10 on a scale from 0 to 100 or, for some disabilities such as a muscle injury, they are evaluated on more general descriptions such as severe, moderately severe, moderate, and slight.  Thus the RVSR is first required to understand the medical condition(s) (e.g., body system affected, symptoms, severity, limitations thereof) and then to match that information to the correct section(s) of the rating schedule to determine the level of disability for each condition.  In addition, there are several factors that make the task even more demanding.  These fall broadly into two categories: workload and ambiguity.

Workload Factors

One of the recurring problems discussed by incumbents was that of incomplete cases and files.  When the RVSR begins the adjudication process, the file should be ready for rating.  However, they report that often they find the case requires additional development, which the RVSR can do or defer back to the VSR.  The problems are varied and range from an incomplete exam, missing justification for diagnosis, unclear information from the veteran that must be clarified, to service records that do not show sufficient information on which to establish service-connection.  While these problems can all be resolved, doing the required additional development adds significantly to the time it takes to process a case.  The RVSR has spent time reviewing a case that was not ready for a rating decision and the decision must wait until the information can be collected. 

A second problem is the sheer volume of cases awaiting adjudication.  The backlog of work has been growing for years and is increasing more rapidly than ever with the influx of veterans from OEF/OIF.  Veterans today file claims at the time of separation and a large proportion of cases contain multiple issues.  A related factor is that often a veteran files additional claims before the first has been decided, with the result of holding up all of the veteran’s claims until they can be rated at once. 

VBA has addressed the workload problem by hiring several thousand VSRs and RVSRs in the past year.  One result of this has been that newly hired RVSRs (and VSRs) do not understand the development process well, and often spend much of their time learning what makes a case “ready to rate.” While they will eventually become proficient rating specialists, they are less efficient than they might be due to lack of understanding of this vital component of claims processing. 

RVSRs also face conflicting demands for prioritizing their work.  OIF/OEF cases are given priority, as are old cases, and those in which the veteran is facing financial hardship or a terminal illness.  Journey-level RVSRs often mentor less experienced RVSRs, reviewing their work and providing feedback, an additional duty in addition to their regular workload. 

Ambiguity

Another common theme heard during the focus groups was that it is critical for RVSRs to understand that what they are rating is most often not black-and-white.  There are gray areas in both the medical and legal aspects of the job.  A good deal of research is often required to establish service-connection, verify stressors, and understand the nature and severity of a medical condition.  RVSRs have a large number and variety of resources available to them to help gather this information, which they then compare to the regulations.  However, even their best efforts often result in having to make evaluations based on incomplete medical and legal information.

Inexperienced RVSRs usually take longer to rate a case than experienced raters.  They are less comfortable making decisions without complete information, and comparing the medical information to the regulations is not an exact science.  RVSRs become more comfortable with this ambiguity over time.  They also become more familiar with the rating schedule, so they are able to use it with more ease and become more savvy in how to use the schedule.  Comments from the focus group respondents indicate that over time, RVSRs develop individual “rules” for how to match the medical evidence to the rating schedule, many areas of which leave room for interpretation.  This is likely one of the factors that allows them to process cases more quickly; they do not have to spend as much time deciding between evaluation levels. This is also a source of rating variance. 

VBA raters select the diagnostic code so the correctness of the rating decision depends on the level of knowledge and understanding each RVSR has about the medical descriptions in the rating schedule.  The rating schedule contains over 700 diagnostic codes representing distinct physical and mental impairments that are grouped by body systems or like symptoms.  Although 700 diagnostic codes sounds like a large number, compared to the several thousands of codes contained in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), used by the medical profession, it can be assumed that the rating schedule is less detailed, and thus provides less information to guide evaluations.  Increased detail in the rating schedule would likely reduce the amount of individual interpretation that currently results in inconsistencies in rating decisions.  It would also require extensive training to learn the revised schedule and additional job aids would need to be developed to improve use of the revised schedule.

These factors make matching the medical evidence to the criteria provided in the rating schedule a challenge.  But an additional challenge is that the rating schedule does not have diagnostic codes for certain specific conditions, such as carpal tunnel syndrome, Parkinson’s disease, pacemakers, or pulmonary embolus.  When a claim includes an unlisted condition, the RVSR rates it by analogy to a closely related disease or injury.  By their nature, these “analogous codes” lack criteria for rating, so raters have to research different body systems to make the evaluation and exercise a wide range of judgments to assign analogous codes. 

A final opinion raised during the site visits was that many appeals were the result of rating decisions that did not include sufficient detail or explanation of why a claim was denied.  As just described, much individual judgment is required in the rating process.  This makes it even more important that all evidence be addressed in the rating decision.  The reasons and bases section should include all subjective and objective evidence.  It is important that veterans feel that their case has been clearly understood and evaluated.  The letter should tell them what evidence was in their service records, what the VA medical examination provided (or why a VA examination was not ordered), and all medical evidence that was submitted (e.g., private medical records).  The letter has to establish the nexus between the medical evidence and the regulations that determined the outcome, describing how service connection was established, the regulations that applied to each issue, and what evidence is needed to establish service-connection or to receive a higher level of evaluation.

Reducing Impediments

The workload of RVSRs is not going to become lighter in the near future, so easing the workload and reducing ambiguity could go a long way toward improving ratings.  We propose some suggestions to help reduce existing impediments. 

First, newly hired RVSRs should work Pre-Development for several weeks to learn the system, why different types of evidence are needed, and how to determine that a case is ready to be rated.  This would have at least two benefits.  The trainee RVSR would not spend time working a case that has insufficient evidence and the mentoring RVSR would not have to do as much explanation about the types and need for different types of evidence.  The obvious drawback to this is that it would take longer for new RVSRs to begin rating cases, but we believe having this additional knowledge would pay off in the longer term.

Second, the rating schedule is being updated, so it is probably not feasible to develop formal training for each new or revised diagnostic code.  But it is important to address the problem caused by individual interpretation of diagnostic codes that are not detailed or specific results.  A job aid that includes more specific information about the medical evidence would reduce the level of individual interpretation in ratings.  In particular, a job aid that included specific codes and descriptions for disabilities that are relatively frequent and that currently fall under analogous codes, would do a great deal for improving accuracy in these ratings.  It is much easier to identify the appropriate evaluation level when the criteria describe specific levels of disability (e.g., range of motion) rather than a more general description such as mild, moderate, and severe.  This would allow the rater to match the code to the diagnosis provided in the medical evidence, again reducing errors and variance in the award level. 

One of the commonly cited reasons for appeals is that the reasons and bases section of the decision does not provide sufficient information or an easily understandable explanation that tells veterans why a claim was denied and what they must do to have the decision reconsidered.  RVSRs use templates or scripts to outline the letter and ensure that required information for each section is included.  Merely including all required information is not the same in terms of customer service or meeting the spirit of VCAA as is a well-written letter, and these templates cannot help with the complicated problem of presenting technical information in a manner that is well organized and that is easy for the veteran to understand.  In this section of the letter, the RVSR describes the material evidence received and the level of evidence required to meet the legal standard as prescribed in the regulations.  The ability to understand, organize and clearly present all this information is difficult to train.  One remedy is to provide multiple examples of well-written letters that RVSRs can use to guide their own efforts.  A more structured approach would be to take the ability to synthesize information and present it in a well-structured, easily comprehensible document into consideration when hiring or promoting RVSRs. 

A final overall recommendation to reduce variance in ratings is to ensure that all RVSRs receive standardized training, both in content and delivery.  Some refresher training is delivered via online tools, broadcasts, or in net meetings, which do provide standardization.  A good proportion of refresher training is determined by the local office and is most often provided by Decision Review Officers.  It is important that they receive comprehensive training in the technical area being addressed, but also understand how to deliver training; a train-the-trainer workshop that teaches basic training principles as well as how to work with adult learners should be required. 

Summary

It has been HumRRO’s pleasure to work with C&P Service for the past seven years.  We are honored to be even a small part of the valuable work the Veterans Benefits Administration does for America’s veterans.  We have watched both the Skills Certification program and C&P Services Training grow over this time.  The resources and effort devoted to training have been reflected in steadily improving pass rates for the VSR Skills Certification test and in the very good pass rate for the RVSR end-of-training test. 

The greatest impediments to rating accuracy are the pressure to produce, the need for large amounts of medical knowledge and understanding, and ambiguity in interpreting the legal requirements.  Being able to spend time in Pre-Development would increase a newly hired RVSR’s understanding of the overall claims process as well as the variety and depth of development that is required to rate a case.  Job aids can neither reduce the ever increasing number of claims nor reduce their complexity, but by providing increased detail they can make the rating schedule easier to interpret and provide standardization that is currently lacking.  Writing and analytical ability were identified as key attributes of good RVSRs, but there is at present no systematic evaluation of these abilities when an individual is hired or promoted to the RVSR position.  Finally, we provide a reminder of the importance of standardized training and delivery in ensuring that all rating specialists have a common understanding and method of working.