While behind-the-wheel and laboratory/classroom assessments to evaluate fitness to drive are prevalent and have been performed for more than 70 years, virtual-reality driving simulators have remained limited in use. Despite this, simulators offer great potential in this regard. Driving simulators provide a safe environment with controlled variables under which all participants can be assessed using the same conditions.
In what ways can driving simulators be used as assessment tools to evaluate fitness to drive, and what validity evidence has been demonstrated by the scientific literature?
I decided to look at how driving simulators are being used to assess fitness to drive in populations with cognitive disorders, including those suffering from traumatic brain injury or stroke. I included an article investigating cognitive impairment and fitness to drive issues concerning subjects with sleep apnea and one dealing with combat veterans. These populations reflect my current client pool as a driving rehabilitation specialist.
A 2013 study conducted by researchers at several Boston hospitals and universities examined the driving performance of post-deployment combat veterans using a driving simulator (Amick, et al, 2013). Vehicle crashes are the most common cause of death for combat veterans in the first few years following deployment (p.463). The researchers compared a group of 25 combat veterans with a control group of 25 non-veterans using a three-screen driving simulator with an unspecified field of view. The simulator logs driver errors as the driver performs a variety of simulated tasks. They found that the combat veterans demonstrated “significantly more total errors” (p.467) than the control group. Via a driving history questionnaire, they also discovered “significant correlations” between committed errors on the driving simulator and “lifetime traffic warnings” and “lifetime tickets” (p.467).
A 2010 research study conducted at the Centre for Research on Safe Driving at Lakehead University in Ontario examined the psychometric properties of validity and reproducibility of results of simulator protocols in the assessment of fitness to drive (Bedard, et al, 2010). A sample group of 15 males and females with a median age of 20 years was administered standard neuropsychological tests including Trails Making A & B and the Useful Field of View (UFOV). A second group consisting of 38 subjects ranging in age from 18 to 83 completed Trails A, UFOV and a simulator session. These subjects were also given an on-road driving evaluation. Researchers used an STISIM simulator with a three-screen field of view of 135 degrees. In both the on-road and simulator session, mistakes were recorded, including “center-line crossing, road edge excursion, failure to stop at a stop sign or red light, speeding (>2.5 mph over the speed limit), illegal turns, off-road crashes, and vehicle collisions” (p.337). The researchers found a .74 correlation between the on-road and simulator assessments, and intraclass correlation coefficients testing reproducibility “ranged from .73 to .87”, leading them to conclude that “simulators could be used to facilitate the assessment of fitness to drive” (p.336).
A 2011 study looks at work performed at several research facilities in Quebec, Canada concerning the effort to improve the fitness to drive of a 23-year-old female with traumatic brain injury utilizing a driving simulator (Gamache, et al, 2011). Researchers had the patient perform a total of 25 simulated drives over a period of a few months, with each drive taking approximately one hour to complete. While the focus of this research was on the driving simulator as a training tool, they did examine two simulator measures in terms of driving performance: time-on-task as indicative of adopted driving strategy and her involvement in crashes within the simulator (p.427).
A 2010 research study conducted by Lesa Hoffman at the University of Nebraska and Joan McDowd at the University of Kansas Medical Center compared driver performance in a driving simulator with the same driver’s history of crashes over a subsequent 5-year period to determine whether performance in a simulator could accurately predict future potential to cause a motor vehicle crash (Hoffman & McDowd, 2010). Of the 152 initial subjects (male and female age 63 and older) who had performed driving tasks in a simulator, the researchers were able to perform follow-up interviews with 114 of them (p.742), and found “that older adults who initially demonstrated greater impairment in a low-fidelity driving simulator were significantly more likely to have reported an at least partially at fault accident in the subsequent follow-up period” (p.744).
A study published in 2005 by researchers in California health care system sought to assess the predictive validity of the use of a driving simulator for patients with traumatic brain injuries (Lew, et al, 2005). The study examined 11 patients with “moderate to severe” brain injury, compared with a control group of 16 healthy subjects. The simulator drive time was approximately 30 minutes, during which a Simulator Performance Index (SPI) of 12 different measures was recorded (p.177). Patients also performed an on-road assessment with a trained evaluator. Ten months later, patients were rated by family members on their fitness to drive using the Driver Performance Inventory (DPI) during a period of 4 weeks of observed driving of at least 3 hours. The researchers found a greater correlation between the automated simulator assessment and future driving behavior than with the on-road assessment.
A 2000 study conducted by researchers at a university hospital in Sweden examined the predictive validity of a neuropsychological battery of tests, which included performance in a driving simulator, concerning fitness to drive in older drivers recovering from a stroke (Lundqvist, et al, 2000). They found that the control group performed better during a divided attention task in the driving simulator than did the patient group, and overall concluded that “the simulator variables could classify 85% of the subjects correctly with respect to overall driving skill with Complex Reaction Time as the significant variable” (p.145).
My research turned up two articles by A. Patomella and colleagues at facilities in Stockholm, Sweden, both concerning the validity of a driving simulator measurement tool called P-Drive. In the earlier research, the simulator, consisting of three screens with a field of vision of 135 degrees, was used by 31 patients suffering from brain injuries (Patomella, et al, 2004). The researchers used a Rasch measurement model to validate the ordinal data from the simulator and, based upon general “goodness of fit”, concluded that “the items in P-Drive suggested preliminary overall internal scale validity” (p.73). In the second article, the researchers used the P-Drive measurement tool on 101 stroke patients who performed driving tasks in the simulator (Patomella, et al, 2006). Again, the researchers concluded that P-Drive results suggest internal scale validity, as well as person response validity.
A 1996 study published by university researchers from Montreal, Canada and The Netherlands compared performance on a driving simulator of 20 patients with closed head injury (CHI) with 20 healthy subjects (Schmidt, et al, 1996). The simulator used here was a single-screen model, and lane-tracking ability was the only driving performance skill measured. The patient group performed “significantly worse” than the control group, demonstrating “clear perceptual-motor slowness in CHI patients” (p.161).
A 2007 study by a researcher at the University of Iowa observed driver performance of subjects suffering from Obstructive Sleep Apnea Syndrome (OSAS) in a simulator consisting of three screens with a 150-degree field of vision (Tippin, 2007). A total of 24 subjects drove the simulator under conditions designed to encourage “microsleep” episodes, and their microsleep performance was compared with their wakeful performance, as measured by an EEG. The research discovered an increase in steering and lane position variability that worsened the longer the microsleep continued.
A 2000 study published by researchers in British Columbia, Canada, examined the driving performance of 28 adults with brain injuries in a driving simulator (Wald, et al, 2000). The driving simulator used a head-mounted display (HMD) with a horizontal field of view of 30 degrees. The researchers found concurrent validity between the simulator measures and on-road assessments.
While each of the studies utilized a different model of driving simulator, there were many similarities. All of the studies included in this literature review used, as a neuropsychological assessment instrument, some form of virtual-reality driving simulation based in a computer and utilizing a steering wheel, accelerator pedal and brake pedal. Many of the simulators used a three-screen configuration to increase field of vision (Amick, et al, 2013; Bedard, et al, 2010; Patomella, et al, 2004; Patomella, et al, 2006; Tippin, 2007).
All of the studies in this literature review attempted to discern the impact of impaired cognitive functioning on fitness to drive. All but two studied individuals with traumatic brain injury, including those resultant from stroke. One (Amick, et al, 2013) considered impaired cognitive functioning as a result of combat-caused Post-Traumatic Stress Disorder (PTSD), while another (Tippin, 2007) considered impaired cognitive functioning as a result of Obstructive Sleep Apnea Syndrome (OSAS). These two are included here as they offer the possibility of expansion of the potential population to which simulator assessment may be viable to individuals with cognitive impairments not caused by brain damage.
Efforts were undertaken to establish concurrent validity in eight of the ten research studies (Amick, et al, 2013; Bedard, et al, 2010; Hoffman, et al, 2010; Lew, et al, 2005; Lundqvist, et al, 2000; Schmidt, et al, 1996; Tippin, 2007; Wald, et al, 2000). Of these, the Trails-Making Tests (Trails A and B) were used most often (Bedard, et al, 2010; Lundqvist, et al, 2000; Schmidt, et al, 1996; Wald, et al, 2000).
According to the Standards for Educational and Psychological Testing issued by the American Educational Research Association, “validation logically begins with an explicit statement of the proposed interpretation of test scores, along with a rationale for the relevance of the interpretation to the proposed use. The proposed interpretation refers to the construct or concepts the test is intended to measure”.
Concurrent validity was also measured in comparison with an on-road driving assessment in only three of the studies (Bedard, et al, 2010; Lew, et al, 2005; Wald, et al, 2000). One study found the simulator to be a superior assessment tool to on-road driving assessments as a safer, easier and more cost-effective alternative to collect the same data (Bedard, et al, 2010). Another study (Lew, et al, 2005) found no correlation between performance in the driving simulator and performance in an on-road assessment. The third study measured relatively low levels of correlation between performance in the driving simulator and the on-road assessment for the same individual (Wald, et al, 2000).
Two of the studies (Amick, et al, 2013; Hoffman, et al, 2010) attempted to assess concurrent validity with driving history questionnaires. One “observed significant correlations between the frequency of errors on the driving simulator and lifetime traffic warnings (r(50) = 0.35, p = 0.01) as well as lifetime tickets (r(50) = 0.43, p < 0.002)” (Amick, et al, 2013). The other “demonstrated that performance in a low-level driving simulator could significantly predict self-reported automobile accidents 5 years later in a sample of 114 older adults, lending evidence of external validity of such measures” (Hoffman, et al, 2010).
Of the driving skills measured, eight of the ten studies examined speed management as a critical factor of fitness to drive (Amick, et al, 2013; Bedard, et al, 2010; Gamache, et al, 2011; Hoffman, et al, 2010; Lew, et al, 2005; Lundqvist, et al, 2000; Patomella, et al, 2004; Patomella, et al, 2006; Tippin, 2007). One study (Amick, et al, 2013) found that their patient group “demonstrated significantly more speeding errors than the civilian control group (F(1,46) = 11.51, p < 0.01, η2 = 0.19)”.
Six of the studies (Gamache, et al, 2011; Hoffman, et al, 2010; Lew, et al, 2005; Lundqvist, et al, 2000; Patomella, et al, 2004; Patomella, et al, 2006; Wald, et al, 2000) involved measurement of driver performance during some sort of divided attention challenge, usually involving audio distraction, to assess cognitive load performance as an assessment of fitness to drive. Five of the studies (Bedard, et al, 2010; Hoffman, et al, 2010; Lew, et al, 2005; Lundqvist, et al, 2000; Wald, et al, 2000) explicitly measured crashes and collisions occurring in the simulator sessions as a factor in determining fitness to drive.
Only four of the studies (Amick, et al, 2013; Bedard, et al, 2010; Hoffman, et al, 2010; Wald, et al, 2000) measured the individual’s attention and adherence to traffic control devices (TCD), including stop signs and traffic lights. Additional performance traits measured included merging (Amick, et al, 2013), course completion time (Hoffman, et al, 2010), and space management / follow distance (Lundqvist, et al, 2000; Wald, et al, 2000).
There are four basic road environments available in most driving simulators: Residential, Urban, Highway and Rural. Residential environments are generally slow speed (25MPH) and feature housing, driveways, two-lane road environments and stop signs. Urban environments typically have similar or slightly faster speed limits, but increase complexity and density of threats and replace stop signs with traffic lights. Highway environments are fast (55MPH-65MPH and can include opportunities to gauge merging and lane changes. Rural environments are generally mid-speed (40-55MPH) two-lane roads with curves, possible hills and variable levels of traffic density.
Assessment over multiple driving environments was the preferred method in only four of the studies (Amick, et al, 2013; Gamache, et al, 2011; Lew, et al, 2005; Wald, et al, 2000). Of those studies that chose to measure driver performance and evaluate fitness to drive using only one type of driving environment, two selected rural roads (Lundqvist, et al, 2000; Tippin, 2007) and one selected to assess only in an urban environment (Bedard, et al, 2010). The actual driving environment chosen to perform the assessment was unspecified in three studies (Hoffman, et al, 2010; Patomella, et al, 2004; Patomella, et al, 2006; Schmidt, et al, 1996).
Many of the studies discussed the fact that, while simulators offer many benefits, including realistic driving in a safe environment with automated scoring and controlled, reproducible variable events, there are currently no universally-accepted protocols for measuring or scoring such experiences. Two groups of researchers attempted to validate their own protocol (Lew, et al, 2005; Patomella, et al, 2004; Patomella, et al, 2006). The P-Drive (Performance Analysis of Driving Ability) tool developed and studied by the team in Stockholm, Sweden (Patomella, et al, 2004; Patomella, et al, 2006) assessed drivers on twenty-one points, as seen in Figure 1.0.
Figure 1.0: Measurement Criteria for P-Drive Assessment Tool (Patomella, et al, 2004).
In contrast, the Simulator Performance Index (SPI) assesses individual performance on 12 measures (Lew, et al, 2005), as seen in Figure 1.1.
Figure 1.1: Measurement Criteria for SPI Assessment Tool (Lew, et al, 2005).
Comparison of the two protocols reveals that the SPI seems to focus more on motor performance measures, while P-Drive seems to focus more on attention and cognitive tasks. No effort was made by any of the studies reviewed herein to compare, through efficacy or validation, different simulator assessment tools.
Conclusion
While most of the studies attempted to equalize gender difference by having a relatively symmetrical separation of males versus females, this was sometimes problematic with the target population. The research conducted on combat veterans (Amick, et al, 2013) had only one female in the target group, but an even split in the control group.
Taken as a whole, these studies represent a concerted effort to study driving simulators as an assessment measure to evaluate fitness to drive over the entire driving lifespan. Participants range in age from 18 (Lew, et al, 2005) to 87 years of age (Hoffman, et al, 2010). All but one (Gamache, et al, 2011) of the studies involved multiple subjects. Of these, all but one (Schmidt, et al, 1996) utilized a control group in comparison with the patient group.
One study concluded that “a moderate to strong relationship exists between the performance assessed on the simulator and neuropsychological tests that are known to predict safe driving and crashes” (Bedard, et al, 2010). Another noted that “the simulator variables could classify 85% of the subjects correctly with respect to overall driving skill with Complex Reaction Time as the significant variable” (Lundqvist, et al, 2000). Another concluded that “automated measures of simulator performance showed moderately strong ability to predict driving skills at the 10-month follow-up (82% predictive efficiency overall)” (Lew, et al, 2005). Another compared the driving simulator with traditional on-road assessments, noting that “driving simulation allows for the assessment of an individual’s performance during challenging driving scenarios (e.g., crash avoidance), which would be both unfeasible and unethical to assess during an actual road test” (Amick, et al, 2013). Another study found that “the automated measures of simulator performance at time-1 significantly predicted more aspects of driving at time-2 (3 of 4 subscales) than did the observer-ratings of simulator performance at time-1 (1 of 4 subscales)” (Lew, et al, 2005).
Several studies expressed concerns or caveats regarding the extent of the technology or the current research. One study noted “that simulator-recorded errors do not at this time weight the severity of the errors and that preventive behaviors (e.g., scanning properly at intersections) are not captured” (Bedard, et al, 2010).
In all, the reviewed studies present a generally favorable view of driving simulators as a possible assessment tool in evaluating fitness to drive in individuals suffering from cognitive impairment, with the caveats that more research needs to be performed, standardized protocols need to be established and validated, and the best utilization of this tool would be as part of a broader neuropsychological battery of tests including more traditional tools.
Referenced Articles
Amick, M. M., Kraft, M., & McGlinchey, R. (2013). Driving simulator performance of Veterans from the Iraq and Afghanistan wars. Journal Of Rehabilitation Research & Development, 50(4), 463-470.
Bédard, M., Parkkari, M., Weaver, B., Riendeau, J., & Dahlquist, M. (2010). Assessment of driving performance using a simulator protocol: Validity and reproducibility. American Journal Of Occupational Therapy, 64(2), 336-340.
Gamache, P., Lavallière, M., Tremblay, M., Simoneau, M., & Teasdale, N. (2011). In-simulator training of driving abilities in a person with a traumatic brain injury. Brain Injury, 25(4), 416-425.
Hoffman, L., & McDowd, J. M. (2010). Simulator driving performance predicts accident reports five years later. Psychology And Aging, 25(3), 741-745.
Lew, H. L., Poole, J. H., Lee, E., Jaffe, D. L., Huang, H., & Brodd, E. (2005). Predictive validity of driving-simulator assessments following traumatic brain injury: a preliminary study. Brain Injury, 19(3), 177-188.
Lundqvist, A., Gerdle, B., & Rönnberg, J. (2000). Neuropsychological aspects of driving after a stroke—in the simulator and on the road. Applied Cognitive Psychology, 14(2), 135-150.
Patomella, A., Caneman, G., Kottorp, A., & Tham, K. (2004). Identifying Scale and Person Response Validity of a new Assessment of Driving Ability. Scandinavian Journal Of Occupational Therapy, 11(2), 70-77.
Patomella, A., Tham, K., & Kottorp, A. (2006). P-DRIVE: ASSESSMENT OF DRIVING PERFORMANCE AFTER STROKE. Journal Of Rehabilitation Medicine (Taylor & Francis Ltd), 38(5), 273-279.
Schmidt, I, Brouwer, W., Vanier, M., & Kemp, F. (1996). Flexible adaptation to changing task demands in severe closed head injury patients: a driving simulator study. Applied Neuropsychology, 3(3/4), 155-168.
Tippin, J. (2007). Driving Impairment in Patients with Obstructive Sleep Apnea Syndrome. American Journal Of Electroneurodiagnostic Technology, 47(2), 114-126.
Wald, J. L., Liu, L., & Reil, S. (2000). Concurrent Validity of a Virtual Reality Driving Assessment for Persons with Brain Injury. Cyberpsychology & Behavior, 3(4), 643-654.
