DriveSmart, drives, cautiousness, advanced driving simulator, driving experience, performance, novice driver, Victorian Transport Accident Commission, experiment, Monash University Accident Research Centre, attentional control, novice drivers, Microsoft Flight Simulator 98, CD ROM training, control groups, control participants, Drive, Monash University, A. Regan, Thomas J. Triggs, MUARC, ROM product, Triggs, Regan, DRIVESMART NOVICE DRIVER CD-ROM TRAINING, insight training, flying training, CD-ROM training, Samia Toukhsati, training program, skill transfer, significant difference, attention control, Risk Perception, training programs, control subjects, Chantel Duncan, safe driving, ability, risk perception skills
SIMULATOR-BASED EVALUATION OF THE DRIVESMART NOVICE DRIVER CD-ROM TRAINING PRODUCT Michael. A. Regan, Thomas J. Triggs and Stuart T. Godley Monash University Accident Research Centre P.O. Box 70A, Monash University, Victoria 3800, Australia ABSTRACT
The Monash University Accident Research Centre (MUARC) recently completed research which culminated in the development of a CD ROM training product designed to accelerate the development of perceptual and cognitive driving skills known to be critical in reducing the crash risk of young novice drivers. This paper describes the design and outcomes of an experiment, using an advanced driving simulator located at MUARC, which evaluated the instructional effectiveness of the product. A total of 103 learner drivers
aged almost 18 years with between 40 and 110 hours of driving experience participated. Fifty two subjects (the treatment group
) completed 5 sessions of DriveSmart training. Fifty one subjects (the control group
) completed 5 sessions of training using the Microsoft Flight Simulator 98 CD ROM product. One week after training, and again 4 weeks later, all subjects performed seven drives in the MUARC simulator which assessed their risk perception and attentional control skills. Overall, treatment subjects drove more safely in the simulator than controls following training and were no more confident in their driving ability following training than controls. These training effects generalised to risky traffic situations which were not encountered during training and persisted for at least four weeks after training.
The over-involvement of young novice drivers in motor vehicle crashes is recognised, in Australia and in other industrialised countries, as a major public health problem. It has been estimated that between 50 and 70 percent of novice driver accidents may be attributable to inexperience (Gregersen, 1996a). Whilst knowledge is increasing about the critical perceptual and cognitive skill deficits which underlie inexperience-related crashes (see Mayhew & Simpson, 1995, for a review), there is little guidance in the literature on how to effectively train these skills (e.g., Gregersen, 1996b; Regan, Deery & Triggs, 1998a; Regan, Triggs and Deery, 1998c; Wheeler et al., 1997). In 1995, the Monash University Accident Research Centre (MUARC) was contracted by the Victorian Transport Accident Commission (TAC) to conduct research using an advanced driving simulator to investigate, and determine techniques for effectively training, four skills which were identified (Triggs, 1994) as being critical in moderating the crash involvement of novice drivers: risk perception (the ability to detect, perceive and assess the degree of risk associated with actual and emerging traffic hazards); attentional control (the ability to prioritise attention); time-sharing (the ability to share limited attention between multiple competing driving tasks); and calibration (the ability to moderate task demands according to one's own performance capabilities). This research (see Triggs and Regan,1998) culminated in the development, in 1999, of a CD-ROM training product known as DriveSmart (see Regan, Triggs and Wallace, 1999). The content for DriveSmart was drawn from the road safety
literature and from the findings of the research program referred to above. Content Areas
drawn from the literature included: insight training: optimism, commentary driving; prediction; and situation awareness. The remaining content areas were derived from the simulator experiments conducted by MUARC. The approach of Incremental Transfer Learning (ITL; Wallace and Regan, 1998) was selected as the general instructional strategy underpinning the product. ITL places considerable importance on the need to plan for both near-transfer and far-transfer of skills (see below). The purpose of this paper is to report on an evaluation of the CD ROM which was conducted by MUARC under the direction of an independently appointed Project Advisory Committee. The evaluation had two major components. The first of these is described in the present paper. It involved an experiment, using an advanced driving simulator, which was conducted to evaluate the instructional effectiveness of the CD ROM product. The second component, using Provus' Discrepancy model for curriculum evaluation (Brady, 1983), is reported in a companion paper at this conference (see Regan et. al., 2000).
Participants A total of 103 participants voluntarily participated in the experiment - 52 treatment participants and 51 controls. Participants were learner drivers residing in metropolitan Melbourne. They were aged between 16 years and 11 months and 17 years and 10 months. All participants had between 40 and 110 hours of on-road driving experience. This was the target group for whom DriveSmart was optimised. There were approximately equal numbers of males and females in each group. Participants were paid AUD $150 to participate in the experiment. Prior to the experiment, participants were screened for self-reported epilepsy and any colour vision deficiencies. About 6 percent of the participants recruited failed to complete the experiment.experimental design
A between-participants design was employed, with only one factor (treatment; control). Participants in the treatment condition completed the DriveSmart CD ROM training program. Control participants completed selected modules of the Microsoft Flight Simulator 98 CD ROM training product, which provided ab-initio flight training in a simulated environment. Modules were selected carefully to minimise the possibility of any positive transfer of flying training skills to the post-training simulator drives (see below). Immediately after training, and again 4 weeks after training, all participants completed seven "transfer-of-training" drives in the simulator. Assignment of participants to the treatment and control groups was counterbalanced for gender, hours of driving experience, amount of experience in driving automatic and manual transmission cars, and level of driver cautiousness. Level of cautiousness was assessed during an "Entrance Drive", described below. Apparatus Four PCs were used to train the control and experimental group
s on an individual basis. An advanced driving research simulator located at MUARC was used during the experiment. It consisted of a Ford Falcon Sedan with normal interior features, surrounded by 4 projection screens. The car was mounted on a motion platform. The experimenter controlled the simulator from a room located adjacent to the simulation room. The following questionnaires were administered by computer to participants at different stages of the experiment: a Demographics Questionnaire, which contained questions pertaining primarily to the amount and type of driving experience participants had acquired prior to the experiment; the NASA-Task Load Index (NASA-TLX), which provides a measure of subjective mental workload; the Situation Awareness Ratings Technique questionnaire (SART), which provides a measure of situation awareness; a Driving Confidence Questionnaire which provides subjective ratings of participants' confidence in their driving ability; and a Simulator Sickness Questionnaire (SSQ), to assess their state of well-being and health during the experiment. Procedure The experiment was conducted over seven sessions which spanned a 9-week period. Sessions 1 to 6 occurred a week apart. Session 7 occurred 4 weeks after the final training session. Session 1(Pre-training). Participants completed a Familiarisation Drive and a Practice Drive in the simulator in order to get used to operating the simulator vehicle. Next, they performed the Entrance drive which was designed to detect any baseline differences in degree of cautiousness between the treatment and control participants. This drive contained 5 everyday traffic events (e.g. approaching a signalised intersection) that could be expected to induce some degree of caution in participants. After that, they completed Autopilot Drive 1. Here, they sat in the simulator vehicle (but did not control the vehicle) and passively observed an unfolding traffic event in which an oncoming car swerved unexpectedly across their path. The purpose of this drive is explained later. All drives took approximately 5 minutes to complete. Finally, participants commenced training using either DriveSmart or the Flight Simulator software (see below). Sessions 2, 3,4 and 5 (Training). The bulk of the training was conducted in sessions 2, 3, 4 and 5. Subjects were instructed to complete only a specified amount of the relevant CD ROM package during each session, and were allowed as much time as they wished to do so. At the beginning of Sessions 2, 3, 4 and 5 all participants completed a driving log book in which they recorded details of all driving done since attending the previous session. Session 6 (Post-Training). Session 6 was undertaken one week after training. Participants completed another Practice Drive in the simulator. They then completed three 5-minute Attentional Control Drives. These assessed the ability of treatment and control participants to share and prioritise their attention. In each, participants saw six successive speed limit signs and were required to change to, and accurately maintain, the new speed. At the same time they performed a continuous self-paced auditory-verbal arithmetic task. In this task (Harms, 1991),
participants heard a series of two-digit numbers (e.g. "63") and were required to say aloud into a microphone the absolute difference between the two digits (e.g. "3"). Participants then performed four 5-minute Risk Perception Drives which were designed to assess the relative ability of treatment and control participants to perceive, assess and react appropriately to potential traffic hazards. Each of these drives contained four potentially hazardous traffic scenarios. These scenarios mostly involved those in which novice drivers in the State of Victoria are over-represented in crashes. Two of the scenarios in each Drive were ones that had been encountered during DriveSmart training (to assess "near transfer" of training) and the other two events were novel events (to assess "far transfer" of training). For each subject, two of the four Risk Perception Drives were performed concurrently with a subsidiary forcedpaced auditory-verbal arithmetic task. Here, participants heard a series of forced-paced two-digit numbers (e.g. "53") and were required to say aloud into a microphone the sum of the two individual digits (e.g. "8"). The subsidiary task was included to assess risk perception ability under high workload conditions. The order of presentation of the various drives was counterbalanced. After each block of drives, participants again completed AutoPilot Drive 1, described previously. Session 7. Session 7 was conducted four weeks after training. Participants completed a Practice Drive in the simulator. Following that, they completed a block each of Attentional Control and Risk Perception drives similar to those completed in Session 6 but with different speed signs and traffic events. Participants undertook three further simulator drives in Session 7, each lasting about 5 minutes. The first was an "Exit Drive", which was identical to the Entrance Drive in Session 1. The purpose of this drive was to provide a before-and-after training comparison of participants' level of cautiousness, subjective workload, and situation awareness. After that they completed "Autopilot Drive 2". This was identical to one of the Risk Perception Drives undertaken in Session 6, but without the secondary task. Here, however, participants did not have control of the simulator vehicle. During the drive, they simply pressed a response button on the steering wheel the moment they judged that each of the four traffic events was becoming unsafe. Following this drive they completed a "Near Miss Drive". This was similar to Autopilot Drive 1 . This time, however, they had control of the simulator vehicle and were be able to take action to avoid a collision with the vehicle swerving across their path. The relevance of these latter two drives is explained below.
The key findings which emerged from the analysis of the data are reported below.
Driving Experience Both at the commencement of the experiment (treatment = 70.3 hrs; control = 67.6 hrs), and during the course of the experiment (treatment = 9.7 hrs; control = 8.7), there was no significant difference in the mean number of self-reported driving hours undertaken by the treatment and control groups. Thus, any differences in performance between the two groups reported here cannot be attributed either to age differences or differences in hours of accumulated driving experience.
Entrance and Exit Drives Mean driving speeds were obtained for both the treatment and control groups in both the Entrance and Exit drives. These speeds were derived by averaging speeds ascross defined zones surrounding each of the 5 traffic situations in each drive. For the Entrance Drive, performed prior to training, there was no significant difference in mean speed between the groups. However in the Exit Drive, performed after training, the control group drove significantly faster than the treatment group (t (101)= 2.83, p = .006). Thus, exposure to DriveSmart led to more cautious driving, as reflected in relatively slower speeds, after training.
Attentional Control Drives For each of the three attention control drives in each of Sessions 6 and 7, the driving performances of treatment and control subjects were compared in 6 data collection
zones. The data were averaged across the six zones.
In Session 6, treatment participants performed significantly better than controls on the driving task, while showing equivalent performance on the HARMS arithmetic task. This was evidenced by treatment participants reaching the posted speed more quickly (F (1, 87) = 3.14, p=.081) and driving closer to the posted speeds (F (1, 87) = 3.65, p=.060).
In Session 7, treatment participants continued to perform better than controls on the driving task, but only in terms of being able to more quickly reach the posted speed (F (1, 87 = 3.14, p=.081). Treatment participants performed relatively better in the Harms reaction time
task as the three drives progressed, as shown by a significant linear trend interaction contrast (F (1, 87) = 5.34, p=.023).
Overall, these findings suggest that, taking both tasks into account, treatment subjects displayed superior attentional control skills relative to control subjects. This superior performance continued to be evident 4 weeks after training.
Risk Perception Drives For each of the 4 potentially hazardous traffic events within each Risk Perception Drive (each of which corresponded to a VicRoads Definition for Classifying Accidents; DCA) the experimenters, in conjunction with an instructional design expert, determined apriori safe driving behaviours that would be expected to be exhibited by treatment subjects relative to controls as a result of exposure to DriveSmart training. In addition, the dependent variable
s associated with these behaviours (e.g., braking onset time) were identified.
Analyses of variance were conducted for each of the dependent variables. Table 1 summarises the outcome of the analyses of variance for each DCA event within each of the drives (labeled Risk Perception Drives 1A, 1B etc). There are two categories of outcome shown in the table: positive and neutral. Positive denotes a statistically significant difference in performance (at the 10% level) between the treatment and control groups in the direction predicted for at least one of the dependent variable measures. For example, in one positive DCA event analysed, treatment subjects drove significantly slower (by about 5 km/hr) than controls (f= 6.08, p=.015) through a 1.5 kilometre stretch of simulated fog. This was a clear demonstration of far transfer of training. Neutral denotes that there were no statistically significant differences in performance between treatments and controls for any of the dependent variable measures. There were no negative outcomes.
Table 1. Outcomes of Risk Perception Drives in Sessions 6 and 7
SESSION 6 RP 1A RP 1B RP 1C RP1D SESSION 7 RP 2A RP 2B RP 2C RP 2D
NEAR TRANSFER Positive Positive Neutral Neutral Neutral Positive Positive Neutral Positive Neutral Positive Positive Neutral Neutral Neutral Neutral
FAR TRANSFER Positive Neutral Neutral Positive Neutral Positive Neutral Positive Neutral Positive Neutral Positive Positive Neutral Positive Neutral
Overall, it can be seen that 15 out of the 32 traffic scenarios analysed were found to be positive. In other words, DriveSmart training enhanced risk perception in about half of the scenarios assessed. It also should be noted that, in many of the neutral events, there was a tendency for treatment participants to perform better than controls. DriveSmart training was equally effective in facilitating both near and far transfer of training, and the training effect was equally strong 4 weeks after training. The positive outcomes are practically significant given that they reflect safe driving behaviours as identified apriori by the experimenters.
Confidence in Driving Ability In Sessions 1 (before training) and 6 (after training), all participants undertook Autopilot Drive 1 (described above). Immediately after these drives, they were asked the following question: "For the situation you just experienced in the simulator, if you were controlling the car, what chance do you think you would have had of avoiding a collision?" Answers were recorded as a score from 1 to 25. There was no significant difference between treatment and control subjects' ratings of their ability to avoid a collision, before and after training. In
addition, an equal percentage of treatment and control participants (94%) were unable to avoid a collision in the Near Miss Drive in Session 7. These findings demonstrate that treatment subjects were no more confident than controls of their ability to avoid a collision following training, implying that DriveSmart training did not induce over-confidence in driving ability.
DISCUSSION AND CONCLUSIONS
The findings from this study suggest that DriveSmart is effective in training both attentional control and risk perception skills and that these skills, once acquired, persist for at least 4 weeks after training. The product appears to be equally effective in preparing young novice drivers to safely handle risky traffic situations similar to those encountered during training (near skill transfer) as well as potentially hazardous situations that are new and novel (far skill transfer). To the knowledge of the authors, this is the first time the effectiveness of a driver training program in facilitating both forms of skill transfer has been demonstrated.
Historically, many driver training programs have failed to enhance novice driver safety because they have not focussed on the development of safety-critical skills and/or have had the effect of making novice drivers overestimate their driving ability and, consequently, making them accept greater levels of risk. The present study shows that, at least within the virtual environment, the instructional strategies
embedded within DriveSmart are effective in training safety critical skills in a manner which does not induce over-estimation of driving ability.
DriveSmart was launched in Victoria in May 2000 and since July 2000, when it was released, over 10,000 copies have been distributed free of charge to learner drivers and various institutions in Victoria. Until DriveSmart is evaluated "on the road", it is not known how effective it will be in reducing the incidence of crashes attributable to inexperience. However it should be noted that previous studies have shown that people drive in the advanced simulator used here in a manner very similar to that in the real world (Fildes et al., 1997). Thus, we have reason to believe that the patterns of safe driving performance observed here are likely to be similarly exhibited in the real-world.
Finally, it should be stressed that DriveSmart training is not designed to provide the range and quality of stimuli encountered through actual driving and therefore does not provide a substitute for actual driving experience. However, it provides exposure to selected high risk circumstances through deliberate instructional methods which aim to establish effective perceptual and Cognitive Strategies
for responding to risky situations in the real world.
The research described here was funded by the Victorian Transport Accident Commission. We gratefully acknowledge this support and the contributions to the research of the following colleagues: Chantel Duncan; Nebojsa Tomasevic; Naomi Kowadlo; Samia Toukhsati; Lorraine Brown; and Patrick Wilken.
REFERENCES Brady, L. (1983). Curriculum Development in Australia. Sydney: Prentice Hall
. Fildes, B.N., Godley, S., Triggs, T.J., and Jarvis, J. (1997) Perceptual Countermeasures: Simulator Validation Study. MUARC Report prepared for Federal Office of Road Safety. Report Cr 169. Gregersen, N. P. (1996a). Young car drivers: Why are they over-represented in car accidents? How can driver training improve their situation? VT. Rapport 409A. Swedish National Road and Transport Institute. Gregersen, N.P. (1996b). Young Drivers' overestimation of their own skill- an experiment on the relation between training strategy and skill. Accident Analysis and Prevention, 28, 2, 243-250. Harms, L. (1991). experimental studies
of variations in cognitive load and driving speed in traffic and in driving simulation. In A. G. Gale, I. D. Brown, C. M. Haslegrave, I. Moorhead, & S. Taylor (Ed.s) Vision in Vehicles III. North Holland
: Elsevier Science
. Mayhew, D. R., & Simpson, H. M. (1995). The role of driving experience: Implications for training and licensing of new drivers. Insurance Bureau of Canada: Toronto, Canada. Regan, M.A., Deery, H. & Triggs, T. (1998a). Training for Attentional Control in Novice Car Drivers (pp. 1452 - 1456). In the Proceedings of the 42nd Annual Meeting of the human factors
and Ergonomics Society (Volume 2). Chicago, Illinois
: HFES. Regan, M.A., Triggs, T.J., Mitsopoulos, E., Duncan, C., & Godley, S. (2000). Provus Discrepancy Evaluation of the DRIVESMART novice driver CD ROM Trianing Product. In the proceedings of the Road safety Research, Policing and Education Conference
. Brisbane, Australia
. (In press).
Regan, M.A., Triggs, T.J. & Deery, H.A. (1998c). Training Cognitive Driving Skills: A Simulator Study (pp 163-171). In the proceedings of the 34th Annual Conference
of the Ergonomics Society of Australia. Melbourne, Australia
: ESA. Regan, M.A., Triggs, T.J. & Wallace, P.R. (1999). DriveSmart: a CD ROM Skills training product for novice car drivers. In the proceedings of the Traffic Safety on Two Continents conference (20/9). Malmo, Sweden, 20-22 September. Triggs, T.J. (1994). human performance
and driving: the role of simulation in improving young driver safety. In the Proceedings of the 12th Triennial Congress of the International Ergonomics Association. Toronto: Human Factors Association of Canada, Vol. 1, 23 26. Triggs, T.J. & Regan, M.A. (1998). Development of a cognitive skills
training product for novice drivers (pp 46-50). In the Proceedings of the 1998 Road Safety Research, Education and Enforcement Conference. Wellington, New Zealand
: Land Transport Authority. Wallace, P.R. and Regan, M.A. (1998). case study
: converting human factors research into design specifications for instructional simulation (pp. 117-121). In the Proceedings of the Third International SimTect Conference. Adelaide, Australia: SimTect. Wheeler, W. A., Deery, H. A., Wilson, N., McCormick, L., M., Tierney, P., & Tomasevic, N. (1997). Investigation into the modification of driving behaviour surrounding low probability events (IA). Monash University Accident Research Centre Technical Research Report (September, 1997), Melbourne, Australia.
MA Regan, TJ Triggs, ST Godley