Development and Validation of a Computerised Self-Report Assessment Platform at STARTTS, J Aroche, M Coello, S Momartin, R Downham, A Iqbal

Tags: pen and paper, Amina Iqbal, client group, interpreter, STARTTS, Shakeh Momartin, STARTTS Jorge Aroche, Helen Bibby NSW Service, Russell Downham, Mariano Coello, Torture and Trauma, Kristine Knaevelsrud, true positives, test scores, validation study, internal consistency, Adnan Zagic Amy Luschwitz Dominica Dorning Emma Boles Gamal Dawlatly Hee Zee Lu, assessment, client assessment, Julia M�ller, Christine Knaevelsrud, choice questionnaires, interpreters, Pilot Study Results, Questionnaire administration, Support studies, odd numbers, Iraq Iran M, Multicentre Study, Depression, administrations, reading comprehension, Pilot Study, Journal of American Deafness and Rehabilitation Association, Computers in Human Behavior, Gallagher, Preliminary Results, clinical assessment, Historical developments, clients, Journal of Clinical Psychology, Journal of Educational Measurement, Julia Muller
Content: Development and Validation of a Computerised Self-Report Assessment Platform at STARTTS Jorge Aroche, Mariano Coello, Shakeh Momartin, Russell Downham, Amina Iqbal, Helen Bibby NSW Service for the Treatment And Rehabilitation of Torture and Trauma Survivors
The Problem · Many reasons & incentives to improve our client assessment methods · Very linguistically diverse client group · Some have low literacy in their own language1 · Large client volume · Problems with using interpreters ­ Very expensive ­ Inconsistent translations impacts on reliability & validity2 · Additional data entry costs for pen and paper forms
Finding a Solution · Meeting Julia Muller and Kristine Knaevelsrud at an ECOTS/ESTSS conference ­ Many discussions ­ Lots of translation and IT work · MultiCASI in use at STARTTS since 2014 ­ Part of routine clinical assessment ­ Installed on laptops in all counselling rooms ­ Several thousand questionnaires completed
What is MultiCASI?3 Multilingual Computer-Assisted Self Interview · Created by Christine Knaevelsrud & Julia Mьller (2008) at Berlin Center for Torture Victims2 · Software platform for administering multiple choice questionnaires: ­ Users upload translations and recordings of different questionnaires ­ Clients see and hear the questions in their own language ­ Clients respond using a mouse or touch screen ­ Data is exported for analysis · Functionality increased by adding Wizards (created by STARTTS and CTP).
MultiCASI at STARTTS · Small assessment battery administered to all clients (optional for clients). · Usually at end of 1st or 2nd session. · Explained through interpreter or video. · Client completes questionnaires following the session. · Battery repeated every 6-10 sessions or at discharge.
Selection Wizard
Client View
Results Wizard
Advantages Over Pen and Paper · Does not require an interpreter ­ Can be completed by client in private4 ­ Suitable for non-literate clients3,5 ­ Lower cost of administration2,6 ­ Allows client to revise and answer at own pace · Standardised delivery ­ Improves reliability6,7 ­ Prevents missing or ambiguous responses · Immediately scores tests and provides data to clinician · Data is already in digital form ­ No data-entry ­ Exports data to clinical data base
Challenges of MultiCASI versus Paper Assessment
Paper Assessment
Questionnaire translation
Reliability requires standardised
Questionnaire administration
Interpreter may change meaning
Some refugee languages don't of items2
have a written form
Interpreter's delivery may
Some clients lack the required influence response10 level of reading
comprehension in their own
Clients may be less forthcoming
language and English7
with interpreter/Privacy Issues11
Client and counsellor attitude/ knowledge of computers 8, 9
Interpreter's cultural background might be problematic10
Procedural validation study: Rationale · Important to ascertain that presenting the test in a computerised form : ­ Does not alter the test's reliability and validity12 ­ Does not affect acceptance by clients8,12 ­ Is equal to or better than pen and paper format · This is needed to: ­ Guide service provision ­ Support studies that use computer administered psychometric tests · Essentially, this study is a prelude to more interesting studies later on
pilot study · 61 STARTTS clients randomly assigned to complete the Hopkins Symptom Checklist13 (HSCL) using either MultiCASI or pen and paper · Participants completed an 11-item Acceptability Questionnaire5,14 · Participants were a convenience sample who completed the forms in English or Arabic
Pilot Study Results: HSCL and Time
· No significant differences were obtained between groups* for:
­ Anxiety (t= 0.51, p=0.60)
2.7 2.8
2.8 2.8
­ Depression (t=0.28, p=0.78)
­ Time taken to complete forms (t=-0.37, p=0.71)
Anxiety MultiCASI
Depression Pen & Paper
* The study had sufficient power to identify a medium effect size
Pilot Study Results: Acceptability
Survey Item
Pen & Paper MultiCASI
No/little experience with questionnaires
Problems completing questionnaires Which mode helps to protect privacy? Autonomous condition (PP/MC) Being asked by a staff member No difference/don't know Which mode helps to answer openly? Autonomous condition (PP/MC) Being asked by a staff member No difference/don't know
47% 33% 30% 37% 30% 30% 40%
23% 45% 13% 42% 55% 19% 26%
How was it/would it be to use the
computer? p<0.05
Comfortable Uncomfortable*
43% 27%
81% 10%
International Multicenter Procedural Validation Study · Consortium members: ­ STARTTS, NSW ­ UNSW Psychiatry Research and Teaching Unit, (PRTU), Liverpool Hospital, NSW ­ ASeTTS, WA ­ Refugees As Survivors New Zealand (RASNZ) ­ Auckland University of Technology, NZ ­ Competence Centre for Transcultural Psychiatry (CTP), Denmark. · 280 subjects · Estimated completion: December 2017
Multicentre Study: Method Administer HSCL13 using both pen and paper and MultiCASI · 15-20 min break between administrations · Order "randomized" based on MRN - Group 1 (odd numbers): Pen and paper first - Group 2 (even numbers): MultiCASI first
Client completes Acceptability Questionnaire5,14
Counsellor completes Demographics Form
Researchers compare MultiCASI responses to pen and paper
Preliminary results: Participants (1)
Gender Age (years) Country of birth Years in Australia
Male M (SD) Iraq Iran M (SD)
50.5 (10.9)
32.8% Sri Lanka
20.7% Other*
4.2 (4.4)
Education (years)
M (SD)
11 (3.4)
34.5% Tamil
Preferred language
26% Dari
* other countries of birth: Afghanistan, Algeria, Bangladesh, Burundi, Democratic Republic of Congo, Indonesia, Morocco, Pakistan, South Sudan, Sudan, Syria
Preliminary Results: Participants (2) · Most scored above the clinical cut-off on the HSCL (Anxiety: 84.5% Depression: 86% Total: 88%) · 47% had no previous experience with questionnaires · 31% had no previous experience with computers · 22% had the pen and paper form read to them by an interpreter
Preliminary Results:
Pen and Paper vs MultiCASI
Pen & Paper MultiCASI
Missing/invalid item responses
Average scores
Subscale variability (SD)
Internal consistency
(Cronbach's )
· No significant differences between the questionnaires for
Average subscale scores
Proportion of clients categorised as clinical `cases'
Preliminary Results: Psychometric Analysis of HSCL via MultiCASI
· Alternate forms reliability Anxiety: r = 0.91 Depression: r = 0.94 Total: r = 0.92 · Test validity (taking paper form as "gold standard")
· ROC curve analysis
Subscale Area Under the Curve (95% CI) p value
0.888 ­ 1.000
1.000 ­ 1.000
0.980 ­ 1.000
Preliminary Results: Acceptability Comparing computer to pen & paper
15% 30%
No difference/ don't know Prefer computer Prefer pen & paper
Comparing computer to staff member
No difference/ don't know Prefer computer Prefer staff member
Conclusions Pilot · No significant differences between MC & PP in terms of test scores, reported problems, timing · Actual discomfort is lower than anticipated discomfort Current study · MC & PP are extremely similar in terms of test scores, variability and internal consistency · Alternate forms reliability is excellent · Specificity (ability to identify `true positives') is excellent · Sensitivity (ability to identify `true negatives') may be a little low for HSCL Anxiety (more data is needed) · Most participants either see no difference between MC and other modes, or prefer computers · More data is needed to examine influence of demographics
future plans 1. Complete the multicentre procedural validation study 2. Develop a new, improved computer based assessment platform 3. Develop a new, culturally robust psychometric tool that addresses a larger number of presenting problems 4. Share 2 and 3 with IRCT members for free
CAMLAP Computer Administered Multi-Lingual Assessment Platform · Commissioned from original MultiCASI developer (Switzerland) · Estimated completion date: December 2017 · Improved compatibility with new operation systems · Modular design · More user-friendly · Full IP, so can be distributed to other centres for their own use
STARTTS validation study counsellors
Adnan Zagic Amy Luschwitz Dominica Dorning Emma Boles Gamal Dawlatly Hee Zee Lu James Malith Jo Haldane Jodie Birks Kat Mikhailouskaya
Kate Zughbi Katherine Theodor Marcel Abdelmasieh Nam-Hee Pellegrin Rebecca Wall Ross Predic Shivani Chavan Tess Reddel Winnie Kamakil
technical assistance
Erik Vindbjerg
Alexander van Vuch
Thank you to the members of the bilingual focus groups for invaluable assistance with translation checking and corrections
Thank You For further information: [email protected] [email protected] [email protected]
References (1) 1. Allen, M., Matthew, S. & Boland, J. (2004). Working with Immigrant and Refugee Populations: Issues and Hmong case study. Library Trends, Vol. 53, No. 2, Fall 2004. 2. Knaevelsrud, C., Wagner, B., Karl, A., & Mueller, J. (2007). New treatment approaches: Integrating new media in the treatment of war and torture victims. Torture, 17, 67-78. 3. Knaevelsrud, C. & Mьller, J. (2008). MultiCASI (Multilingual Computer Assisted Self Interview). CD-ROM, CDSP: ISBN: 978-3-540-71780-5. Berlin: Springer 4. Hofer, P. J. (1985). Developing standards for computerized psychological testing. Computers in human behavior, 1, 301-315. 5. Knaevelsrud, C., & Mueller, J. (2007). Multi-CASI ­ Acceptability of a computer assisted self-interview in a population of traumatized refugees. Oral presentation. 10th European Conference on Traumatic Stress (ECOTS), Opatija, Croatia. 6. Noyes, J. M. & Garland, K. J. (2008). Computer- vs. paper-based tasks: Are they equivalent? Ergonomics, Vol. 51, No. 9, 1352­1375. 7. Freire, M. (1990). Refugees: ESL and literacy trying to reinvent the self in a new language. Refuge, Vol. 10, No. 2 , December.
References (2) 8. Butcher, J. N., Perry, J., & Hahn, J. (2004). Computers in clinical assessment: historical developments, present status and future challenges. Journal of Clinical Psychology, 60, 331-345. 9. Gallagher, A., Bridgeman, B., and Cahalan, C., 2002. The effect of computerbased tests on racial, ethnic and gender groups. Journal of Educational Measurement, 39, 133­147. 10. Hamerdinger, S., & Karlin, B. (2003). Therapy using interpreters: Questions on the use of interpreters in therapeutic settings for monolingual therapists. Journal of American Deafness and Rehabilitation Association 36(3), 12­30. 11. Rentmeester, C. A. (2008). Trust, Translation, and HAART, Hastings Center Report Volume 38, Number 6, November-December 2008. 12. Weber B, Schneider B, J. F et al. (2002). Acceptance of computerized compared to pencil-and-paper assessment in psychiatric in-patients. Computers in Human Behavior. 19:81-93. 13. Parloff, M. B., Kelman, H. C., & Frank, J. D. (1954). Comfort, effectiveness, and self-awareness as criteria of improvement in psychotherapy. Am J Psychiatry, 111(5), 343-352. 14. Simхes, A.A., Bastos, F.I., Moreira, R.I., Lynch, K.G., Metzger, D.S. (2005). Acceptability of audio computer-assisted self interview (ACASI) among substance abusers seeking treatment in Rio de Janeiro, Brazil. Drug and Alcohol Dependence.;82(1):103­107.

J Aroche, M Coello, S Momartin, R Downham, A Iqbal

File: development-and-validation-of-a-computerised-self-report-assessment.pdf
Title: STARTTS MultiCASI Battery for assessing refugee trauma: Progress and Proposals
Author: J Aroche, M Coello, S Momartin, R Downham, A Iqbal
Published: Fri Jun 30 12:22:34 2017
Pages: 28
File size: 1.33 Mb

, pages, 0 Mb

A big bet on gluten-free, 5 pages, 0.09 Mb

To Kill a Mockingbird, 65 pages, 1.17 Mb
Copyright © 2018