Guide to measuring soft outcomes and distance travelled, J Eccles, ND Tackey, A Jackson

Tags: soft outcomes, self-assessment, indicator systems, progress, Beneficiaries, assessment, scales, outcomes, Copyright, soft skills, indicator system, target groups, the question, trivial questions, performance dimensions, aggregate score, Project workers, Stepping Stones Tyneside Careers Careers Centre Interchange Centre West Street Gateshead, precise questions, respondent, Assessment systems, vocational outcomes, Target group, communication skills, indicators, travelled, personal development, personal skills, assessment tools, assessment format
Content: the Institute for Employment Studies Guide to Measuring Soft Outcomes and Distance Travelled Sara Dewson Judith Eccles Nii Djan Tackey Annabel Jackson
Produced for the DfEE by: THE INSTITUTE FOR EMPLOYMENT STUDIES Mantell Building Falmer Brighton BN1 9RF UK Tel. + 44 (0) 1273 686751 Fax + 44 (0) 1273 690430 http://www.employment-studies.co.uk Crown © Copyright 2000 If you have any comments about this guide, please write to: Jacqui Hansbro ESF Evaluation Team Room W620 DfEE Moorfoot Sheffield SI 4PQ E-mail: [email protected]
Contents
1. Introduction
1
1.1 The aim of this guide
1
1.2 Some definitions
2
1.3 The importance of measuring soft outcomes and
`distance travelled'
3
2. What are Soft Outcomes?
5
2.1 Core outcomes
5
2.2 Target group-specific outcomes
6
2.3 Other issues to consider
7
3. How to Measure Soft Outcomes
8
3.1 Process or method
8
3.2 Who measures?
12
3.3 When to measure?
12
3.4 Management of systems
13
3.5 Presentation and use of soft outcome and distance
travelled information
13
4. The Tools Themselves
14
4.1 Phrasing the questions
14
4.2 Scoring the questions
18
4.3 Final word
21
Points of Reference
22
iii
iv
1. Introduction
1.1 The aim of this guide This guide has been written to assist projects and organisations to understand and implement systems for recording soft outcomes and distance travelled. It offers a definition of these terms and presents different methods for recording soft outcomes and measuring `distance travelled'. It has been written primarily for organisations running projects under the ESF Objective 3 programme and particularly those in Policy Field 2 (equal opportunities for all and promoting social inclusion). In addition, it will be of interest to other projects and organisations working with ESF target groups (primarily those people who are some distance from labour market participation). The guide is intended to offer an introduction to soft outcomes and distance travelled. It is not designed to be fully comprehensive, or to answer all the questions that you may have, as that would not be possible within the scope of this guide. However, we hope that you will use this guide as a basis for discussion within your organisation and importantly, you will use it to inform the development of your own soft outcome monitoring system. We have presented a range of different activities and options but it is important to recognise that: ! It is not possible to touch on all the monitoring systems and tools that can be used. Your project may be using a system that is not presented here. ! Not all suggestions will be suitable for your particular project. You will have to decide what is most appropriate. ! Several projects have developed soft outcome systems and some of these are listed at the back of this guide. They have provided information to us about their systems and they are happy to discuss them in more detail with you if that would be helpful. We encourage the exchange of good practice in this way -- you can only learn by doing!
Guide to Measuring Soft Outcomes and Distance Travelled
1
1.2 Some definitions Throughout this guide, we make reference to various terms that may or may not be familiar. As it is important to have a common understanding of these, we provide some brief definitions below. To begin with there is often a degree of confusion as to what is meant by the terms soft outcomes, soft indicators and distance travelled. 1.2.1 Soft outcomes These are outcomes from training, support or guidance interventions, which unlike hard outcomes, such as qualifications and jobs, cannot be measured directly or tangibly. Soft outcomes may include achievements relating to: ! interpersonal skills, for example: social skills and coping with authority ! organisational skills, such as: personal organisation, and the ability to order and prioritise ! analytical skills, such as: the ability to exercise judgement, managing time or problem solving, and ! personal skills, for example: insight, motivation, confidence, reliability and health awareness. 1.2.2 Soft indicators There is interplay between indicators and outcomes, in that indicators are the means by which we can measure whether the outcomes have been achieved. The term soft indicators therefore can be used when referring to the achievements which may `indicate' acquisition or progress towards an outcome. A project may wish, for example, to explore whether an individual's motivation has increased over the length of the project. However, this is, to all intents and purposes, a subjective judgement, indicators (or measures) such as improved levels of attendance, improved time keeping and improved Communication Skills, can suggest strongly that motivation has increased. Not all indicators will be suitable for all target groups, and some will be target group specific. 1.2.3 Distance travelled The term distance travelled refers to the progress that a beneficiary makes towards employability or harder outcomes, as a result of the project intervention. The acquisition of certain soft outcomes may seem insignificant, but for certain individuals the leap forward in achieving these outcomes is
2
Crown © Copyright 2000
immense. A consideration of distance travelled is very important in contextualising beneficiaries' achievements. It is easy perhaps, to think that: Indicators (or measurements) of soft outcomes can be used as tools for measuring distance travelled towards labour market participation. 1.2.4 `Models' and `tools' for measuring soft outcomes and distance travelled Within this guide, we use these terms interchangeably to refer to any system that attempts, in some form or another, to measure, assess or record soft outcomes and progress towards greater employability. These take a myriad of forms, ranging from simple paper-based tools, to complex computerised systems. 1.3 The importance of measuring soft outcomes and `distance travelled' As a project, you may feel that you have enough monitoring responsibilities, and would like to concentrate on the `job in hand', such as providing training or support and guidance. However, if you do not already do so, it is important to recognise that soft outcomes and distance travelled, and the use of systems designed to capture information on them, should be an integral part of your project. Monitoring soft outcomes and a consideration of the distance travelled by beneficiaries should be an intrinsic and mainstream element of your project evaluation. Table 1.1 below, highlights some of the key benefits of incorporating soft outcome and distance travelled monitoring into project systems. In addition to the benefits that accrue to the project and the beneficiary, it is general good practice to measure soft outcomes and distance travelled. It improves the process of working with beneficiaries and raises the standard of service delivery. Consideration of soft outcomes also provides a valuable context for clients' needs and progress. It provides a truer, more rounded picture of successes.
Guide to Measuring Soft Outcomes and Distance Travelled
3
Table 1.1 Benefits of measuring soft outcomes and distance travelled
General benefits
Hard outcomes such as jobs obtained, numbers of qualifications, and numbers progressing onto further education and training (though useful in some cases), do not show the success of the project as a whole. They are an insufficient indicator of a beneficiary's increased employability. Target groups that are facing multiple barriers to employment may be a long way from being able to acquire a qualification or employment. Consideration of soft outcomes for such groups is a crucial indicator of success. Measuring soft outcomes can also help with the national level evaluation to provide a fuller picture of the impact of the programme as a whole.
Benefits at a project level
Many projects may have specific objectives that relate to the acquisition of soft outcomes. It is vital that projects have the systems in place to be able to measure progress towards these objectives. Soft outcomes can provide a further indicator as to the additionality of the intervention, ie positive results that may not have occurred in the absence of the activity in question.
Benefits at a beneficiary level
Employers are particularly interested in soft skills and abilities, and have been found greatly to value key skills, and personal attributes and attitudes. The beneficiary will therefore be at a distinct advantage in the labour market if they are able to demonstrate possession of such skills and attributes through evidence gained during the project.
Working with the beneficiary to record and monitor soft outcomes and distance travelled, and involving the beneficiary in the assessment process, can be a very empowering experience. A project worker may be able to demonstrate to the beneficiary that they have pre-existing skills and attributes of which they were unaware. In addition, if beneficiaries are made aware of the distance they have travelled it can be an enormous confidence boost.
The development and recognition of soft skills is part of many beneficiaries' long term integration into the labour market.
Source: IES
4
Crown © Copyright 2000
2. What are Soft Outcomes?
So, what exactly are soft outcomes? Soft outcomes can be grouped in many different ways. Some `core' indicators and outcomes are likely to be applicable to most target groups and clients. It is perhaps useful to look at these in the first instance before considering some of the other soft indicators and outcomes that are appropriate for specific individuals or project target groups. 2.1 Core outcomes As you will see, there are no set rules regarding which indicators relate to particular outcomes, but some of the headings or groupings which may be useful in classifying `core' soft outcomes are: ! key work skills ! attitudinal skills ! personal skills, and ! practical skills. These outcomes, and some of the indicators that you might use to measure them, are highlighted in Table 2.1. Although some outcomes, such as key work skills and practical skills, are perhaps easier to measure, less tangible attitudinal outcomes are equally important as they can move beneficiaries towards `harder', more vocational outcomes and jobs. Personal skills and attitudes are as important to employers as their vocational counterparts. Increased confidence, motivation and self-esteem are extremely common indicators that are used in the context of ESF projects. Many beneficiaries face multiple barriers to labour market participation and suffer high levels of disadvantage, which converge to increase the likelihood of low confidence and self-esteem. Projects need to work towards measuring these outcomes to show the real and full impact of their efforts.
Guide to Measuring Soft Outcomes and Distance Travelled
5
Table 2.1 Examples of `core' soft outcomes and indicators
Types of `soft' outcomes Key work skills
Examples of indicators The acquisition of key skills eg team working, problem solving, numeracy skills, Information Technology Numbers of work placements The acquisition of language and communication skills Completion of work placements Lower rates of sickness related absence
Attitudinal skills
Increased levels of motivation Increased levels of confidence Recognition of prior skills Increased feelings of responsibility Increased levels of self-esteem Higher personal and career aspirations
Personal skills
Improved personal appearance/presentability Improved levels of attendance Improved timekeeping Improved personal hygiene Greater levels of self-awareness Better health and fitness Greater levels of concentration and/or engagement
Practical skills Source: IES
Ability to complete forms Ability to write a CV Improved ability to manage money Improved awareness of rights and responsibilities
2.2 Target group-specific outcomes The outcomes and indicators presented here are by no means an exclusive listing. Some indicators will be more suited to some target groups than others, and Table 2.2 below highlights this. It is important to stress again that as projects are working with people, and everyone is different, the suggested indicators may not be appropriate for every individual. Equally, some individuals will have additional barriers to overcome and therefore other indicators of
6
Crown © Copyright 2000
Table 2.2 Target group-specific outcomes
Target group Disaffected young people
Possible indicators Improved behaviour Improved attendance at school or at project sessions Improved engagement with learning or project activity
Women returners
Recognition of prior skills and experience
Drug/alcohol (ex)abusers
Better health Lower levels of drug or alcohol use Improved personal appearance/presentability
Ex-offenders
Lower rates of reconviction Lower rates of reoffending A slowing of reoffending
Homeless people
Permanent accommodation
People with learning disabilities or mental illness
A greater level of self-awareness Reduced anxiety and depression
Source: IES
progression and soft outcomes will be necessary. In essence, it is important to recognise the uniqueness of the individual and the great variety within `target groups'.
2.3 Other issues to consider The decision as to which outcomes should be considered, and how they should be measured, does not rest on the client group alone. Project objectives will need to be taken into account too. It may be useful to ask the following questions: ! Is the achievement of soft outcomes explicit in the project objectives, and if not, should it be? ! What is the ultimate aim of the intervention or project, and what soft outcomes will be relevant? Clarity around the project objectives will allow a rigorous and meaningful framework to be established around which monitoring and evaluation can take place.
Guide to Measuring Soft Outcomes and Distance Travelled
7
3. How to Measure Soft Outcomes 3.1 Process or method Methods to measure soft outcomes and distance travelled are numerous and varied and we must reiterate that one system will not suit all. What may work well for one project may not work for another. The system that you choose, and how you implement, will depend very much on the activities and objectives of your project, your target group, the length of the project, and the resources you have at your disposal. The options we provide here represent some of the methods you may wish to consider when designing a soft outcome monitoring system for your project. 3.1.1 Establishing a baseline This is a vital process if soft outcomes and distance travelled are going to be measured. It is crucial to establish a baseline of soft skills, aptitudes and attitudes from which individual progress can be measured. This can normally be done during the initial assessment phase when clients' needs are established, barriers to employability identified and personal development targets are set. An examination of `soft' skill needs falls naturally from this phase of the project. 3.1.2 How to collect and record information on soft outcomes and distance travelled It is vitally important that your methods for collecting and recording information on soft outcomes and distance travelled are rigorous and targeted to your client group. Some of the methods, which can be used to collect information on soft outcomes, are shown in Table 3.1 below. Selecting and implementing a range of methods to collect data on soft outcomes is more likely to capture a full picture of clients' progress.
8
Crown © Copyright 2000
Table 3.1 Collection methods
Type of collection method Individual Action Planning, personal action planning and goal setting Reviews between trainers/assessors and clients to record soft outcomes Daily Diary or personal journal In-depth reflection during or after the course Recorded observations of group or individual activities Presentation of material in a portfolio Tests
Comments and issues to consider The drawing up of individual action plans is normally carried out during the initial assessment session and then reviewed at regular intervals to gauge whether goals have been met. An action plan can include personal objectives, priorities and reflections on progress. Improvements over time can be noted and recorded during regular formal or informal reviews. This system is largely reliant on a sound judgement from the client or project worker and will not provide an absolute or formal measure of distance travelled. Clients can be encouraged to write about progress towards soft outcomes. Issues of confidentiality should be considered. Beneficiaries could be asked to consider and review their progress as they come to the end of their training course, or a particular element of the project (such as a work placement). This could be incorporated as an assignment that could be included in a beneficiary's portfolio of evidence of achievement. Baseline information is particularly useful here as data can be compared over time. Questionnaires are an important tool for this purpose. It is important to have comprehensive documentation systems that will allow for the recording of anecdotal evidence of outcomes achieved and progress made. This method requires a high level of observer skill, and there is the danger of observer bias, and also that the observer will influence the behaviour being observed. If the beneficiary/ies are unaware that they are being observed, this may negate the problem. This could include evidence of tasks completed successfully indicating achievement of outcomes, or progress towards them. An evidencebased portfolio would be a concrete output that could be presented to an employer. Some projects use psychometric testing within the assessment process. This is generally a diagnostic procedure but could be adapted to establish a baseline and measure distance travelled. Tests may be useful in establishing a beneficiary's existing skill level. The test could then be conducted at a later stage to illustrate any progress made.
Source: IES
3.1.3 The assessment process Assessment is usually a two-way process between trainer and client to ensure that judgements are reliable, robust and professional. Assessment should serve as a support function for the client to assist them in identifying real needs and bringing them to the attention of project workers. Beneficiaries must be willing participants and be able to see the value in the process. Any system of assessment needs to be built around positive affirmation of beneficiaries' current skills and attributes and
Guide to Measuring Soft Outcomes and Distance Travelled
9
highlight examples of positive progression. This creates an atmosphere where people feel valued and motivated. Essentially, assessment and review systems should enable beneficiaries to recognise the skills they already possess and identify their own capacity for learning and progress. assessment systems should also be mindful to guard against highlighting unnecessary and negative regression. This is not to say that regression should always be ignored, as clearly it may sometimes be due to circumstances beyond the clients' control. It is important therefore to identify the cause of the regression and then work with the client to support positive progression once more. Time is also a consideration when devising a soft indicator system, as some techniques require a fairly intensive input of time. Quite clearly, this would not be suitable for a project where contact is only sporadic such as those associated with outreach working. 3.1.4 Different assessment formats Most systems focus on paper-based methods of assessment, where beneficiaries complete written questionnaires, or write narrative accounts of their progress or their skills. Although this is appropriate for some target groups, there are circumstances in which the use of other assessment formats may be more suitable. Beneficiaries with numeracy, literacy or learning difficulties may experience problems in undertaking written assessments. One solution may be to work in conjunction with clients, using other media which is perhaps more appropriate to their needs. In addition to paper-based assessment tools, you may wish to consider the examples shown in Table 3.2 below. 3.1.5 Measuring soft outcomes and distance travelled Unlike `harder' job and qualification outcomes, it is difficult to achieve an `absolute' measure of achievement and progress in the realm of soft outcomes. In many cases, projects can only record that an improvement has occurred rather than being able to quantify how much improvement. However, some projects attempt to measure soft outcomes and distance travelled more systematically (and numerically) through scoring systems and scales. Commonly, beneficiaries are asked to complete questionnaires based on scales of feeling or agreement. A statement can be presented, such as, `I feel optimistic about my future' and the beneficiary can then
10
Crown © Copyright 2000
Table 3.2 Assessment tools
Assessment format
Example
Comments
Computer-based systems
eg where a beneficiary may complete assessment forms on-line
PC based systems of self-assessment have the added advantage of giving clients the opportunity to learn and develop IT skills at the same time (although projects should not assume that clients possess IT skills).
This medium may allow for a more systematic aggregation of data.
Games
eg self-assessment cards
This method may not be as intimidating as a written assessment. If carried out together with a skilled project worker, the use of games may encourage clients to talk about themselves even when they are lacking selfawareness.
3D media
eg where a beneficiary may move an `actual' sliding scale along a board
Again, this assessment format may be more suitable for those with numeracy and literacy difficulties, and may be less intimidating than a paper based assessment tool.
Source: IES
indicate their response on a scale of 1 and 10, for example, or `I agree strongly' through to `I disagree strongly'. The client's starting point can then be established (in the case of initial assessment) or a measurement of distance travelled over time be calculated (if used to review progress and ultimate outcomes).
It is important to ask the right questions and to avoid being ambiguous, meaningless or patronising when devising questionnaires and soft indicator systems, a point to which we return when we discuss `tools'.
3.1.6 Inferences Needless to say, the achievement of soft outcomes and progress towards employability cannot always be (entirely) attributable to the project. Other external factors can exert strong influences on beneficiaries' attitudes and achievements. A client may be more confident because of positive changes in his personal life rather than because of the training and support they are receiving from the project. It is important to understand a client's external situation in order to gain a better understanding of the context in which progress is being made and the factors that are most likely to be attributable to the project.
Guide to Measuring Soft Outcomes and Distance Travelled
11
Table 3.3 Type of assessment
Type of assessment Comments and issues to consider
Client or beneficiary self-assessment
Beneficiaries may answer in a way they feel is expected of them, and answers to questions may not be a true refection of their ability: just how they were feeling at the time. It is possible that a more honest and rounded response would be elicited if there were an honest and trusting relationship between the beneficiary and the member of staff responsible for the assessment.
Self-evaluation is a skill in itself, and not all beneficiaries will have sufficient levels of self-awareness to be able to self-assess in a meaningful way.
Employer assessment
This could provide a very useful addition to the soft outcome and distance travelled picture. If beneficiaries are undertaking work placements, employers could be asked to comment on the progress that beneficiaries have made.
Project staff assessment
Staff need to be well trained in the system which is being used and all need to working to common definitions. Due to the danger of subjectivity in assessment, external quality control or verification should be put in place to ensure a consistent approach. It is important to note that it is not always possible to remove bias from the way information is collected.
Staff need to be fully committed to the process and concept of soft outcomes and distance travelled.
Peer assessment
positive reinforcement from peer group members may be particularly encouraging. However, as a measurement of progress this method of assessment should not be used on its own, as there is a danger of subjectivity and bias.
Source: IES
3.2 Who measures? Who actually carries out an assessment of soft outcomes and distance travelled will depend on your particular project. Beneficiaries are clearly the mainstay of self-assessment, usually in conjunction with, and perhaps supplemented by, assessments from project staff and External Individuals. Those involved in assessment should be those who have regular contact with the beneficiary and whose professional judgements may add to the stock of knowledge about their progression. We have highlighted in Table 3.3 above, some of the issues that may emerge with different assessors.
3.3 When to measure? In order to measure progress and distance travelled, it is essential that assessment takes place at the very least at the beginning and end of your project. If time and resources permit, it is advisable to carry out assessment at regular intervals during the project. Not only does this allow for a more responsive service to changing needs, it also provides a more thorough and reliable evaluation of whether true progress is being made.
12
Crown © Copyright 2000
3.4 Management of systems The shift to consideration of soft outcomes and outputs may require a cultural shift both internally and externally, and result in a significant learning curve for project staff. If monitoring and evaluation is carried out externally, these personnel must be signed up to the concepts of soft outcomes and distance travelled. Communication is particularly vital if systems are to operate effectively. Good communication is extremely important between beneficiaries and staff, between staff, and between projects and other organisations to share good practice. Clear instructions and guidance must be given to project staff to enable them to carry out assessments of soft outcomes and distance travelled. 3.5 Presentation and use of soft outcome and distance travelled information It is important that you consider how the information that has been collected will be used and how it will be presented. You may want to use the information to: ! Provide evidence that the aims and objectives of the project have been fulfilled. ! Assess the added value of the project at an aggregate level. ! Feed into the overall project evaluation and dissemination strategies, and inform appropriate audiences of the project lessons. ! Provide feedback to individual beneficiaries to increase their confidence and satisfaction with their own achievements. Importantly, through feedback they will be able to see how they are meeting their individual aims and objectives, and fulfilling their personal goals. How the information will be used will help determine its presentation format. You may want to present it as: ! a written report which collates evidence at the project level. This could be included in an evaluation report, or could be presented as a stand alone document ! case studies of individual beneficiaries within a broader report to highlight success ! portfolios where beneficiaries can use the evidence of progress and outcomes in interviews or to construct CVs.
Guide to Measuring Soft Outcomes and Distance Travelled
13
4. The Tools Themselves There are two basic requirements for the tools that you devise to measure soft outcomes and distance travelled. These are: ! reliability, and ! validity. Your tool should obtain consistently robust results over time, with different client groups and with different staff members applying it, and the questions you ask must measure the outcomes, or performance dimensions, for which they were intended. The tool you use should allow you to improve your learning and understanding of the client group that you are working with, which will then allow you to provide more tailored provision for them. This need not be a complicated process. Clarity of thought and simplicity of design can go a long way towards ensuring a high quality soft indicator system. This technical note provides some overarching principles to help you design the tools for this task. 4.1 Phrasing the questions There are four tasks that you need to undertake: ! Think about the outcomes, or dimensions of performance, that you wish to measure. ! Translate these into meaningful and precise questions. ! Review the overall mix of questions to ensure they work together. ! Decide on the order in which questions will be asked. The questions you use should pass the following tests:
14
Crown © Copyright 2000
1: Is this the most direct way of asking the question? Think very carefully about the area in which you are interested. Does the question you have written ask about this? You should avoid proxies, which are simplified or trivial questions that stand in for more important subjects. 2: Is this the most concrete way of asking the question? Without straying from the area in which you are interested, you should ask your question in the most concrete way possible. The more factual and descriptive the question is, the less subjective the answer will be. An example You might not obtain a clear response if you ask `are you on time?' Punctuality could be considered to be composed of four elements: ! how often the person is late ! how late they are ! whether they are late due to circumstances outside their control ! whether they telephone to explain the reason. Perhaps a better way to ask a question regarding timekeeping would be to use a number of separate statements with which the client agrees or disagrees: ! I always arrive for the beginning of work/the course. ! I always telephone with a reason if I know I am going to be late. ! I only miss work/the course due to events outside my control. ! I am never late. ! My personal circumstances mean I often have to leave work/the course early. 3: Is the question really an element of employability? Can you show a logical link between the dimension you are investigating and employability? It is arguable that we should not include personality dimensions in soft indicator systems because this implies that there is only one acceptable form of personality, which is not true: pessimists do get jobs! Measuring behaviour and attitude has greater validity because these can be changed and are less intimate.
Guide to Measuring Soft Outcomes and Distance Travelled
15
4: Is the question measuring something that will change as a result of your work? Your soft indicator system will need to measure progress across a wide range of clients and individual circumstances. Without making a link between what you do as part of the project and the possible effect this has on clients, the soft indicator system will not show you at your best. If you do not have a good picture of the soft skills affected by your work, the soft indicator system should be able to help you, so review the initial results with this in mind. 5: Are you measuring each element of your project work? Look at the overall pattern of questions. Does it give a balanced picture of the work that you do? If the system concentrates on one element at the expense of others, you might inadvertently be valuing some parts of the project and particular staff more than others. This could distort the work of your project by encouraging staff to divert effort into the elements being measured. 6: Is the question taking too much for granted? The questions should not assume that the respondent has a particular possession, attitude or set of behaviours. For instance, questions for homeless people need to be reviewed very carefully to ensure that they are not implying that the respondent has a home, car, established set of friends, or daily routine. Questions about drink problems should not assume that all respondents drink alcohol: a filter question might be needed to ask if respondents drink alcohol (telling them to skip the next question(s) if they do not). 7: Is a positive answer necessarily good? Equivocal questions are difficult to interpret. For example, asking the respondent if they have `stopped themselves when they wanted to shout at someone' or `been pleased with something they have done' does not tell you whether these were appropriate responses. They could reflect passivity or poor self-awareness rather than assertiveness.
16
Crown © Copyright 2000
An example One system had the following questions for clients to agree or disagree with: ! This week I filled in a form. ! I did something fun. ! I have been pleased with something I have done. ! I have borrowed money. ! I have found it difficult to concentrate on things. Each of these is equivocal. The form could have been in a police station; the something fun could have been drugs; the selfcongratulations could have been misguided, showing poor selfawareness; the money borrowed could have been to buy clothes for a job interview, a positive action; the difficulty concentrating could have been good if it meant they were trying to listen to lessons where previously they would not have bothered. 8: Does the question duplicate information asked elsewhere in the tool, or elsewhere in the client records? A lean system is more cost effective. Repetition can be annoying both to the client and to project workers. Sometimes soft indicator systems (and other systems such as psychometric tests) have several questions that address similar issues, so as to check internal consistency; but this is not essential and should not be over-done. 9: Is the question unambiguous? Only ask one question at a time. Never use questions that have two or more parts to them. For instance, the question: `Have you been angry or violent with others?' is really two questions, one asking about anger, and another asking about violence. 10: Is the question in plain English? Phrasing your questions simply will reduce the time taken for the client to read them, and avoid misunderstanding or misreading. Avoid jargon, slang, or idiom as these will not be understood equally or consistently by all respondents. It is useful to check that all staff are working with the same definitions of any ambiguous words used in questions, and that the client has understood what is meant by the question.
Guide to Measuring Soft Outcomes and Distance Travelled
17
11: Is the question without any possible offence to women, ethnic minorities or other groups? As well as the obvious mistakes like sexist language, avoid the more subtle offences of taking cultural norms as standard. For instance, what some cultures consider as confidence and assertion might in others be considered unacceptable bragging and pride. 12: Is the question worded from the client's perspective? You will get a better response from clients if the questions imply some benefit to them, rather than simply a requirement for them to fit into society. Thus questions about `taking responsibility for yourself' could be seen as patronising or preaching. The same issues phrased in terms of `overcoming setbacks' could be seen as helping them develop streetwise behaviour. Similarly, wording questions in terms of clients' `realism' could be offensive as it implies there is only one correct way of seeing things. A second element of the client's perspective is that the order of questioning must make sense. It is a good idea to put simple, less personal questions at the beginning of the questionnaire and more personal ones at the end. 4.2 Scoring the questions Some form of measurement (quantification) is needed in soft indicator systems if a meaningful assessment of distance travelled is to be gained. The three reasons for this are: ! to avoid the subjectivity of words ! to give precision to the descriptions ! to allow comparisons between users, and across time periods. Having said this, measurement does not need to be heavy handed. Matters of degree can be shown through: ! scales: for example, rating from one to five, or one to seven ! symbols or pictures: for example, smiley faces ! physical artefacts, such as sliding counters on a board ! the accumulation of evidence. The following issues should be considered when designing your scales:
18
Crown © Copyright 2000
1: Is the context clear? People behave differently in different environments. If the context is not clear, the respondent might be frustrated with the question and tempted to reply `it depends'. And remember: you are only really interested in the user's behaviour at work, not what they do during their leisure time. An example A question about drugs might be better received and easier to apply if tied specifically to the work context, for example: ! My drug-taking affects my co-ordination and judgement ! When I use drugs I have difficulty getting up in the morning are better questions than the more general: ! My drug use is out of control. 2: Is the time period clear? Questions about behaviour, attitudes, or feelings should be tied to a specific time period. This should be long enough to allow the emotion or response to appear, but not so long that the respondent will be unable to remember what they were doing. 3: Are the scales anchored? People naturally use scales differently: some people group their responses around the middle of the scale whilst others stick to the extremes. You can encourage respondents to use the full range of the scale by giving a brief description for each number. For example A project that is interested in gauging the extent of drink or drug use might signpost scales in the following way: 1. No problem of this kind during the period rated. 2. Some over-indulgence within the social norm. 3. Loss of control over drinking or drug-taking but not seriously addicted. 4. Marked craving or dependence on alcohol or drugs, with frequent loss of control and risk taking under the influence. 5. Incapacitated by alcohol or drug problem.
Guide to Measuring Soft Outcomes and Distance Travelled
19
4: Do the scores need weighting? You can make some dimensions or questions more important than others by multiplying their score up before you calculate the total score. 5: Is an aggregate score meaningful? Aggregating scores across all the questions can hide more than they reveal, and some soft outcome areas might go down as others go up. An example of this is where greater awareness leads to a client acknowledging a problem that was previously denied. Aggregate figures should therefore be used with care. Project workers should always be given the chance to comment on figures so that apparently negative results can be explained. 6: Is the scoring positive? It is more meaningful to clients if high numbers used for scoring equate with high performance. You should also think about balancing the scoring so that even poor performance receives a result rather than nil. The soft indicator system is intended to motivate clients not to demoralise them! 7: Consider using a norm or standard People do not need to be perfect to get a job. They only need to be good enough and to meet some form of standard or competency. In some cases, the standard is implied in the anchoring used. For instance, in the question regarding drink and drug use given above, a score of 1 or 2 might be considered acceptable, depending on the type of job for which the candidate is applying. If the norm or standard is not obvious, then you might like to apply the soft indicator questionnaire to a sample of your colleagues or friends to give a baseline of people in employment. 8: Is the scoring too obvious to the respondent? Phrase some of the questions positively and some negatively. You do not want respondents to be able to guess that there is a `right answer', obtained by ticking the right hand column of the scales all the way down the page. 9: Minimise arithmetic The value of soft indicator systems can be reduced by mathematical errors. Project staff are busy people and might
20
Crown © Copyright 2000
4.3 Final word
not appreciate the task of adding up scores. Computerised systems that aggregate scores and reduce errors are often preferable. As you will appreciate, developing and using a soft indicator system will take a good deal of time from you, your staff and your users, so it is important that you get it right. You may want to look at some examples of other systems to assist with the development of your own, and we have given some contact details overleaf to help you with this. Exchanging examples and good practice in this way should minimise duplication and any potential reinvention of the wheel. However, you must be prepared to trial early versions of the system and obtain feedback as you go along. Ultimately though, measuring soft outcomes in a systematic way, in addition to gathering data on the harder job and qualification outcomes, will allow you to identify the real progress and achievements of your project and your clients.
Guide to Measuring Soft Outcomes and Distance Travelled
21
Points of Reference These projects have helped with the development of this Guide. If you would like any further information on the systems that they have devised to measure soft outcomes and distance travelled, please write to them, or e-mail directly. Stewart Bowman Mentoring Action and Stepping Stones Tyneside Careers Careers Centre Interchange Centre West Street Gateshead NE8 1BH email: [email protected] careers.co.uk Keith Stead The Rickter Scale Choiceworks 75 Townsend Crescent Kirkhill Morpeth Northumberland NE61 2EX email: [email protected] Amanda Pavon-Lopez Barrier Busters NOW Addington High School Fairchilds Avenue New Addington Surrey CR0 0AH email: [email protected]
22
Crown © Copyright 2000
Tom Farrier HARP (Care Skills for Working Life) Bishop Auckland College Woodhouse Lane Bishop Auckland Durham DL14 6JZ email: [email protected] Fiona Philipson and Fiona Langskaill Bridges Project 55 Albany Street Edinburgh EH1 3QY email: [email protected] Chris Llewellyn learning partnership West Floor 1 4 Colston Avenue Bristol BS1 4ST Robert Triggs Anglia Polytechnic University Bishop Hall Lane Chelmsford Essex CM1 1SQ Linda Keith and Karen Burgin Health and Social Gains Rehab UK Melbourne House Melbourne Street Newcastle-Upon-Tyne NE1 2JQ email: [email protected]
Guide to Measuring Soft Outcomes and Distance Travelled
23

J Eccles, ND Tackey, A Jackson

File: guide-to-measuring-soft-outcomes-and-distance-travelled.pdf
Author: J Eccles, ND Tackey, A Jackson
Published: Fri Aug 4 09:41:52 2000
Pages: 27
File size: 0.19 Mb


, pages, 0 Mb

Section C, 44 pages, 1.74 Mb

Jeff Wall, 13 pages, 0.07 Mb
Copyright © 2018 doc.uments.com