Full title:

Validation of the Christo Inventory for Substance-misuse Services (CISS): a simple outcome evaluation tool

Authors:

George Christo and Sally Spurrell

Royal Free Drug Service, Royal Free NHS Trust

457 Finchley Road

London NW3 6HN, UK.

And

Ron Alcorn

Substance Misuse Services

Camden and Islington Community (NHS Trust)

London NW1 2LT

Proof directions: All correspondence to George Christo

CISS forms with directions and comparison score tables available from first author

The Christo Inventory for Substance-misuse Services (CISS) was developed as a single page outcome evaluation tool completed by drug / alcohol service workers either from direct client interviews or from personal experience of their client supplemented by existing assessment notes. Its 0 to 20 unidimensional scale consists of 10 items reflecting clients’ problems with social functioning, general health, sexual / injecting risk behaviour, psychological functioning, occupation, criminal involvement, drug / alcohol use, ongoing support, compliance, and working relationships. Comparison scores indicating low, average or high problem severity were produced by 243 drug users attending a harm minimisation out patient service and 102 alcohol users at an outpatient alcohol service. Means and cut-off scores for abstinence oriented treatments were derived from a six-month follow-up of 90 treated drug users. Sub sets of the harm minimisation sample were used to derive item alpha, test-retest and inter-rater reliability coefficients of .74, .82 and .82 respectively. The inter-rater coefficient increased to .91 when retests were conducted the same day. Among the abstinence oriented treatment sample the CISS produced correlations ranging from .43 to .99 with the Opiate Treatment Index and measures of trait anxiety, unpleasant life events, poor quality of life and low self-esteem. The simplicity, flexibility and brevity of the CISS make it a useful tool allowing comparison of clients within and between many different service settings.

Keywords: Substance misuse, alcohol, treatment outcomes, evaluation, clinical audit.

Providers of substance misuse services are frequently required to empirically demonstrate the efficacy of their treatments. However, outcome evaluations pose a challenge for smaller services which may have neither the time nor the expertise to conduct detailed outcome research with multidimensional outcome questionnaires.

The areas considered relevant among substance misuse outcome questionnaires are social functioning, general health, sexual / injecting risk behaviour, psychological functioning, occupation, criminal involvement and drug / alcohol use. Examples of assessment tools in existence include the Addiction Severity Index (ASI; McLellan et al., 1980; McLellan et al., 1992), the Opiate Treatment Index (OTI; Darke et al., 1992) and the Maudsley Addiction Profile (MAP; Marsden et al., 1998). Other factors linked to good outcomes include continued use of ongoing support or aftercare (Christo and Franey 1995; Ouimette et al., 1988), and treatment compliance as illustrated by length of stay or treatment level attained (Shwartz et al., 1997; Toumbourou et al., 1998).

Current multidimensional outcome questionnaires take a while to complete, require the presence of the client in question, and cannot combine sub sections to produce a single total score for easy analysis by non-researchers. There is still a need for a clinical evaluation instrument that can produce a single score reflective of client problems across all relevant outcome domains. To be acceptable to busy workers, the instrument would have to be short, simple and not dependent on the unreliable attendance of substance misuse service clients. Since workers generally have a wealth of qualitative data from detailed client notes and assessment interviews, it may be assumed that workers are able to furnish required information on their clients' behalf. Although the information would largely comprise of subjective impressions which may be difficult to validate, a competent worker should be familiar with their client's status within the relevant outcome areas. Thus the Christo Inventory for Substance-misuse Services (CISS) was developed from elements within the aforementioned references to elicit workers’ assessments of their clients in a quick, standardised and reliable way.

The CISS prototype was used in the evaluation of outcomes of treatment placements purchased by Social Services (Christo, 1998). This pilot study illustrated the scale’s usefulness, sensitivity to change and ease of completion. Feedback from workers was used to refine the nine original items and a tenth item was added reflecting the quality of the working relationship with the client. Therapeutic relationships, pretreatment motivation and program engagement have been shown to be central attributes of effective treatment (Simpson et al., 1997).

The aims of this validation study were to produce reliability data for the refined scale and to standardise it against cross-sectional ‘snapshots’ of criterion groups attending different types of substance misuse services. Concurrent validity data were also produced by comparison to OTI sub scales and some measures of psychological functioning.

The CISS is a single page outcome evaluation tool which can be completed by drug / alcohol service workers either from direct client interviews or from personal experience of their client supplemented by existing assessment notes. Its 0 to 20 unidimensional scale consists of 10 items reflecting clients’ problems (in the past month) with social functioning, general health, sexual / injecting risk behaviour, psychological functioning, occupation, criminal involvement, drug / alcohol use, ongoing support, compliance, and working relationships. These areas are each scored on a three point scale of problem severity (0 = none, 1 = moderate, 2 = severe), and each point is illustrated with relevant examples for guidance (see Appendix). Thus a CISS total score of 0 indicates no problems, and a maximum score of 20 indicates severe problems in all areas.

The CISS was applied to all subjects but the following measures were only used with the abstinence oriented group.

The abstinence oriented group were given the above test battery during a structured interview at their place of residence. Interviews were conducted by the first author who is a psychologist specialising in substance misuse treatment research. He was also an independent interviewer not related to the subjects’ treatment facilities and all interviews were carried out in the strictest confidence. The interviews reported here were the third set of interviews conducted on the same client group and used in an earlier piece of extensive research (Christo, 1995). The CISS was completed retrospectively by the first author from personal knowledge of the subject’s functioning supplemented by detailed qualitative interview transcripts and some OTI items as an aide memoir. When completing the CISS, the interviewer was blind to OTI total scores as these were calculated in 1995 and recorded on a separate database not accessed at the time of completing the CISS forms (in 1998). Similarly, the interviewer was also blind to the contents of the other questionnaires as these were also not referred to during CISS completion.

The CISS was applied to the other three groups by experienced clinic staff keyworking their respective subjects. All staff had nursing, psychology, medical or social work backgrounds. Thirteen staff participated from the harm minimisation service, eleven staff from the alcohol service, and two from the outpatient unit supplying inter-rater reliability data. All workers were given a 15-minute briefing on scoring procedures, they then completed the CISS from client notes and recent personal knowledge of their clients. Where these sources were insufficient, further information was elicited during face to face interviews. The inter-rater reliability group were interviewed face to face by both workers successively.

Random urine screens of the 61 abstinence oriented subjects from participating therapeutic communities revealed no discrepancies with their confidential self-reports. Drug users' self-reports are generally valid when they know that they may be checked by urinalysis (e.g., Strang et al., 1989; Kleyn et al., 1993). Also, in a comparable follow-up on a similar population, Powell et al. (1993) found an 80% agreement between self-report and the results of urinalysis. Discrepancies were mostly due to occasional use being reported but not detected in the urine. The harm minimisation group generally gave valid self-reports unless they perceived some advantage to be gained from doing otherwise. However, their prescriptions were rarely contingent on behaviour or drug use, and keyworkers had the benefit of recent urine tests to assist in the assessment of supplementary drug use.

Analyses were carried out using two-tailed tests on SPSS Windows version 5.0.1. statistical software. Non-parametric analyses were used for the analysis of ordinal variables or where normal distributions were not present. Test-retest and inter-rater reliability coefficients were calculated using intra-class correlations (ICC; Bartko, 1976) derived from analysis of variance mean squares using the formulae specified by Streiner and Norman (1991).

When assessing outcome on the basis of six-month timeline follow back, the CISS scores for abstinent (n = 33, M = 2.9, SD = 2.0), lapsed (n = 22, M = 4.5, SD = 2.9) and relapsed (n = 35, M = 11.2, SD = 4.5) sub groups were found to vary significantly (F (2, 87) = 56.5, P < 0.001). A post hoc multiple range Bonferroni test indicated that mean CISS scores of the abstinent and lapsed groups did not significantly differ and they were classified as having a good outcome, both scored significantly lower than the relapsed group which was classified as having a poor outcome.

A CISS cut-off score of six or less indicating good outcome, was best able to discriminate between good and poor outcomes. This correctly identified 88% of all outcomes where drug use was assessed during the month preceding follow-up, and 84% of outcomes assessed using timeline follow back.

Table 1

Concurrent validity data, Spearman correlations N = 90

All correlations are significant at P < 0.001

Twenty consecutive assessments at an outpatient abstinence oriented service were conducted exclusively for the purposes of gaining further data on inter-rater reliability. Since only two assessors were used in random order of presentation, assessor effects could also be incorporated into the ICC calculation. Subjects were interviewed by both assessors on the same day. The Pearson correlation coefficient was .91 (P < 0.001). A two factor analysis of variance produced mean squares of 30.9, 0.4 and 1.5 for subjects, assessor and error respectively, this produced an ICC of .91. There was no systematic variation due to the assessors.

Table 2

Mean item score comparisons between drug and alcohol users

Non-parametric comparisons were conducted with Mann-Whitney U tests. A Bonferroni correction for 10 multiple tests set the alpha level at 0.005.

The item alpha coefficient of internal consistency was .74, the test-retest coefficient was .82, and the inter-rater coefficient was .82. The inter-rater coefficient increased to .91 when retests were conducted the same day. These figures indicated the scale to be of satisfactory reliability.

The CISS demonstrated good face validity as its items were quite acceptable to workers and clients. Participating workers stated that they preferred it relative to other instruments which were less simple, took longer to complete and required the presence of the client. Clients generally preferred face to face interviews that did not require self-completion questionnaires, and a significant minority of our clients had reading difficulties in any case.

As mentioned in the introduction, the CISS was developed from relevant outcome research and existing multidimensional instruments. As such, all CISS items address areas considered relevant to substance misuse outcome, so content validity is good.

The concurrent validity of the CISS was satisfactory, its scores correlated well with other measures of treatment outcome. Individuals with a high CISS score were more likely to score highly on trait-anxiety and all OTI sub scales. They were also more likely to have low self-esteem, a greater impact of recent unpleasant life events, and were more likely to have a low subjective opinion of the quality of their life.

The CISS appears to have good discriminant validity. Treated drug users who became abstinent scored an average of three CISS points, and those who relapsed scored an average of eleven. For abstinence oriented treatments a cut-off score of six or less was indicative of a good outcome among drug users. The spread of CISS scores covered the entire range and there was no restricted variance, floor or ceiling effect in the score distribution. Score ranges for low, average, and high problem severity, were produced for drug users and alcohol users. Alcohol users generally scored one CISS point less than drug users. Alcohol users were less likely to score on problems of social functioning, sexual / injecting risk behaviour and criminal involvement, but they were more likely to score on psychological problems.

Due to the constraints of conducting research in a busy harm minimisation service, the retest interval for test-retest and inter-rater reliability was about seven weeks and ideally could have been shorter. When the retest interval was minimised in the exclusive inter-rater reliability group, the inter-rater reliability coefficient rose to .9. However, the harm minimisation clients were mostly stabilised on methadone and had been engaged with the service for an average of about two and a half years, so no improvements were expected during a seven-week retest period. Nonetheless, Pearson correlations have been recommended as reliability coefficients in these cases as they take into account any differences in mean score between the first and second testing occasions (Rust and Golombok, 1989). However, no differences emerged between the Pearson and ICC correlations. Test-retest and inter-rater reliability coefficients were found to be high despite the long retest interval, but these figures were produced by experienced professionals (mostly nurses) who had been properly briefed on CISS administration. These findings may not generalise if the CISS is used without prior reference to the instructions. The CISS can be completed in a client’s absence, but in these cases it is intended for use by individuals who have previously worked with the client in question.

Comparison scores enable practitioners in other services to assess their clients’ status relative to a ‘typical’ group of presenting clients. However, the criterion groups used to standardise the CISS were all obtained from London based services. The score distributions they have generated are a useful guide, but further research is necessary to ensure these findings generalise to services in other geographical areas.

Although OTI scale scores were unknown at the time of CISS completion, some OTI items were referred to in order to assess subjects’ functioning within the relevant CISS domains. As such, there is no full independence between CISS and OTI scoring. It is thus possible that correlations with the OTI could have been inflated. However, lapse status and the other questionnaires used in the concurrent validity process were not referred to and can be seen as fully independent.

The CISS is principally a clinical evaluation tool and may not be suitable for detailed research where separate measures of functioning in independent domains (e.g., drug use and health) may be required. Breakdowns showing the percentage of clients with none, moderate or severe problems may be produced for separate CISS items. However, the individual item three point scales were not designed to be used separately. Sensitivity to change is only gained when the 10 items are combined to form the 0 to 20 CISS total score. Some services using the CISS have added supplementary scales to produce a more detailed record of an outcome of particular interest (e.g., drug use, self-esteem). However, the CISS has been shown to correlate well with such scales and their use may be unnecessary for evaluation purposes.

The mean scores obtained by alcohol users on the criminal involvement and sexual / injecting risk items were so low that some alcohol services may consider those items to be unnecessary. However, their inclusion allows comparison across all types of substance misuse settings. Also, our anecdotal clinical observations indicate that alcohol users are a heterogenous group, the use of other drugs is not uncommon and neither is disinhibition leading to violent, anti-social or sexual risk behaviours.

The CISS score is a general index of client problems across domains considered relevant to treatment outcome. Only two of its items directly relate to substance use, so it has sufficient scope to monitor outcomes where abstinence is not the main goal of treatment. However, it also produced markedly reduced scores where abstinence had been achieved. The CISS is thus being used for ongoing evaluation in abstinence based and harm minimisation treatment services, where it is incorporated as a regular part of the intake, discharge and case review procedures. CISS scores may assist in the prioritisation of new clients and the distribution of caseloads of equal difficulty among keyworkers.

Self-completion outcome scales are of little use when clients discharge themselves without notice. When using the CISS a keyworker can at least retrospectively report discharge status even after their client is no longer accessible.

The use of a single outcome measure facilitates the production of simple and readable clinical evaluation reports, without the need for a professional researcher / statistician trained in data handling and the analysis of multiple outcomes. Data analysis can be further simplified by reducing the CISS total score to the two or three outcome categories identified in this study e.g., ‘good / poor’ outcome or ‘low / average / high’ problem severity.

The CISS has been shown to be easy to use and very flexible in adapting to different service requirements. Its single page is easy to photocopy and uses little space in clients' files. It can be completed face to face or from personal experience of a client supplemented by existing assessment notes. Workers familiar with the instrument were able to complete it in three minutes during their administration time. This study has shown the CISS is applicable to either drug or alcohol users, and it is suitable for abstinence or non-abstinence oriented services as comparison scores have been provided for both.

With thanks to Amber Astron, Bob Blizard, Richard Leachman and the Ceres Randal Trust, Dr Christine Franey of the Centre for Research on Drugs and Health Behaviour, John Gordon-Smith and the staff of Hammersmith and Fulham Social services alcohol and drugs team, and Shereen Sadiq of Ealing, Hammersmith & Hounslow Health Authority. Thanks also to the staff and clients of the following London based services: Response Royal Free Drug Service, Camden and Islington Community Trust Alcohol Advisory Service, CORE trust, Phoenix House, SHARP, Bethlem Royal Hospital, Hope House, Crescent House, Thurston House and Milton House.