Article by Colin Bradbury appearing in issue 115, October 2000 of GLAAS mailing, the newsletter of Greater London Association of Alcohol Services

Time was spent going through the instrument and giving detailed guidance as to how to interpret the 10 items and thus obtain an accurate score. In addition, two guest speakers (of which I was one) were asked to talk about their impressions of using the CISS in practice in their own services.

I have used the CISS in the evaluation of the ADS Probation Partnership service run in Greater Manchester. The sample used was compiled from 410 CISS questionnaires which were administered by partnership staff over a six month period.

CISS scores were collected from 347 clients, at the first, fourth, eighth, twelfth, sixteenth and twentieth sessions respectively.

The main reservation that delegates seemed to have was around the CISS’s brevity. They questioned that, given the 0-2 scoring range for each of the 10 items, how in-depth and rich could the resulting data therefore be? Surely such a short questionnaire could not give us meaningful information about such a complex issue as alcohol or drug addiction?

George Christo responded by conceding that there are limitations to his instrument, but made the point that that there is no single "correct" way of carrying out service evaluation which is acknowledged as a "Gold Standard." He also reminded the audience that all evaluation has to be in some sense an approximation and draws its strength from numbers. In other words, it is not how descriptive an individual item of the CISS is about an individual client, but rather how well the client or client-group as a whole are described that is the key issue.

This issue reminded me of two other aspects of trying to monitor services that I think need to be appreciated by service providers and purchasers alike.

Firstly, one should also not fall into the trap of trying to "prove" that your service’s intervention has had a direct impact on clients’ behaviour and well-being. One can only say that treatment coincides with people improving. To turn the argument on its head, it is correct to say that there is no conclusive proof that smoking causes cancer. All that can be said is that people who smoke have a higher probability of getting cancer when compared to their non-smoking peers.

Secondly, one should remember where this drive for evidence based practice has come from, i.e. the medical field. The thinking is that, for example, there may be several different ways of performing a particular operation and the method that should be favoured is the one which research shows to be the most effective (however that is defined). When new evidence becomes available, then practice should be updated accordingly.

Consider however, that when you go to your GP with an illness and s/he prescribes you penicillin, they do not do an outcome study on you to ascertain whether or not their treatment has been effective. This is because they already know through research that this intervention is likely to be effective for your condition and see no point in duplicating work that has already been done.

Although this is not a perfect comparison, I would invite you to consider that when it comes to showing your services are effective to purchasers, part of this should be achieved by demonstrating that your clinical work follows principles of best practice as recommended by the most recent research evidence.

Therefore, services should perhaps worry less about putting in place an outcome monitoring system to rival Project Match or NTORS and instead use simple standardised tools such as the CISS. This will leave more time to be devoted to digesting and disseminating best practice.

The conference itself appeared to be a success. The CISS seems to have filled a gap in the current market of standardised outcome monitoring tools.

I am disinterested from the CISS in as much as I have nothing to gain from it being taken up as an evaluation tool. However, if someone is looking for a simple and brief tool to measure service outcome and effectiveness, then I would recommend very strongly that they give it a try.

Colin Bradbury, Research, Outcome Monitoring & Evaluation Manager, ADS, 87 Oldham Street

Manchester, M4 1LW. Tel: 0161 214 6409

* See "Christo, G., Spurrell, S. & Alcorn (2000)

"Validation of the Christo Inventory for Substance-misuse Services (CISS): a simple outcome evaluation tool."

Drug and Alcohol Dependence: 59 189-197.

The Howard Johnson Hotel, West Bromwich

Mr Paul Wells, Manager, Coventry Community Drug Team

Dr George Christo, Ph.D., Psych.D, Research & Clinical Psychologist, Royal Free & University College Medical School, University College London

Mr Paul Wells, Manager, Coventry Community Drug Team

Mr Colin Bradbury, Research Outcome Monitoring & Evaluation Manager, ADS

There will be an informal buffet lunch at 12.30 p.m. and the conference will start promptly at 1.30 p.m. and finish at 4.30 p.m.

evaluations

chosen.

evaluation designs: cross sectional, or follow-up.

Jennie Lowdell, West Midlands Regional Drug Misuse Database,

3^rd floor, West House, Lombard Street West, West Bromwich, West Midlands B70 8EG.

Fax: 0121 525 7980

The Howard Johnson Hotel, West Bromwich

Mr Paul Wells, Manager, Coventry Community Drug Team

Dr George Christo, Ph.D., Psych.D, Research & Clinical Psychologist, Royal Free & University College Medical School, University College London

Mr Paul Wells, Manager, Coventry Community Drug Team

Mr Colin Bradbury, Research Outcome Monitoring & Evaluation Manager, ADS

There are 80 available places for the above conference. There is a standard charge is of £25.00 per person.

Title:

First Name:

Surname:

Organisation:

Address:

Postcode:

Telephone Number:

Fax Number:

E-mail:

I require places for the Christo Inventory for Substance Misuse Services Conference,

Friday 8^th September 2000.

Invoices will be raised on receipt of this application.

Please return to: Mrs Jennie Lowdell, West Midlands Regional Drug Misuse Database Project, 3^rd floor, West House, Lombard Street West, West Bromwich, West Midlands. Tel: 0121 580 4331, Fax: 0121 525 7980, e-mail: dmdproj@hsrc.org.uk

AGENDA

FRIDAY, OCTOBER 27^TH 2000

SOCIAL CLUB, BLACKBERRY HILL HOSPITAL

South West Drug Problem Database, Cedar House, Blackberry Hill Hospital, Fishponds, Bristol BS16 2EW.

9.30 Coffee/Registration

10.00 Introduction Ian Sherwood, RDAS

10.05 Review of Databases and Ourcomes Barbara Boulton, SWDPD

10.15 Outcomes Measures Richard Elliott, S W Audit Project

10.35 Outcome Measures compared TBA

10.55 Coffee

11.50 CISS Dr George Christo

12.30 Plenary Barbara Boulton, Speakers

1.0 Lunch

Map Led by Michael Robinson

CISS Led by Dr George Christo

The uses of audit in outcomes Led by Richard Elliott, RDAS

Implications for Workers Led by Ian Sherwood, RDAS

Outcomes and Clinical Practice Led by Dr Judy Myles, RDAS

2.45 Coffee

3.00 Feedback from workgroups/Plenary Barbara Boulton

3.30 Round up Ian Sherwood, RDAS