Preventing crime in Australia 1990 - 2002: a selected register of crime prevention projects
Evaluating crime prevention in Australia
Contents
An attempt has also been made to identify whether the projects presented had been evaluated in any way. Crime prevention evaluation is the process by which methods are applied to make judgments about the worth or value of a particular preventative project. Clearly, the evaluation of crime prevention is an important endeavour because it enables those working in the area to ascertain if the particular projects they are implementing really make a difference. In addition, it is also important to determine the return on investment for Australia's crime prevention expenditure. Crime costs Australia a considerable amount each year. In 1996, John Walker (1997) estimated that crime cost the Australian community a minimum of $11 billion annually in addition to up to $6.4 billion spent annually by criminal justice system agencies. He also estimated that the crime prevention security industry in Australia was worth $1.3 billion in 1996. It is therefore important that Australia's crime prevention projects be evaluated in some form or other to establish which are the most effective and merit resources being spent on them (Chisholm, 2000).
It is important that society's investment in crime control be evaluated to ensure evidence-based effectiveness. People want to know 'what works' and one can never be sure if something is working without evaluating it. Without evaluation it is difficult to establish whether existing crime prevention measures are having the desired effects. However, there are a number of questions relating to design, method and outcome that make the process of carrying out evaluation a complex and, at times, a contestable process.
Evaluation design can involve describing project resources, activities and outcomes. Broadly speaking, evaluations can be categorised as either implementation or impact orientated (see Wholey, Hatry & Newcomer, 1994).
Types of Evaluation
Implementation evaluations are monitoring exercises concerned with the actual process of implementing a crime prevention project. They tend to be managerial exercises concerned with monitoring, understanding and improving project delivery. Key questions raised by implementation evaluations may include whether the prevention project is reaching the appropriate targets or beneficiaries, whether it is being properly delivered, and what if any practical difficulties resulted. Implementation evaluations are important because they help to improve project designs prior to an evaluation of impact. Evaluations of impact are concerned with project effectiveness or outcomes. Key questions are whether or not crime was prevented or reduced and if so, could the reduction be attributed to the project rather than random fluctuations in criminal activity? In addition, were there unintended effects from the project such as displacement of crime into other areas and how cost effective was the project?
Evaluation Methods
The method used to evaluate a crime prevention project depends on the type of evaluation that is taking place (implementation or impact) and the kinds of questions that are being asked. Evaluations may be conducted qualitatively or quantitatively. Qualitative approaches often rely on open-ended interviews, personal observations, focus groups and/or ethnography to tease out the pros and cons of a particular crime prevention project. Qualitative techniques are generally more suited to implementation evaluation. Obtaining more rigid quantifiable data is not really necessary during project development where the aim is simply to obtain feedback regarding process. Impact evaluations on the other hand call for more scientific rigour, such as the use of quantitative techniques including experimental or quasi-experimental designs.
'Pure' scientific approaches involve controlled experimentation to determine whether a given crime prevention project is meeting its goals. This approach owes much to the field of medicine where questions about the extent to which treatments do or do not work are the subject of controlled experimentation (Laycock, 2001). A true scientific design compares a randomly selected control population with an experimental population to determine what works. In the field of crime prevention, however, true experimentation is rarely possible. We cannot take people out of society, place them in a laboratory setting, hold every facet of their lives constant, introduce a crime prevention measure and then wait to see what happens. Instead, social researchers usually rely on quasi-experimental designs and conduct their research out in the real world. Quasi-experimental designs are based on comparisons, either of the same group of people before and after a crime prevention measure has been introduced, or of communities in a crime-prevention scheme versus non-participating communities. Of course such an approach inevitably has to contend with extraneous factors which are beyond a researcher's control.
Quasi-experimental designs require evaluators to rule out other reasons for changing crime rates after a crime prevention project has been implemented. The main problem lies in detecting the presence of change and estimating its size. Government departments have the luxury of using crime statistics that have been collected over large areas and long periods of time. At the local level, where most crime prevention measures take place, actual offence numbers are small so that fluctuating crime levels may appear great. In this case, the effects of small, community-based crime prevention interventions may be difficult to distinguish from expected variations in criminal activity. One way to compensate for this would be to monitor the project over a long period of time. Such an undertaking would, however, be costly and beyond the budget of many community groups. The issue of how much change can be attributed to a specific project is also complex because keeping track of cause and effect can be very difficult. Crime rates within a given area can be affected by a multitude of factors. For example, police patrols, the local economy, the weather, and the incarceration of normally active offenders can all have a major impact on offending rates (Ekblom & Pease, 1995: 596-597).
The common perception amongst researchers, government and the public in general is that the preferred model of evaluation is quantitative ('scientific') and thus experimental to some degree. Both quantitative and qualitative have positive and negative aspects, and often the resources available determine the method that is chosen. For example, quasi-experimental research requires large-scale and sophisticated data collection. Analysing data can be a complex and highly skilled task. Subsequently, complex quasi-experimental designs may simply not be feasible because they demand extensive human and financial resources well beyond the budget of many who are involved in the everyday practise of crime prevention.
As we have already noted, the bottom line in impact evaluation is whether or not the crime prevention project has actually prevented or at least reduced crime. However, the area of impact evaluation, just like crime prevention itself, is somewhat contested. There are on-going disputes over what should actually constitute success or failure in crime prevention. One counter argument is that crime reduction/prevention per se should not be the only criterion used to assess a preventative project's success or failure. This raises several questions. Should crime prevention evaluations focus solely on the question of projects' measurable benefits, including decreases in offending? Or should evaluations be broader in scope and consider what other positive effects emerge from the processes of crime prevention? Quality of life issues, for example, may be a key outcome to take into consideration.
While not actually diminishing crime, reducing people's fear of crime could be considered a positive outcome from a crime prevention project. For example, women's self-defence courses aim to decrease the likelihood of victimisation but they may also reduce women's fear of crime. Thus, a woman who has attended a self-defence course may consider the program a great success simply because she feels less vulnerable and relaxed when she is at home alone. Whether or not the likelihood of her being victimised has actually decreased is an entirely different matter. Similarly, offenders' (or potential offenders') quality of life may be increased by involvement in a crime prevention project. Gang members who attend a life skills course could continue to offend regularly but may learn how to have closer, happier and more 'functional' relationships with their families. Both the offenders and their families would undoubtedly consider this outcome a success but if the project were only assessed in terms of crime reduction, it would fail dismally. Crime prevention projects may also result in people generally feeling better about the area in which they live. Projects may bring the community closer together and empower people to feel as though they have more control of their environment. These less quantifiable outcomes need to be taken into account in any crime prevention project evaluation.
This register seeks to identify crime prevention projects that have taken place in Australia since 1990 and the extent to which they have been subject to evaluation. Evaluation is defined broadly as any attempt made to monitor the process or outcome of a crime prevention measure. Thus, evaluations are viewed as occurring on a spectrum from very structured systematic quantified quasi-experimental designs to any unstructured informal qualitative feedback given about a specific crime prevention project. In line with this view, evaluations in the register will be categorised and indexed as being process (systematic or informal) and/or impact (systematic or informal) orientated.
In the past, Australian crime prevention practitioners have been criticised for failing to undertake evaluation (see Chan, 1994). A major obstacle to evaluation is the perception that it costs too much, is too hard to do and takes too long. Indeed, and as already noted, such concerns are often valid when it comes to sophisticated quasi-experimental designs. The purpose of this publication is not to critique the types of evaluation that have taken place. Rather it simply aims to highlight what has been done. Judgments about what should or should not be considered legitimate evaluative outcomes must await further research.