Australian Institute of Criminology

Skip to content

Review methodology

The AIC has developed a methodological framework to guide the systematic review of prevention strategies that combines elements of both experimental and theory-based research designs and approaches to evaluation. The methodology for selecting, analysing and recommending strategies is outlined in detail in this section.

Systematic reviews and evidence-based crime prevention

Evidence-based crime prevention involves the practical application of research and evaluation findings in the development and implementation of measures to reduce crime. These measures should be targeted to areas of the greatest need (based on a detailed analysis of the problem) and adapted to suit local conditions, supported by a focus on outcomes and a commitment to demonstrating measurable results (and contributing to the evidence base) through evaluation and performance measurement (ECOSOC 2002; Homel 2009a; UNODC 2010). The importance of ensuring that strategies are based on a thorough understanding of the problem being addressed, have evidence of effectiveness and are rigorously evaluated are common principles underpinning contemporary crime prevention programs. These principles are reflected in both national, and state and territory crime prevention strategies currently operating in Australia (Cherney 2000; Cherney & Sutton 2007; Crime Prevention Queensland 1999; Homel 2005; Homel et al. 2007; NSW Department of Premier and Cabinet 2008; Office of Crime Prevention 2004), international strategies (Homel et al. 2004; Home Office 2007; National Crime Council 2003; National Crime Prevention Centre 2007) and in various publications and practice guides developed to assist policymakers and practitioners (ECOSOC 2002; UNODC 2010). Central agencies such as the NSW CPD aim to promote, coordinate and deliver evidence-based interventions through their various policies, programs and guidelines.

The implementation of evidence-based practice requires not only a strategy for accumulating evidence in relation to identified and agreed priority areas, but also requires a strategy to enable the active dissemination of evidence (targeted to those areas where it is most needed) and strategies to ensure that evidence is integrated into policy and actively used by practitioners (Nutley, Walter & Davies 2003). Numerous forms of good practice guides have been developed to support evidence-based crime prevention. These include guidance on various intervention or prevention methods, methodologies for determining and selecting appropriate interventions, approaches for ‘activating’ the principles or mechanisms designed to address the causes of crime, or to support the application of skills developed through practical experience (Tilley 2006).

Advocates of evidence-based policy argue that crime prevention policy and practice should be rational and based upon the best available evidence, which they suggest comes from systematic reviews of high-quality experimental and quasi-experimental evaluation research (Welsh & Farrington 2006a). Systematic reviews are a popular mechanism for drawing together the evidence base across multiple evaluations. Systematic reviews of the research and evaluation literature that combine elements of both experimental and theory-based research designs and approaches to evaluation may provide more useful information relating to both the effectiveness of different prevention strategies and their successful implementation. In giving greater recognition to contextual factors, they may also increase the integration of evidence into practice (Leeuw, van der Knaap & Bogaerts 2007).

In discussing the role of evaluation in learning what works to prevent crime, Eck (2005) identifies important considerations and questions that may confront policymakers and practitioners when making decisions regarding investment in particular crime prevention interventions. Specifically, Eck (2005: 700) suggests that the evidence needs to show that

for a specific crime problem an intervention is an appropriate choice; the application of this intervention [has previously] resulted in the prevention of the type of crime we are interested in; and if we applied the intervention again we will obtain similar results.

This is where systematic reviews are particularly valuable.

A combined approach to systematic review

The AIC developed a comprehensive approach to the systematic review of crime prevention strategies that combined both experimental and theory-based approaches to research and evaluation.

Experimental approach to research and evaluation

The review of evaluation research has drawn upon the SMS (see Table 1). The SMS was designed as a guide to assess the strength of scientific evidence and methodological quality of outcome evaluations, and forms the basis of systematic reviews undertaken by the Campbell Collaboration (Farrington et al. 2006). It is primarily focused on ensuring the highest possible level of internal validity and drawing valid conclusions regarding the causal relationship between interventions and the outcomes observed.

Table 1 provides an overview of the specific criteria for each level on the five point scale. A research design that achieves level three on the SMS (quasi-experimental design) is often considered the minimum design for drawing conclusions regarding the effectiveness of a crime prevention intervention (Farrington et al. 2006). Any outcome evaluation that does not reach this level is excluded from systematic reviews conducted by the Campbell Collaboration.

Table 1: Scientific Methods Scale

1

Correlation between a prevention program and measure of crime at one point in time

2

Measures of crime before and after the program, with no comparable control condition

3

Measures of crime before and after the program in experimental and comparable control condition

4

Measures of crime before and after the program in multiple units with and without the program, controlling for other variables that influence crime, or using comparison units that evidence only minor differences

5

Random assignment of program and control conditions to units

Source: Farrington et al. 2006: 16–17

However, there are limitations in adopting a strictly experimental approach to assessing the effectiveness of crime prevention interventions. In particular, this approach has been criticised for an emphasis on internal validity (ie attributing causation to the intervention in question) at the expense of external validity (ie generalisability of the study to others) and for failing to give adequate consideration to the mechanisms that underpin crime prevention interventions or the context in which these mechanisms are applied (Eck 2005; Lab 2003; Pawson & Tilley 1997; Tilley 2009). Critics suggest that systematic reviews that aggregate the findings from studies using experimental research designs by comparing outcomes for different types of interventions do not give adequate consideration to the various contextual factors, intervention characteristics or requirements for successful implementation that may have impacted upon the effectiveness of a crime prevention strategy.

Further, previous reviews have found a great number of evaluations do not meet level three on the SMS, mainly because the evaluators have been unable or unwilling to include an appropriately matched comparison group (or area) in the study, against which the results for the intervention group (or area) may be compared. There are practical challenges associated with strict adherence to an experimental approach to evaluation (Bowers, Sidebottom & Ekblom 2009). Some authors have suggested that rigorous adherence to experimental research design is not necessary or optimal in the evaluation of situational crime prevention initiatives, particularly given the practical challenges associated with finding similar geographic areas suitable for comparison (Eck 2002; Knutsson & Tilley 2009).

Similarly, the nature of some interventions enables comparison groups to be more readily identified. For example, in evaluating interventions that draw participants from an established institution, such as the criminal justice system, schools or hospitals, administrative data relating to key outcomes for comparison groups may be more readily available to evaluators. There are systems and processes in place to collect information (or through which additional information can be collected) on all people who come into contact with that institution, not just people who participate in certain programs. Conversely, interventions that draw participants from community settings (such as youth drop-in centres or recreational facilities and services), or which occur outside of established institutions, may be less likely to have systems in place to collect administrative data and the organisations involved may have limited access to data that might be available from other sources. As such, they may be less amenable to evaluations that employ experimental research designs.

There are, as a result, a large number of evaluation studies that are unable to meet this standard, particularly in terms of finding a suitable comparison group or area. Nevertheless, there is a strong argument for not disregarding evaluation findings simply because they do not include a comparison group. There are several advantages to non-experimental designs:

  • they tend to not be as costly as experimental research designs;
  • they can usually be conducted in-house or by locally available consultants, as they do not require the expertise needed for more complex statistical techniques often used in experimental studies;
  • they can be conducted after an initiative has been implemented (since they do not require the selection of randomly assigned or matched control and experimental groups prior to the intervention);
  • they can involve less intrusive data collection sources/methods;
  • they are appropriate when intervention and comparison groups are not able to be distinguished or separated;
  • they are suited to those cases where there are only limited time periods before and after an intervention that can be assessed; and
  • they can be used when crime rates are low for the targeted behaviour (Eck 2006b: 346–348).

However, the key to generating lessons from small-scale and non-experimental evaluations is the application and testing of theory, informed by research into the causes of crime. Theory is therefore becoming increasingly recognised as an important feature of crime prevention evaluation and practice (Eck 2005).

Realist evaluation

Realist evaluation provides an alternative (although not mutually exclusive) method of reviewing and assessing the effectiveness of crime prevention strategies. The realist approach argues that generalisation is permitted through the repeated application and testing of theory in the evaluation of crime prevention strategies (Eck 2005).

In practice, the realist approach does not preclude the use of experimental research designs or quantitative research methods (Ekblom & Pease 1995). Instead, it emphasises the importance of understanding the context in which crime prevention interventions are delivered and the mechanisms that bring about the observed outcomes (Pawson & Tilley 1997). This emphasis is designed to address the need to understand how and why a strategy is effective (and not just whether it is effective), and to assist others to be able to replicate and adapt evidence-based interventions, and identify what conditions are needed for optimal outcomes. Realist evaluations are concerned with not only determining which interventions work, but also why, for whom and in what circumstance (Tilley 2006). For this reason, studies that focus solely on whether an initiative is effective or ineffective offer an incomplete understanding of the intervention.

Realist evaluation is also concerned with the requirements for effective implementation (such as the availability of funding, stakeholder support and engagement, the knowledge, skills or access to expertise required, and the political climate), acknowledging that these factors have an important influence on the overall success of an intervention (Pawson & Tilley 1997). This level of detail is often (but not always) overlooked in the method-driven evaluations involving experimental and quasi-experimental research designs. Realist evaluation is based on the premise that what is found to be an effective intervention in highly controlled and experimental conditions will often not reflect how successful an intervention would be in other ‘real-world’ conditions (Pawson & Tilley 1997).

The value of the realist approach is that it shows that the application and testing of theories (such as routine activity theory) can be used to add weight to the findings of research that do not meet level three on the SMS, without comprising internal or external validity (Eck 2006b). There must be a relevant theory underpinning the intervention being evaluated and that theory must be sound and based on research into the causes of crime (Eck 2006b; Knutsson & Tilley 2009). This is particularly valuable when experimental research is neither practical nor possible, but is by no means incompatible with experimental research designs (Eck 2006b).

The benefits of a combined approach

Given the strengths and limitations of the two approaches described above, it makes sense to not rely on only one approach for the purpose of this review. In describing the benefits and pitfalls of both approaches, Eck (2006b: 356–357) suggests that

the solution is to create a mixed portfolio of intrusive, less-intrusive, and non-intrusive evaluations, so we can draw more valid conclusions about generalisability.

The AIC has applied this principle to the systematic review of crime prevention strategies.

Combining the experimental and realist approaches requires that, in addition to gathering information on the outcomes from evaluations that meet level three on the SMS, information is also recorded on the context in which the evaluated strategies are implemented and the mechanisms that work to deliver the observed outcomes. The effectiveness of interventions in different contexts (context sensitivity) and the conditions necessary for interventions to be effective can then be determined (Eck 2002).

A recent meta-evaluation of strategies designed to prevent repeat domestic burglary adopted a similar approach, integrating systematic review and scientific realism techniques (Grove 2011). This research was able to draw a number of conclusions about the effectiveness of strategies in the United States, United Kingdom and Australia in preventing repeat burglary victimisation, while also highlighting the importance of giving due consideration to the specific context of the problem being addressed and overcoming a number of common implementation problems in adapting evidence-based interventions (Grove 2011). This approach has also been used elsewhere (ie the Netherlands), resulting in a high degree of uptake in the application of the resulting evidence in informing policy and practice (van der Knaap et al. 2007). Further, combining the two approaches can help to overcome the fact that past systematic reviews have demonstrated that there is a lack of evaluation in community-based crime prevention meeting level three on the SMS (Sherman et al. 1998; Welsh & Hoshi 2006), but not at the expense of drawing meaningful results.

By combining both approaches, this review will be able to assist in adding to the knowledge base on the requirements for successful implementation combined with an assessment as to the effectiveness of different interventions. This will serve to enhance the ability of practitioners to choose an appropriate intervention, adapt that intervention to the local context and minimise the risk of implementation failure.

Criteria for including studies in the review

For the purpose of the current project, the AIC used the following criteria in selecting studies to be included in the review:

  • The study needed to meet level two on the SMS, insofar as there was a measure of crime before and after the program was implemented (based on recorded crime, survey or self-report data).
  • There needed to be a measure of at least one of the priority crimes before and after the intervention had been applied or, in the absence of this measure, a general measure of crime for the target group or area (for projects that specifically target one of the priority crime types).
  • Data used to measure key outcomes needed to be both valid and reliable.
  • There needed to be a sound theoretical basis underpinning the intervention that had been evaluated (as determined by the research team, using the theory underpinning the different approaches to crime prevention presented in this report).
  • The evaluated strategy needed to have been implemented by a community-based organisation (such as a local government) or delivered at the local level and to be appropriate to the NSW context. This included strategies for which local government were the lead agency with primary responsibility for implementation, those for which local government could contribute to in some capacity (albeit in a supporting role) and those that might be included in a local government crime prevention plan.
  • There was sufficient information to enable the research team to determine the mechanisms that had been ‘activated’ by the intervention.
  • There needed to be evidence that the intervention had been implemented as it was designed (ie implementation fidelity) so that outcomes could reasonably be attributed to the intervention(s) described.
  • There had been some accounting for, or an attempt to reject, alternative explanations for the outcomes that were observed, based on additional supporting evidence (not limited to the use of a comparison group).

Notable exclusions and inclusions

The selection process aimed to identify those strategies that were suitable for local government and as such, certain initiatives were immediately excluded from the review. While evaluations of operational policing strategies (as the sole intervention) were excluded from the review, initiatives that were delivered by police but could also be delivered by local government have been included. Other strategies that are the primary responsibility of criminal justice agencies (such as violence prevention programs delivered by correctional agencies) or strategies delivered in institutional settings (including bullying prevention delivered in schools as part of the school curriculum) have also been excluded (see Joliffe & Farrington 2009; Heseltine, Sarre & Day 2011 for a recent review of correctional programs for violent offenders and Gottfredson, Wilson & Najaka 2006; Ttofi, Farrington & Baldry 2008 for a comprehensive review of school-based programs).

There are some other important differences between the methodology used for the current review and the approaches used elsewhere. While some systematic reviews have included studies that measure the impact of interventions on risk and protective factors for certain types of crimes (eg Sherman et al. 2006; WHO 2010), the AIC was reluctant to draw upon research findings where the impact on a priority crime type was not directly or indirectly measured (with some exceptions). Given the lack of evaluations of community-based crime prevention with a high degree of statistical rigour (Welsh & Hoshi 2006), the AIC has not excluded studies on the basis that the statistical significance of results has not been calculated.

Table 2: Search strategy
Primary search terms Secondary search terms Sources of evaluation studies

Non-domestic violence related assault:

  • Assault
  • Violence
  • Aggression

Residential burglary:

  • Residential burglary
  • Break and enter
  • House/home and theft
  • House/home and stealing

Stealing from motor vehicles

  • Steal/ing from motor vehicle
  • Steal/ing from car
  • Theft from motor vehicle
  • Theft from car

Malicious damage

  • Property damage
  • Damage
  • Vandalism
  • Graffiti

Stealing from person

  • Steal/ing from person
  • Theft from person
  • Stealing
  • Theft
  • Pickpocketing
  • Bag snatching/dipping

Stealing from retail store

  • Steal/ing from retail store
  • Steal/ing from store
  • Steal/ing from shop
  • Shop stealing
  • Shoplifting
  • Theft and shop
  • Theft and store

Prevention

Reduction

Evaluation

Study

Review

Impact

Outcome

Intervention

Project

Program/programme

Previous reviews of the literature on the effectiveness of strategies to prevent violence and residential burglary (including systematic reviews and meta-analyses)

Online databases:

  • ProQuest
  • CINCH
  • National Criminal Justice Reference Service Abstracts
  • Criminal Justice Abstracts
  • AGIS
  • AFPD
  • FAMILY
  • SocINDEX
  • SAGE journals
  • Criminal Justice Periodicals
  • International Security & Counter Terrorism Reference Centre
  • APAIS
  • Business Source Premier
  • Regional Business News
  • Legal Online Journals
  • Google Scholar

Targeted websites:

  • Problem-Oriented Policing Centre
  • Archived UK Crime Reduction
  • Swedish National Council for Crime Prevention
  • Research centre (eg NSW Bureau of Crime Statistics and Research (BOCSAR), Crime Research Centre, RAND, Jill Dando Institute)
  • Organisational (Australasian Evaluation Society)

Search terms and sources of information

The AIC has undertaken a rigorous and comprehensive review of research, evaluation and review studies in the criminological, social sciences and other relevant literature, international and Australian literature, and published and unpublished literature. Appropriate search terms were identified to assist in the identification of relevant research literature. Searches involved a combination of primary search terms (crime type) and secondary search terms (focus of publication). Primary and secondary search terms and online sources of information are described in Table 2.

The review was rigorous and substantial, and covered key studies within the criminological and social science literature. Both primary sources of research and previous systematic reviews were included in the search. Where possible, the original source material for studies used in previous systematic reviews was located. However, given time constraints and the age of some of the material, studies were not excluded if the original source material could not be located. Experienced staff from AIC Information Services assisted the research team by searching and locating research materials, sourced through the AIC library, websites and various online databases.

Table 3 Framework for assessing effective crime prevention practices
Variable Description

Specific crime type

The specific type of offence(s) targeted by the intervention, including whether it was the primary focus of the project

Context

The context in which the intervention was applied. Includes details regarding:

  • the problem being addressed (not limited to crime type);
  • the target group or beneficiaries (eg families, young people, Indigenous communities);
  • the region in which the intervention was delivered (metropolitan, regional or remote);
  • the specific location targeted by the intervention (eg licensed premise, private residences, public spaces, housing estate)

Intervention(s)

A short description of the type of intervention(s) delivered as part of the evaluated strategy. Interventions were coded retrospectively in accordance with the classification scheme described in this report (see Attachment 1)

Mechanism(s)

Processes underpinning the effect of the intervention on the causes or precursors to the crime problem, based upon the conjunction of criminal opportunity (CCO). Multiple mechanisms may be specified. A brief explanation of the application of the mechanism as part of the intervention was recorded. The CCO is explained in the theory section of this report (see Table 4)

Outcomes

Brief description of key outcomes delivered by project (positive and negative, measured using quantitative or qualitative data). Specifies effect size where available. Outcomes may not be limited to impact on crime levels

Success factors

Factors that were identified as contributing to the overall effectiveness of the intervention as a crime prevention strategy and to the outcomes that were observed. Where the project was ineffective, identified those factors that contributed to negative outcomes (or factors that were identified by evaluation as necessary for success of future initiatives)

Implementation requirements

Factors that were necessary for the successful implementation of the project (ie what was required in order to put in place the proposed intervention). These factors may reflect general requirements for effective implementation (eg thorough planning and problem solving, community engagement, functional partnership arrangements) as well as practical considerations specific to certain types of interventions. Where the project was not implemented as planned, identified those factors that inhibited the ability of project personnel to implement the proposed project

Research design

Details relating to the research methods used to evaluate the impact of the intervention, including:

  • focus of the evaluation (process and/or outcome evaluation);
  • evaluation design (experimental, quasi-experimental, before-after, naturalistic, realist);
  • quantitative and/or qualitative research methods and source of data used to measure key outcomes;
  • level on the SMS

Classification framework

Findings from the review of crime prevention strategies were documented in accordance with the classification framework outlined in Table 3. Elements of this framework drew upon similar classification frameworks used by the AIC in other research projects, such as the comprehensive classification scheme used for the Review of the National Community Crime Prevention Programme (Homel et al. 2007).

In addition to collecting basic information about the intervention/s that were delivered, the outcomes that were observed and the research design used in the evaluation, populating the framework required detailed information to be provided on the context in which the strategy had been implemented and the mechanisms underpinning the intervention/s that were delivered. The review also sought to identify the requirements for effective implementation for each of the crime prevention strategies examined. This involved identifying factors that have been identified in a process evaluation (where available) as being necessary for the successful implementation of the evaluated strategy, reflecting the principles for good practice and requirements for successful implementation outlined in the National Crime Prevention Framework (AIC 2012), as well as more specific requirements relevant to particular interventions. Factors that contributed to the overall success of these strategies as crime prevention measures have also been identified. This was particularly important, as it helps to understand why the same intervention may not have worked in a comparable location with a similar crime problem.

Interventions supported by evidence of effectiveness

Using this methodology and framework, the AIC then identified those crime prevention interventions that were supported by evidence of effectiveness in reducing the priority crime types. This included those interventions that were most common among crime prevention strategies, had been subjected to multiple rigorous, high-quality evaluations (according to the criteria outlined above) and had demonstrated results. While previous research provides guidance with respect to determining the threshold for what constitutes an effective intervention (eg Sherman et al. 2006), specific criteria for assessing whether a particular crime prevention intervention could be regarded as effective were not set (ie how many positive evaluations meeting the standard for acceptance would be required in order to classify a strategy as effective).

The AIC focused on identifying interventions supported by evidence of effectiveness for a number of reasons:

  • The majority of evaluated strategies involved multiple interventions delivered as part of a multifaceted, comprehensive strategy and isolating the intervention or interventions that were most effective was regarded as problematic.
  • There was considerable variation in the quantity and quality of evaluation across the different priority crime types and to dismiss interventions because they did not reach a specific target would rule out promising interventions and overlook important lessons for practitioners.
  • Assessing an intervention as effective or ineffective ignores the important role of contextual factors, the mechanisms that underpin an intervention (or interventions) and implementation.

Interventions that appeared most frequently among evaluated strategies with evidence of effectiveness (ie the majority of studies demonstrated a positive impact) and the types of interventions that were most commonly delivered alongside them were identified. This enabled the AIC to draw some conclusions about the effectiveness of certain interventions, the type of interventions that have been delivered in combination and their impact on the identified priority crime types.

Interventions suitable for implementation by local government

In addition to identifying interventions that were supported by evidence of effectiveness, the review process then sought to determine which of these interventions would be suitable for local government to implement and to a lesser extent, those that would be suitable for CPD to lead and implement. This was determined using findings from the review process (in terms of who was responsible for the implementation of effective and promising strategies), the AIC’s previous experience in undertaking reviews of community-based crime prevention programs in a number of jurisdictions, the AIC’s research into crime prevention more broadly and discussions with the CPD. A discussion of the role of local government is presented in this report and was used to guide the AIC’s assessment. It is important to note that rather than reviewing the evidence in relation to a much wider range of crime prevention interventions than would be practical given the short timeframe and then determining which are suitable for implementation by local government, the AIC has drawn on previous research and experience to determine which strategies are suitable for local government and selected studies for inclusion in the review accordingly.