Evaluating neighbourhood justice: Measuring and attributing outcomes for a community justice program

Foreword | Community justice programs such as the Red Hook Court in New York and the Neighbourhood Justice Centre in the City of Yarra in Melbourne represent an innovative approach to a range of crime and safety issues by engaging the community in identifying problems and generating solutions. However, as with many small-scale or specialist court programs, community courts have found it difficult to provide robust outcome evaluation data to demonstrate their effectiveness. These barriers include establishing cause and effect relationships between community court interventions and programs and large-scale social outcomes, controlling for the variations in offender characteristics (particularly risk) that arise as a by-product of the community court model, and detecting differences in outcomes when sample sizes are relatively small. In this article, three criminal justice outcomes relevant to community courts are considered—crime rates, community order completion rates and recidivism rates, and the methodological challenges involved in using them in evaluation studies. Clear benefits arising for the community justice model are reported; however, the importance of considering a wider range of outcomes when assessing a program’s worth, such as victim satisfaction and community engagement is noted.

Chris Dawson APM

One of the most important recent developments in criminal justice has been that of ‘neighbourhood’ or ‘community’ justice. Community justice recognises the important role that local communities play in responding to crime and provides a way to engage the community in identifying crime problems and solutions (Karp & Clear 2000). Dealing with crime under a community justice model involves extending the role of the justice system to help build community resilience in relation to the problems that make crime possible or more likely. The first court established on a ‘community justice’ model was the Midtown Community Court, located in the Times Square district, New York, in 1993. Since then, the idea of community justice has gained wide acceptance and support. The Center for Court Innovation (New York) reports that more than 30 community court programs have been established in various jurisdictions in the United States. The first UK community court was established in Liverpool in 2005, followed by a further 12 courts around the United Kingdom in 2008. There are now community or neighbourhood courts in South Africa, Canada, Scotland, Singapore and Australia, with more scheduled to follow.

As with any new justice program, the establishment of new community courts involves the investment of substantial amounts of public funds. It is therefore critical that governments understand whether these programs provide benefits commensurate with the funds invested in them. However, the evaluation of community and neighbourhood courts, and the calculation of cost-benefit measures pose a number of significant challenges. These include the complexity of the program model and in particular, its reliance on engagement with community service providers, the relatively small scale of many programs and the difficulty in attributing outcomes to a single cause.

This Trends & Issues paper examines data from the first comprehensive Australian community justice initiative—the Neighbourhood Justice Centre (NJC) in the City of Yarra, Melbourne—and suggests a number of strategies to improve understanding of how programs like this contribute to improved justice and community outcomes.

Key concepts in community justice

In one of the earliest attempts to define community justice, Karp and Clear (2000: 324) proposed that it included

all variants of crime prevention and justice activities that explicitly include the community in their processes, and set the enhancement of community quality of life as a goal.

They nominated five ‘core elements’ of community justice, including that:

• community justice operates at the neighbourhood level.
• community justice is problem solving.
• community justice decentralises authority and accountability.
• community justice gives priority to a community’s quality of life.
• community justice involves citizens in the justice process (Karp & Clear 2000).

A more detailed framework on nine ‘critical dimensions’ for community justice was developed by Goldkamp, Weiland and Irons-Guynn (2001) in their evaluation of the Hartford Community Court. These included:

• a focus on distinct crime problems (‘quality of life’ offences) and on distinct neighbourhoods and groups of people,
• special arrangements for the processing and disposition of cases and the screening and enrolment of participants,
• provision of on-site client services,
• the involvement of community members,
• differences in the outputs and outcomes resulting from their activity,
• and an emphasis on cooperative effort across a range of government agencies.

More or less the same set of distinctive features was identified by McKenna (2007) in her evaluation of the North Liverpool Community Justice Centre. However, it would be wrong to conclude that there is a single or agreed common model for community courts. Community courts vary in the range of onsite services available to clients, the extent to which clients are case managed and the extent to which mediation, restorative justice and therapeutic jurisprudence approaches are used. Perhaps the greatest area of variation is in the extent of local community involvement in the governance of community justice centres, with US-style courts having less centralised governance and more local accountability than those in United Kingdom or Australia.

The Neighbourhood Justice Centre

The NJC is a community court established to provide new and innovative ways of dealing with crime and other forms of social disorder, disadvantage and conflict in the City of Yarra, an inner-urban part of Melbourne characterised by high levels of social disadvantage and high crime rates. The NJC opened in 2007 and offers a range of justice and social services. It is a venue of the Magistrates’ Court of Victoria, with jurisdiction to hear most summary criminal offences, and family violence and personal safety matters involving residents of the City of Yarra. It is also a venue for the Victorian Civil and Administrative Tribunal, hearing mainly residential tenancies and civil matters. The NJC has a Client Services team that provides assessment, treatment and referral services to people referred through its justice processes, as well as directly to residents of the City of Yarra. Client services include mental health, drug and alcohol, housing, financial counselling education and other support agencies. A variety of justice-related agencies also have staff located at the centre, including Victoria Legal Aid, Fitzroy Legal Service, Community Correctional Services (who supervise offenders living in the City of Yarra), Dispute Settlement Centre of Victoria and Victoria Police.

The goals of the NJC reflect the key themes in community justice. The first goal is to prevent and reduce criminal and other harmful behaviour in the Yarra community. This goal is addressed by a three-part strategy comprising engagement with the City of Yarra Community to increase the community’s capacity to prevent and manage the impacts of crime and other harmful behaviour, providing dispute resolution and restorative justice practices, and enhancing offender accountability and thereby reducing recidivism. The second goal of increasing confidence in, and access to, the justice system for Yarra communities is also addressed via several distinct strategies, including two-way engagement between the justice sector and Yarra communities, improving community understanding of legal and human rights through education, and providing support services to victims of crime. The third goal of the NJC is to further develop the NJC justice model and facilitate the transfer of its practices to other courts and communities. This goal reflects the relatively new status of community justice in Victoria and Australia, and mirrors elements of the linkages between the Center for Court Innovation and some US community courts.

Evaluating community justice

Outcome evaluation is a key component in contemporary public sector program management approaches. The standard program evaluation model is relatively demanding in the conditions it imposes. Interventions should be well-defined and their intensity or ‘dose’ must be able to be accurately measured. The program to be evaluated should provide a sample of individuals of known characteristics (ideally selected in an unbiased way) to whom the measured intervention is applied. These participants should be matched to a control group that does not receive the intervention. Finally, the outcomes of interest should be well-defined and accurately measured, and the intervention should be isolated from any other factors that might bring about change in these outcomes (Maxfield & Babbie 2005).

However, when we consider the evaluation of innovative court programs like community courts or other forms of problem solving court we find that many of these conditions don’t apply. To begin with, such programs typically include a range of services (eg restorative justice processes, clinical interventions, case management and various forms of support) and different participants may receive different levels of the services on offer. A corollary of this is that there are a range of outcomes of interest to evaluators. The standard community justice model claims to have impacts on both community level variables (eg crime rates or community capacity) and individual level variables (reoffending, confidence in justice). Community justice processes can serve as ‘gateways’ to other forms of human services and outcomes like housing status, employment and mental health may also be relevant. The following sections examine three evaluation outcomes relevant to community justice evaluations using data from the NJC and consider the methodological challenges associated with each of them.

Crime rates and the problem of attribution

A central element in the community courts model is reducing neighbourhood crime rates through collaborative crime prevention and more effective sentencing. However, while crime rate data are often readily available, in general, community court evaluations have had difficulty in drawing meaningful conclusions about observed changes to crime rates. A key problem is that of attribution; that is, linking outcomes with the community justice interventions and not some other external factor. Even at the level of individual outcomes, community justice activities are often embedded in community networks, making it difficult to separate out the effects generated by the community court inputs from the effects attributable to linked services. Attributing evaluation outcomes is particularly problematic for community level outcomes like crime rates that are subject to change arising from a wide range of economic, social, institutional and political factors. For example, the first evaluation of the Midtown Community Court showed a small drop in prostitution and illegal vending crime, but suggested that this may have been the result of economic development in the neighbourhood (Sviridoff et al. 1997). Similarly, the Red Hook evaluation noted drops in arrest rates in the Center’s catchment area that were not replicated in other areas of Brooklyn; the evaluators noted that the data ‘do not allow us to establish a causal relationship between the Justice Center’s opening and the observed changes’ (Lee et al. 2013: 177).

In its foundation document, the NJC was tasked with reducing crime in the Yarra community, with a particular focus on burglary and motor vehicle theft. This was a substantial challenge—the City of Yarra has the highest crime rate of any Victorian Local Government Area (LGA) other than the City of Melbourne, with an aggregate crime rate in 2007–08 of around 18,000 per 100,000 population. By comparison, the crime rate for the state as a whole for that year was just over 7,000 per 100,000 population. Yarra had two and half times the rate of property crime as the state as a whole (14,500 versus 5,400 per 100,000) and three and a half times the rate of drug-related crime (980 versus 270 per 100,000). In the period after the NJC was established, crime rates in Yarra have fallen, with a 31 percent decline in total crime (see Figure 1), largely as the result of a 40 percent decline in property crime. Crime rates have generally fallen in Victoria over the same period (again, largely as the result of reductions in property crime) but the decline in Yarra is greater than that observed in comparable inner urban LGAs (Melbourne, Darebin, Port Phillip, Maribyrnong and Stonnington) or LGAs with high levels of social disadvantage (Dandenong, Frankston).

Figure 1: Variation in total crime rates: 2007–08 to 2012–13 (selected LGAs) [see attached PDF for bar chart]

However, as with evaluations of other community courts, the problem remains that the greater improvement in Yarra cannot be directly or solely attributed to the impact of the NJC. The fall in property crime rates over the last decade has been apparent across Australia and in many other developed countries, and has been attributed to changing population demography, more effective policing, better property security and falls in heroin consumption (Wan et al. 2012; Weatherburn & Holmes 2013). It is unclear why these factors would be much more significant in Yarra than in the adjoining LGA of Port Phillip or in Frankston with similar levels of social disadvantage (SEIFA disadvantage indices of 1,019 and 997 respectively). Thus, like the Red Hook evaluation, one is left with the conclusion that the observed changes in crime rates are what would be desired from an effective community court in Yarra, without being able to say definitively that they are directly caused by the activities at the NJC.

The answer to the problem of attribution in relation to crime rates is to tie this form of outcome measurement to specific crime prevention activities; for example, by comparing crime rate changes in locations or activity types where there has been a targeted intervention with comparable areas or activities where there was no intervention.

Offender accountability

Community courts aim to increase offender accountability through therapeutic jurisprudence practices that encourage greater engagement of the offender with the court, together with strict enforcement of non-custodial sanctions (Lang 2011). Evaluations can use community order completion rates as a measure for accountability, as offenders who take responsibility for their actions are considered to be more likely to successfully complete a community order. However, evaluations need to deal with the problem of ensuring that they use like-with-like comparisons. Community courts typically offer an enhanced range of sentencing and diversion alternatives, problem solving approaches and judicial monitoring, and may offer levels of social and therapeutic support to offenders that allow the court to sentence offenders differently than would be the case in a conventional court. As a result, it is necessary to structure analyses to take into account the inevitable differences in the groups that arise from the way that these processes operate.

One of the primary considerations in making like-with-like comparisons is the risk profile of offenders at different sites, as offenders of different risk levels are likely to be offered different levels of support and to experience different degrees of monitoring as part of their community orders. While offender risk assessment instruments are designed to predict the likelihood of future offending, they can also be used as part of the analytical strategy of ensuring like comparisons when comparing order completion rates. Victorian offenders recommended for a community order are assessed using the VISAT (Victorian Intervention Screening and Assessment Tool) and are assigned to low, moderate or high-risk categories based on the predicted probability of future offending. Statewide, 12 percent of people who completed a Community Based Order (CBO) between July 2008 and June 2011 were classified as high risk, 44 percent as moderate risk, and 44 percent as low risk. However, there is considerable variation in offender risk profiles between Community Correctional Services locations and offenders supervised on a CBO at the NJC during this period were nearly twice as likely to be classified as high risk (22%), less likely to be moderate risk (30%) and slightly more likely to be low risk (48%) compared with CBO offenders statewide.

CBO completion rates for offenders with different risk classifications can be compared across the NJC and other metropolitan Community Correctional Services locations. Four other urban locations were selected to be comparable to the NJC. Locations that do not manage their own breach actions, or that manage breaches on behalf of other locations, were excluded as these distort the apparent breach rates. Successful completion was defined as expiry of the order without any breach action being undertaken. Unsuccessful completion was defined as cancellation of the order as a result of breach action. Cases where breach action was undertaken but did not result in cancellation of the order; for example, where the court varied or confirmed the existing order or where the order was termination for reasons other than a breach, were excluded from the analysis.

* Adjusted standardised residual >1.96

** Adjusted standardised residual >2.92

Table 1 shows the unsuccessful order completion rates from July 2008 to June 2011 for each risk category at the NJC at the four comparison locations and statewide. The adjusted standardised residual statistic was used to identify locations with order completion rates that were higher or lower than expected. The relationship between risk category and the probability that the offender’s order was cancelled is clearly evident—60 percent of orders involving high risk offenders were cancelled, compared with 35 percent of those where the offender was moderate risk and only 18 percent of orders for low-risk offenders. Overall, the NJC had a significantly lower rate of unsuccessful orders than the four comparison sites (23% versus 34% across all 5 sites). However, where the NJC generated substantially better results than the other sites was in the high risk and moderate-risk cases, where the unsuccessful completion rates were less than half that at comparable sites. Interestingly, the unsuccessful completion rate for low-risk offenders at the NJC was not significantly different to the rates at comparable sites, suggesting that for cases where the NJC does not provide specialised therapeutic jurisprudence and support, its performance in regard to offender accountability is no different than those of other sites.

Recidivism

A key outcome for any justice program is the impact it has on reoffending and recidivism measures are a feature of many community court evaluations. However, the relatively small scale of many of these programs poses a significant problem for evaluators. A rough rule of thumb for recidivism comparisons is that intervention and control groups should have at least 200 members in order to detect a 10 percent change in reoffending rates (Colledge, Collier & Brand 1999). Innovative court program evaluations are generally only concerned with cases that receive an intensive intervention and this may require that a sample is accumulated over several years. The Red Hook Community Court is probably the most intensively studied example and the site of the strongest recidivism outcome study. The Red Hook study was based on 1,564 cases in each group and showed a statistically significant reduction in re-arrest two years after the intervention (Lee et al. 2013). By contrast, in several other studies, small sample size was reported as a problem, including the analysis of recidivism of juvenile offenders at Red Hook (N=102), at the North Liverpool (N=424) and Salford (N=94) Community Courts (Joliffe & Farrington 2009), at the Midtown Community Court (Sviridoff, Rottman & Weidner 2005) and in the first evaluation of the NJC (N=100) (Ross et al. 2009). In each of these evaluations, the data either showed no difference in recidivism or a difference that did not reach statistical significance.

Two possible strategies to deal with the problem of small sample sizes are to accumulate a sample over a long period of operation, or to carefully match intervention and control cases to remove the effect of any covariates of recidivism. An early (2009) attempt to measure recidivism at the NJC examined 100 cases dealt with over the 2007–08 year but was not able to show a statistically significant improvement (Victorian Auditor-General 2011). In 2014, a larger follow-up recidivism study was done using NJC cases that had been dealt with between May 2009 and March 2011, and where the defendant received a therapeutic intervention or referral. Each case was matched with a case heard at another Magistrates’ Court venue where no therapeutic programs were available, with matching based on the age, sex, primary offence committed, the number of recorded prior offences and the number of proven counts in the presenting matter. Initially, a sample of 200 cases in the intervention and comparison groups was generated, but on further review, 13 cases in each group were removed, leaving an analysis cohort of 187 in each group. Comparison of these two groups showed that they were more or less identical in terms of the demographic matching characteristics, with mean ages of 36.8 for the NJC group and 36.1 for the comparison group, and the proportion of female offenders (26% in each group). This proportion of females is higher than the overall representation of women in Magistrates’ Court matters and is likely to reflect the increased likelihood that females will receive a therapeutic intervention. Cases were matched for seriousness using the National Offence Index (Andersson 2003) and the two groups had almost identical mean National Offence Index scores (84.9 vs 85.1). The NJC group showed a higher number of prior offences than the comparison group (mean of 8.0 vs 6.2) and a higher number of proven offences in the presenting matter (mean of 4.9 vs 4.2). These criminal history differences are consistent with the finding in the offender accountability study that NJC offenders tend on average to be higher risk than those from other venues.

Recidivism was measured by following up each member of the NJC and comparison group for two years after their initial sentence. Where this involved a period of custody, the follow-up period commenced at the expiry of the custody episode. Recidivism was defined as a new proven offence. The analysis examined recidivism in two ways—a chi-square test to examine the proportion of recidivists in each group and survival analysis to examine the variation in recidivism rates over time.

In the NJC group, 61 of the 187 (33%) had a new proven offence in the two year follow-up period, compared with 83 of the 187 (44%) in the comparison group (see Table 2). This yields a Likelihood Ratio of 5.48, which is significant at the 0.025 level.

The second way to evaluate the impact of the community court model on recidivism uses survival analysis to examine the variation in recidivism over the follow-up period. In the 2014 study, this was carried out in two stages. In the first stage, the two survival distributions were directly compared using a Kaplan-Meier procedure. This showed a difference between the survival curves that was not statistically significant. The log-rank test was used to compare the variation in these two curves and this yielded a chi-square value of 2.65, with a significance of 0.104. This was smaller than required to demonstrate a statically significant difference in recidivism. However, we know that NJC offenders sentenced to community orders show a higher level of predicted risk than their counterparts in other locations and it seems likely that this may also have an effect on recidivism. Risk assessment information was only available for offenders who received a community order, but the number of prior offences provides a measure of the person’s prior criminal history and the number of charges in the presenting matter provides a measure of the seriousness of the index episode. In the second stage of the analysis, a Cox Regression procedure was used to compare the two groups, with the number of prior offences and number of charges in the presenting matter as covariates. A forward stepwise procedure was used, with the likelihood ratio change as the entry criterion and the number of proven offences truncated at 10.This yielded a regression model with the parameters shown in Table 3. When the effect of the two criminal history variables is held constant, the case type variable (NJC vs comparison) has a chi-square value of 4.36. This showed a statistically significant difference (p=0.038) between the NJC and comparison groups, with an increased hazard (likelihood of failure) for the comparison group.

Figure 2 Cox regression survival functions for NJC and control groups, with number of proven offences and number of prior offences as covariates [see attached PDF for graph]

Figure 2 shows the failure curves (1 minus survival) for the NJC and comparison groups in this regression model. As is usually the case in recidivism analysis, the rate of increase in the failure rate is greatest at the beginning of the follow-up period (ie after the sentence date for the index offence or the end of any period of custody imposed for that offence) but the slope of the failure curve (the increase in the failure rate over any period) gradually declines as time goes on. The Cox Regression analysis estimates recidivism for up to 900 days after the index event and even at that point, the likelihood of failure is continuing to rise, albeit slowly. It can also be seen that the likelihood of failure is lower for the NJC group at any point in time after the index event and that the gap between the failure rates of the NJC and comparison groups also increases as time goes on.

Conclusion

One of the defining features of the community justice model is that it seeks to be effective across a variety of domains—individual, community and justice systemic. The results presented here show that the NJC has achieved significant improvements in at least two areas critical to the justice system: community order compliance and recidivism. To the extent that government decision making about justice programs is driven by cost-effectiveness considerations, it seems likely that these traditional outcome measures will remain a central component in evaluations. The significance of evaluation in relation to these programs is illustrated by the closure of the North Liverpool following an evaluation that showed that it was not successful in reducing recidivism rates when compared with mainstream courts (Booth et al. 2012).

However, the versatility in approach of the community justice model also means that it is difficult to fully assess the effectiveness of programs within the constraints imposed by a rigorous evaluation framework.

This article has shown how some of the evaluation barriers facing community court projects can be addressed by focusing on outcomes that are directly related to the court’s activities, taking into account the variations in the inherent risk associated with different groups of offenders and ensuring that sample sizes are sufficient to show whether differences in outcomes are present.

Beyond this, there are a variety of other areas that are important in understanding the value of the community justice model. Evaluations tend to focus on certain outcome measures to the exclusion of others. Victim satisfaction with the criminal justice system is recognised as an important attribute governing how people feel about justice, but is rarely measured and is therefore absent from most evaluations. The same thing applies to outcomes like offenders reconciling with their family, finding stable accommodation or employment, or receiving treatment for mental health or substance abuse problems. We need to look beyond the conventional evaluation outcomes of crime rates, order compliance and recidivism and ask how we can incorporate these ‘broader sets of questions’ about changes in the quality of life of offenders, victims and communities in ways that demonstrate the contribution that community courts can make. These concerns were summed up by the Director of the Center for Court Innovation, Greg Berman (2011):

We’d like to see it be cost-effective, for example. We’d like to see it be efficient. We’d like to see it be fair. We’d like to see it treat defendants and victims with humanity and decency, and I think that in general, we need to move beyond this kind of pass/fail approach to evaluating criminal justice reform and ask these broader sets of questions if we’re really going to develop a kind of nuanced understanding of the field.

Note

For further information about the Neighbourhood Justice Centre go to www.neighbourhoodjustice.vic.gov.au

Acknowledgement

The author acknowledges the advice and suggestions provided by Dr Karen Gelb in the preparation of this article.

References

All URLs correct at February 2015

• Andersson C 2003. Development of a national offence index for the ranking of offences. Paper presented at the 2003 AIC/ABS Conference: Evaluation in Crime and Justice: Trends and Methods, Canberra
• Berman G 2011. Try again, fail again, fail better: Lessons from community courts. Interview for the Real World Seminar Series. http://nij.ncjrs.gov/multimedia/video-berman.htm
• Booth L, Altoft A, Dubourg R, Goncalves M & Mirrlees-Black C 2012. North Liverpool community justice centre: Analysis of re-offending rates and efficiency of court processes. London: Analytical Services, Ministry of Justice
• Colledge M, Collier P & Brand S 1999. Crime reduction programme and constructive regimes in prisons. Programmes for offenders: Guidance for evaluators. London: Home Office Research Development and Statistics Directorate
• Goldkamp J, Weiland D & Irons-Guynn C 2001. Developing an evaluation plan for community courts: Assessing the Hartford community court model. Washington DC: National Institute of Justice
• Joliffe D & Farrington D 2009. Initial evaluation of reconviction rates in community justice initiatives. Research summary 9/09. UK: Ministry of Justice
• Karp D & Clear T 2000. Community justice: A conceptual framework, in Friel C (ed.), Criminal justice 2000, volume 2. Washington DC: National Institute of Justice
• Lang J 2011. What is a community court? How the model is being adapted across the United States. New York: Center for Court Innovation
• Lee C et al. 2013. A community court grows in Brooklyn: A comprehensive evaluation of the Red Hook Community Justice Center. Williamsburg, VA: National Center for State Courts
• Maxfield MG & Babbie E 2005. Research methods for criminal justice and criminology, 4th ed. Belmont, CA: Wadsworth
• McKenna K 2007. Evaluation of the north Liverpool community justice centre, Ministry of Justice Research series 12/07. UK: Ministry of Justice
• Neighbourhood Justice Centre Strategic framework 2010–2013. Melbourne: Department of Justice
• Ross S, Halsey M, Bamford D, Cameron N & King A 2009. Evaluation of the neighbourhood justice centre, City of Yarra: Final report. Melbourne: Department of Justice
• Sviridoff M et al. 1997. Dispensing justice locally: The implementation and effects of the midtown community court. Washington DC: National Institute of Justice
• Sviridoff M, Rottman D & Weidner R 2005. Dispensing justice locally. The impacts, costs and benefits of the midtown community court. New York: Center for Court Innovation
• Victorian Auditor-General 2011. Problem solving approaches to Justice 2010–11. Melbourne: VAGO
• Wan W-Y, Moffatt S, Jones C & Weatherburn D 2012. The effect of arrest and imprisonment on crime. Contemporary Issues in Crime and Justice no. 158. Sydney: NSW Bureau of Crime Statistics & Research
• Weatherburn D & Holmes J 2013. The great property crime drop: A regional analysis. Issue Paper no. 88. Sydney: NSW Bureau of Crime Statistics & Research

About the authors

Stuart Ross is Senior Fellow in the School of Social and Political Sciences at the University of Melbourne. His research interests include sentencing decision making, criminal justice population modelling, and the evaluation of criminal justice programs