Introduction

Outcomes are an invaluable management tool, but like all performance measures, they inevitably have limitations. Awareness of these limits should inform programs’ use of outcomes.

Outcomes are only one of the data types programs need to assess and improve client services and operational effectiveness and efficiency.

To effectively assess and enhance their operations and client services, programs need a wide range of data. Information about the type, volume, and scope of services (outputs) and the results of those services (outcomes) are both needed to effectively evaluate the impact of client services.

Further, effectively understanding the needs of the client community and the efficacy of different delivery approaches and advocacy strategies may include the analysis of information such as:

• Census data and data from other federal and state sources regarding the demographic and economic characteristics of the low-income population in the service area
• Court records and data regarding the services and operations of other governmental and private entities serving low-income people
• On-going engagement with client and community groups
• Surveys and interviews with staff and community members
• Reports by academic institutions, policy groups, and research organizations
• GPS or similar data to align services with the needs of the client community
• Other relevant data

All Measures are Imperfect

Every outcome measure is based on assumptions that can limit their validity and value. Programs should consider the following interrelated questions:

1. Are we measuring the right things? To what degree are they aligned with the program’s mission and goals as well as the types of issues addressed in particular cases?
2. Are the outcome measures based on appropriate indicators? For example, to what degree do civil protection orders “ensure the safety of domestic violence victims”?  Research shows they reduce the likelihood that victims will be subjected to further violence by the same abuser but, as advocates know too well, perpetrators may violate these orders and subject clients to further violence. This is not to say that securing protection orders is not critically important; they certainly are. The point is that one must be aware of and transparent about the shortcomings of the outcome measures used and consistently strive to improve them.
3. Are the formulas used to calculate financial benefits valid and appropriate?
4. Are the outcomes measures framed in ways to limit subjectivity? Consider the extent to which data quality could be undermined by the subjective judgments of the persons that identify the outcomes of particular cases.
5. Consider the data entry process: are staff entering data into the system accurately?

A broader issue is that a wide range of interrelated factors—social, economic, political, juridical, and others—influence the outcomes of programs' work. That all outcomes measures are imperfect is not a problem in and of itself.  Rather, programs must be aware of and transparent about the underlying assumptions and inherent limitations of the data on which particular outcome measures are based.

Many Aspects of Programs’ Work Cannot be Meaningfully Quantified

These include, but are not limited to:

• The quality of an advocate’s casework
• The value and impact of the program’s “other services” (e.g., pro se clinics, community legal education, self-help materials on websites)
• The impact of the program’s work on a client’s sense of well-being (e.g., from avoiding homelessness, escaping from an abusive partner) or empowerment
• If and how the program’s work increases the accountability and accessibility of the justice system and government entities to the low-income population
• The impact of the program’s work with community organizations

Beware of Unintended Consequences

The use of outcomes and other performance measures can have adverse unintended consequences. In particular, the outcome measures may end up driving a program’s work. The measures a program employs define—implicitly, if not explicitly—what is and what is not important. The measures used will inevitably influence decisions about the types of cases accepted and the types of representation provided. This could cause the program or individual staff to “cherry pick” cases that can be easily measured and result in high success rates but which may not address the most important legal needs of the client population.

Ongoing Evaluation and Improvement of Outcomes Systems Are Essential

Once a program establishes an outcomes measures system, program staff must regularly assess the system’s value and efficiency. Program assessment and improvement are ongoing processes. The outcomes and other data and systems a program uses to assess its effectiveness must likewise be continually evaluated and modified to best meet the program’s needs.