Developing Measurement Frameworks

August 20, 2013

The Edge Initiative

Over the past two years I’ve been working with the American Library Association’s Office for Information Technology Policy as a part of a larger coalition of organizations that has been collaborating to develop a management and leadership tool to support continuous improvement and reinvestment in information, communication, and technology services in public libraries across the United States.

Millions of people use public technology at libraries each year to continue their education, find jobs, access e-government services, research health information, connect with family and friends, and more – making this a critical service, especially for those without regular internet access at home. Yet, libraries face both technical and financial challenges in sustaining high quality public technology (computers, wi-fi networks, technology trainings, etc.). Edge, which made the tool available for a soft launch in June 2013, supports library managers in addressing these challenges.

Picture 1

Specifically, Edge offers ways to assess current public technology and its use in the library; helps identify ways to strengthen or enhance this technology and its uses; and engages with key leaders about the value of libraries and public technology in strengthening communities. The development phase of the project has been supported by the Bill and Melinda Gates Foundation.

My role has been to help the Edge team understand best practices in the development, design, and implementation of measurement frameworks such as benchmarks, standards, and rating systems. As part of this work I examined how respected frameworks in several fields – ranging from human rights to sustainable fishing to parks certification emerged, the challenges they’ve faced along the way, and what they do to maintain excellence. This informed the development of the Edge Benchmarks Version 1.0, a keystone of the larger Edge Initiative.

What are measurement frameworks?

Measurement frameworks can help non-profits, businesses, and governments to understand how they relate to others in their field, how well they are progressing towards specific targets, where they are doing well and where they could use improvement – or how their performance changes over time. Framework results – in the form of certifications, ratings, rankings, etc. – can help communicate this information to stakeholders.

In addition, the process of framework development can have important benefits for a field as a whole. It requires experts and practitioners to examine their understanding of the field, come to agreement on key problems, and encapsulate this knowledge in a straightforward way. This leads to greater consensus about what the community’s goals are and can act as a guide for action.

Measurement frameworks give insight on things like:

• progress on a particular goal (for example, air safety or equality in the workplace)
• how a particular organization (such as a library or a software manufacturer) measures up to industry best practices,
• and/or encourage a desired outcome (such as healthy children)

Examples of established measurement frameworks include the U.S. Green Building Council’s rating system for green building – LEED, the Human Rights Campaign’s Corporate Equality Index, Marine Stewardship Council’s certification for sustainable seafood, the AASHE STARS self-reporting framework for sustainability in universities and colleges, the Global Animal Partnership’s 5-Step standards for meat production. and Greenpeace’s Guide to Greener Electronics ranking.

Picture 3      Picture 4

How are measurement frameworks created?

The process of developing measurement frameworks is complex, but can be categorized into three main phases, Development, Design, and Implementation. Time frames for development can vary widely, depending on the complexity of the measure, the intensiveness of the work, and how many pilot tests are undertaken before the release of a first official framework. This process can be as short as six months, though more often falls in the range of two to five years, or even a decade. Even when the first official framework is “ready,” continuous improvement can shift its content, coverage, or approach significantly in later years.

Key practices in developing measurement frameworks include:

Be iterative and seek consensus. It is critical for the framework development team to invite opportunity for the larger community – for whom the framework is being designed – to participate and provide feedback. Some of the best frameworks are developed and maintained in a highly iterative way. Iteration –developing, seeking input, testing, and revising – can happen throughout the entire process, from setting goals, to choosing indicators, to maintaining the framework.

Mirror the way the field already works. Frameworks that build on practices that already are common to a community can get extra leverage, and may be more likely to be adopted. For instance, as a certification system, LEED echoes the way that the building industry already works: all contractors must keep up to speed and comply with building codes, which are enforced and often changing – and LEED is an extension of that.

Present data in the way your stakeholders understand and can use. Frameworks should be designed so that they are understandable to the audience that the frameworks are intended to inform and/or influence.

Be transparent and support benchmark choices with evidence. Transparency practices can help build legitimacy for the framework (e.g. Why should I believe that this framework really embodies best practices that can help my organization? Or, how can I know that organizations rated by this framework are truly high quality?). This may include: making public the names and affiliations of the framework development team and related experts; showing the links between goals, indicators, and data; and even sharing things such as ongoing meeting minutes.

Resources

The following resources give a particularly helpful overview of measurement framework development and related challenges:

1. The ISEAL Alliance, a global membership organization for sustainability standards, provides a number of meta-resources related to standards development. Check out their Emerging Initiatives modules for detailed information on topics such as governance (PDF), stakeholder engagement (PDF) and Conflicts of Interest (PDF) as they relate to standards.

2. This paper, “Measuring outcomes of United Way-funded programs: Expectations and reality” (PDF), discusses and critically evaluates the United Way of America’s efforts to encourage outcome measurement throughout its large network. While this is not an example of measurement framework development per se it documents challenges that standards/benchmarks/self-reporting systems can also encounter,  including: how to train organizations to use a new tool (e.g. benchmarks or outcome measurement), how to develop useful documentation, how to support and encourage organizations to make use of the results, and how the tool can or should relate to funding decisions.

It is also helpful to take a look at the websites and documentation of existing measurement frameworks which, generally, in line with the best practice of transparency named above, are relatively open about their decisions and process.

Picture 1Picture 5Picture 6

Tags: , , , , ,

Evaluation, Projects, Social Impact