Evaluation Design

The design phase is where groups make specific decisions about what information will be collected, how, and when. In an ideal situation, these decisions would be driven by the questions the group has chosen to ask, and the level of specificity, surety and timeliness needed to answer the questions. In the real world, evaluation design usually considers those issues, but final decisions are often made based on available capacities and resources for evaluation, the kinds of information that key audiences believe matters, the difficulty or ease of getting data from various people, groups and systems, and other real world issues. This is a key step to watch for privilege influencing the work. Privilege infects evaluation when these real world issues override the integrity of evaluation – often not with that intent, but with that impact. This happens when, for example, communities or programs are judged as “failures” based on looking for results too soon, but reportin [...]

Resources

Title
Author
Organization
Evaluating Comprehensive Community Change
The Annie E. Casey Foundation
Getting Smart, Getting Real: Using Research and Evaluation Information to Improve Programs and Policies
Leila Fiester et al.
The Annie E. Casey Foundation
We Did it Ourselves: An Evaluation Guidebook (Chapter 11)
SRI International; Sierra Health Foundation
Ways to Improve the Quality of Your Program Evaluations
University of Wisconsin Cooperative Extension
Evidence-Based Practices, Practice-Based Evidence and Community Defined Evidence in Multicultural Mental Health
Ken Martinez
American Institutes for Research
Greater Portland Pulse: Challenges to Measuring Racial Equity
Meg Merrick
Institute of Portland Metropolitan Studies, Portland State University
Evaluating Complexity: Propositions for Improving Practice
Hallie Preskill, Srik Gopal, Katelyn Mack, and Joelle Cook
FSG
Raising the Bar – Integrating Cultural Competence and Equity: Equitable Evaluation
Jara Dean-Coffey, Jill Casey, and Leon D. Caldwell
jdcPartnerships; ABFE; The Foundation Review
Medicine Wheel Evaluation Framework
Atlantic Council for International Cooperation
Conducting Culturally Competent Evaluations of Child Welfare Programs and Practices
Alan J. Detlaff and Rowena Fong
Child Welfare
Evaluation of Capacity Building: Lessons from the Field
Deborah Linnell
Alliance for Nonprofit Management
How to Design and Manage Equity-Focused Evaluations
Michael Bamberger and Marco Segone
UNICEF Evaluation Office
LGBT-Inclusive Language in Data Collection
Jen Hsu
LGBT Student Services, Western Michigan University
Universal Design Evaluation for Checklist
Jennifer Sullivan Sulewski and June Gothberg
Historical Trauma and Unresolved Grief: Implications for Clinical Research and Practice with Indigenous Peoples of the Americas
Maria Yellow Horse Brave Heart, PhD
Native American & Disparities Research, Center for Rural & Community Behavioral Health
How Do We Approach Impact and Evaluation in the Context of Scale?
Grantmakers for Effective Organizations, Scaling What Works
Collaborating to See All Constituents Reach Their Full Potential: Memorandum on Research and Resources on Equity and Collective Impact
Collective Impact Forum
Moving Toward Equity: Stakeholder Engagement Guide ... in Designing Educator Equity Plans
Center on Great Teachers & Leaders (GTL) at American Institutes for Research
Discussion Guide: Theory of Social Change
Building Movement Project

Tools

Title
Author
Organization
Processes and Indicators for Measuring the Impact of Equality Bodies
Niall Crowley
Equinet (European Network of Equality Bodies)
Advocacy Evaluation Mini-Toolkit: Tips and Tools for Busy Organizations
Learning for Action Group
Equal Access Participatory Monitoring and Evaluation Toolkit
J. Lennie, J. Tacchi, B. Koirala, et al.
Equal Access Nepal; Equal Access International; Queensland University of Technology; University of Adelaide, Australia
4 of 5 - Considering Culture: Building the Best Evidence-Based Practices for Children of Color
Putnam Consulting Group, et al.
The Annie E. Casey Foundation
Evaluation Handbook
W.K. Kellogg Foundation
Logic Model Development Guide
W.K. Kellogg Foundation
Avoid These Mistakes in Your Program Evaluation
University of Wisconsin Cooperative Extension
The Advocacy Progress Planner
The Aspen Institute Advocacy Planning and Evaluation Program
Community Development Evaluation Storymap and Legend
Lynn Bachelor and Margaret Grieve
NeighborWorks America

“If a community has to be evaluated by a foundation, then the foundation has to be evaluated by the community. This is scary for foundations, and difficult for communities.”

~ Barbara Major, Racial Justice Activist,
Flipping the Script: White Privilege and Community Building

SPOTLIGHT

Leading for Equity – Gita Gulati-Partee, TEDxUNCSalon

Also in this section:
  • Evaluation Plan

GLOSSARY

Cultural Appropriation

Theft of cultural elements—including symbols, art, language, customs, etc.—for one’s own use, commodification, or profit, often without understanding, acknowledgement,or respect for its value in the original culture. Results from the assumption of a dominant (i.e. white) culture’s right to take other cultural elements.


SOURCE:  Colours of Resistance Archive, “Cultural Appropriation (accessed 28 June 2013).

Cultural Racism

Cultural racism refers to representations, messages and stories conveying the idea that behaviors and values associated with white people or “whiteness” are automatically “better” or more “normal” than those associated with other racially defined groups. Cultural racism shows up in advertising, movies, history books, definitions of patriotism, and in policies and laws. Cultural racism is also a powerful force in maintaining systems of internalized supremacy and internalized racism. It does that by influencing collective beliefs about what constitutes appropriate behavior, what is seen as beautiful, and the value placed on various forms of expression. All of these cultural norms and values in the U.S. have explicitly or implicitly racialized ideals and assumptions (for example, what “nude” means as a color, which facial features and body types are considered beautiful, which child-rearing practices are considered appropriate.)


SOURCE:  RacialEquityTools.org


Related Resources:  Racism (scroll down alphabetically to the box for “Cultural Racism”)

Location: FUNDAMENTALS / Core Concepts

Ethnicity

A social construct that divides people into smaller social groups based on characteristics such as shared sense of group membership, values, behavioral patterns, language, political and economic interests, history, and ancestral geographical base.


Examples of different ethnic groups are: Cape Verdean, Haitian, African American (Black); Chinese, Korean, Vietnamese (Asian); Cherokee, Mohawk, Navaho (Native American); Cuban, Mexican, Puerto Rican (Latino); Polish, Irish, and Swedish (White).


SOURCE:  Teaching for Diversity and Social Justice: A Sourcebook, edited by Maurianne Adams, Lee Anne Bell, and Pat Griffin, Routledge, 1997.


Related Resources:  Race, Ethnicity, and Indigeneity (scroll down alphabetically to the box for “Ethnicity”)

Location: FUNDAMENTALS / Core Concepts

Inclusion

Authentically bringing traditionally excluded individuals and/or groups into processes, activities, and decision/policy making in a way that shares power.


SOURCE:  OpenSource Leadership Strategies

People of Color

Often the preferred collective term for referring to non-White racial groups. Racial justice advocates have been using the term “people of color” (not to be confused with the pejorative “colored people”) since the late 1970s as an inclusive and unifying frame across different racial groups that are not White, to address racial inequities. While “people of color” can be a politically useful term, and describes people with their own attributes (as opposed to what they are not, e.g., “non-White”), it is also important whenever possible to identify people through their own racial/ethnic group, as each has its own distinct experience and meaning and may be more appropriate.


SOURCE:  Race Forward, “Race Reporting Guide” (2015).

Racial and Ethnic Identity

An individual’s awareness and experience of being a member of a racial and ethnic group; the racial and ethnic categories that an individual chooses to describe him or herself based on such factors as biological heritage, physical appearance, cultural affiliation, early socialization, and personal experience.


SOURCE:  Teaching for Diversity and Social Justice: A Sourcebook, edited by Maurianne Adams, Lee Anne Bell, and Pat Griffin, Routledge, 1997.


Related Resources:  Race, Ethnicity, and Indigeneity (includes “Resources by Specific Racial/Ethnic Groups”)

Location: FUNDAMENTALS / Core Concepts

Structural Racialization

Structural racialization connotes the dynamic process that creates cumulative and durable inequalities based on race. Interactions between individuals are shaped by and reflect underlying and often hidden structures that shape biases and create disparate outcomes even in the absence of racist actors or racist intentions. The presence of structural racialization is evidenced by consistent differences in outcomes in education attainment, family wealth, and even life span.


SOURCE:  Systems Thinking and Race: Workshop Summary by john a. powell, Connie Cagampang Heller, and Fayza Bundalli (The California Endowment, 2011).


Related Resources:  Structural Racism

Location: FUNDAMENTALS / Core Concepts

Whiteness

  1. The term white, referring to people, was created by Virginia slave owners and colonial rules in the 17th century. It replaced terms like Christian and Englishman to distinguish European colonists from Africans and indigenous peoples. European colonial powers established whiteness as a legal concept after Bacon’s Rebellion in 1676, during which indentured servants of European and African descent had united against the colonial elite. The legal distinction of white separated the servant class on the basis of skin color and continental origin. The creation of ‘whiteness’ meant giving privileges to some, while denying them to others with the justification of biological and social inferiority.

  2. Whiteness itself refers to the specific dimensions of racism that serve to elevate white people over people of color. This definition counters the dominant representation of racism in mainstream education as isolated in discrete behaviors that some individuals may or may not demonstrate, and goes beyond naming specific privileges (McIntosh, 1988). Whites are theorized as actively shaped, affected, defined, and elevated through their racialization and the individual and collective consciousness formed within it ... Whiteness is thus conceptualized as a constellation of processes and practices rather than as a discrete entity (i.e. skin color alone). Whiteness is dynamic, relational, and operating at all times and on myriad levels. These processes and practices include basic rights, values, beliefs, perspectives, and experiences purported to be commonly shared by all but which are actually only consistently afforded to white people.


SOURCE:

1. PBS, “Race: The Power of an Illusion” (2018–2019 relaunch of 2003 series).

2. Robin DiAngelo, “White Fragility” (International Journal of Critical Pedagogy, 2011).


Related Resources:  System of White Supremacy and White Privilege

Location: FUNDAMENTALS / Core Concepts