An Introduction to Culturally Responsive Evaluation for Healthy Marriage and Relationship Education Programs

Deana Around Him*, Rachel Powell*, Lindsay Anderson**, and Sydney Briggs**

Download brief
Download supplementary worksheet

Overview

Evaluation is a process that involves the systematic collection, analysis, and use of information to understand and improve the effectiveness of a program’s services and activities. Culturally responsive evaluation (CRE) is a framework that aligns evaluation efforts with a program’s values, beliefs, and context. CRE helps ensure that evaluation results accurately reflect how the program’s services and activities support the achievement of relevant outcomes for the intended population.

Culturally responsive evaluation (CRE) is defined as “a theoretical, conceptual, and inherently political position that includes the centrality of and to culture in the theory and practice of evaluation. That is, CRE recognizes that demographic, sociopolitical, and contextual dimensions, locations, perspectives, and characteristics of culture matter fundamentally in evaluation.”¹

Healthy marriage and relationship education (HMRE) programs are designed to equip youth and adults with skills that can help them develop and maintain healthy relationships.² The populations served by HMRE programs vary widely; therefore, programs use various curricula and techniques to meet participants’ diverse strengths and needs related to healthy relationships. CRE can help account for the variety of populations served and the outcomes of interest to HMRE programs by engaging a range of perspectives in the evaluation process. HMRE evaluation teams composed of program staff, external evaluators, program participants, and community members can collect these varied perspectives. When HMRE evaluation teams prepare to conduct an evaluation, it is important that they consider more than just the technical steps of the evaluation, especially in settings that are culturally diverse or focused primarily on marginalized populations (e.g., Black, Indigenous, and people of color; couples and individuals who identify as lesbian, gay, bisexual, transgender, queer, or questioning; economically disadvantaged families). For example, evaluation teams should consider the historical and structural factors that contribute to the marginalization of certain groups within the community and the implications of these conditions for HMRE program content, outcomes, and evaluation activities.

HMRE evaluation teams can apply the CRE framework to inform the design and implementation of evaluations in various contexts to strengthen the relevance, trustworthiness, and, ultimately, the usefulness of evaluation findings. This brief provides an overview of the foundational concepts of CRE, describes why these concepts matter for HMRE programming and evaluation, demonstrates how CRE can be applied in HMRE program evaluations, and provides more information about other evaluation approaches that complement a CRE approach.

CRE’s foundational concepts foster a deeper understanding of program culture, context, and power dynamics

At its core, CRE asserts that an evaluation should not be separated from the social and cultural context within which a program is implemented. The importance of CRE can be viewed from the perspectives of validity (being logically or factually sound), ethics (alignment with moral principles), privilege (special rights or advantages), and power (capacity or ability to direct or influence). When applying a CRE framework (Figure 1), HMRE evaluation teams should acknowledge that each community and program has its own culture and context that should be considered when thinking through and completing each step of the evaluation. To understand the culture and context, it is important that community and program voices are present in and central to all steps of the evaluation process. With CRE, the evaluation process becomes shared between all partners, leading to more inclusive efforts and valid results.

CRE encourages evaluation teams to reflect on “who participates, under which conditions they participate, to what extent they participate, and whose knowledge frames the evaluation and is considered of most worth,” as well as whose agenda(s) are served by the evaluation.⁴ CRE seeks to recognize historically marginalized populations by bringing more balance to the evaluation process.¹ As such, applying the CRE framework in HMRE program evaluations could result in intentional engagement of priority populations such as couples who are economically disadvantaged, noncustodial and custodial single parents, or youth aging out of the foster care system or involved in the juvenile justice system.²

A benefit that CRE brings to evaluation practice is a critical focus on power dynamics and assumptions within the evaluation process. CRE acknowledges that power relations shape all aspects of evaluation and encourages evaluation teams to reflect on their own cultural values and beliefs. With increased awareness of the perspectives team members bring to the evaluation process, the evaluation team can collectively grapple with and attempt to mitigate differences in power among themselves and among those who participate in the evaluation.^4,5 For example, funders of HMRE programs may prioritize outcomes different from the couples or individuals who participate in HMRE programs, requiring evaluation teams to consider how multiple interests can be addressed in an evaluation plan and how constraints on the evaluation’s scope and scale can be transparently communicated across partners.

Ideally, attending to the social and cultural context of a program should be part of the entire evaluation process, from planning and design (steps 1 through 6) to data collection, analysis, and use of the results (steps 7 through 9). However, an evaluation team’s ability to thoroughly consider a program’s social and cultural context at each step of the evaluation process will vary based on time, capacity, and resources. Before beginning the CRE process, HMRE evaluation teams should carefully consider the evaluation team’s and community partners’ interest and readiness to implement CRE as well as the time and budget needed for the approach. Engaging in CRE can be rewarding and beneficial to all partners, but it is also a process that can become burdensome when there are unclear or frequent demands of community members’ time and/or they are not appropriately compensated for their contributions to evaluation activities.

That said, abundant time and funding are not required to apply a CRE approach. Evaluation teams choosing to implement CRE should work together to define and carry out an evaluation plan that incorporates the varying perspectives of partners, including program participants and community members, in ways those partners feel are appropriate and meaningful. For example, evaluation teams may need to do most of the data analysis and create clear, structured touch points for community members to contribute to interpreting the data. Then the evaluation teams should be open to revising analyses or running additional analyses based on community input. In instances with more limited resources and capacity, an advisory board with representatives who hold relevant lived experience can be formed to guide evaluation activities. When using CRE, HMRE evaluation teams should be transparent about the evaluation activities that are not flexible (e.g., requirements to measure specific outcomes due to funding source) as they engage partners and other community members who may have an interest in the evaluation.⁶ Evaluation teams can also integrate, or more deeply engage, CRE at any point to strengthen and improve an ongoing evaluation.

Building relationships and gaining trust with the community throughout the evaluation process itself can facilitate a deeper understanding of how the cultural norms of the program, and the individuals it serves, relate to a program’s success.³ In addition, HMRE evaluation teams equipped with CRE knowledge and skills can facilitate a more nuanced understanding of program activities and create opportunities to engage the community in program improvement efforts.^1,7 HMRE program staff and evaluators should seek opportunities to apply CRE to the evaluation planning and implementation processes.

CRE centers the culture and context of HMRE programs
and participants to identify factors that support achieving
program outcomes

HMRE programs promote knowledge, skills, attitudes, and behaviors associated with developing and maintaining healthy relationships.⁸ Studies support the positive impact healthy relationships can have on an individual’s physical, emotional, and psychological health⁹ and the ways HMRE programming can improve relationship quality for individuals and couples.^10,11 Yet, few studies have examined how the cultures of HMRE programs and the participants they serve are considered in the delivery of programming or the achievement of key program outcomes. Culture can be understood as “a cumulative body of learned and shared behavior, values, customs and beliefs common to a particular group or society.”¹² For relationships and marriage, culture likely influences many of the observed differences across populations, including partner selection, dating behaviors, and expectations for marriage.^13-18 Therefore, culture should be considered as an important context for understanding how the components of HMRE programming can more effectively and efficiently promote positive relationship and marriage outcomes for participants.

Many health and social service programs—including HMRE programs—have not been developed with specific cultural or demographic groups in mind.¹⁹ HMRE programs typically cover a common set of topics across the diverse participants they serve, including building communication skills, enhancing conflict management, and promoting intimacy.²⁰ However, program providers may intentionally or unintentionally adjust programming and curricula to the language, behavior patterns, and cultural values of the participants they serve. These adjustments, based on context or culture, are often made without formal documentation of what changed. As a result, measuring the full impact of these adaptations can be challenging, which, in turn, may lead evaluations to miss critical programmatic components driving change. Using CRE creates an opportunity to center program and participant culture and provide context to evaluation results, no matter which topics a given HMRE program covers. Understanding these adjustments through CRE helps to identify factors that may support achieving program outcomes so that they can be considered within the evaluation and, if helpful, carried forward in future programming to achieve maximum results for participants.

HMRE evaluation teams can use guiding questions aligned
with each step in the CRE framework to plan and implement evaluation activities

This section of the brief provides HMRE evaluation teams an explanation of and guiding questions for each step in the CRE framework. The steps and guiding questions apply across many different evaluation types from process evaluation to outcome and impact evaluation. The information may help evaluation teams identify ways to use CRE when a comprehensive approach is not feasible or when a program wants to add CRE to an ongoing evaluation process. In HMRE programs where cultural responsiveness has been an essential part of program planning and implementation, CRE will be especially important for understanding whether and how those culturally responsive program elements shape program and participant outcomes. Evaluation teams can use the guidance provided for each CRE framework step to design and conduct a culturally responsive evaluation. The guiding questions may help evaluation team members discuss, strategize, and document how they will implement each step.

HMRE evaluation teams can download this worksheet to support CRE planning.

CRE Steps and Guiding Questions for HMRE Program Evaluation^a

STEP 1: Prepare for the evaluation

Examine the context of the HMRE program, including the characteristics of the organization, its location and community history, and the people involved. Begin forming an evaluation team with the skills and competencies needed for basic evaluation and the lived experiences that promote genuine connection with the local context.

• What are the stories of the HMRE program and community? Who is telling them, and what are the communication and relational styles represented in them, especially those that highlight formal and informal power dynamics?
• What dimensions of diversity are relevant for the HMRE program and essential for the evaluation team? Whose voices are missing?
• How can external evaluation team members respectfully enter the evaluation partnership?

STEP 2: Engage partners

Identify potential partners who have a deep understanding of the community’s norms, values, traditions, and beliefs. Partners should include those directly and indirectly impacted by the HMRE program and those with differing status, power, and resources. Solidify a diverse group of partners who will formally join the evaluation team or serve in clearly defined supporting roles. Establish meaningful roles and activities for all partners (e.g., that match skills and contribute to the evaluation goals), especially community members with an interest in the program (e.g., HMRE program participants), and engage all partners in ways that cultivate and model trust and respect.

• What may potential partners perceive as the likely benefits of participating in the HMRE evaluation, and what concerns are they likely to have? How can this information inform initial outreach, training needs, and sustained engagement of partners?
• Which potential partners are uniquely equipped to support specific components of the evaluation (e.g., educating the evaluation team about the HMRE program’s history, the community’s context, developing appropriate and respectful data collection strategies)?
• How will HMRE program participants be engaged, and what resources or trainings must be in place to support their engagement?
• What are the appropriate ways to acknowledge partners’ time and contributions to the project (e.g., compensation, titles, authorship)?

STEP 3: Identify the purpose(s) of the evaluation

Define the motivation and aspirations for the evaluation and consider how the stated purpose maintains or challenges existing power dynamics within the HMRE program and community context. Consider the priorities of different members of the evaluation team (including all partners) and how to navigate potential conflicts.

• Why is the evaluation being considered? Is it required by an external entity (e.g., funder) or being conducted as an internally motivated effort to improve implementation or outcomes of the HMRE program?
• Is continuation or expansion of the HMRE program, or components of the program, contingent on the evaluation results?

STEP 4: Frame the right questions

Draft, revise, and refine the evaluation questions with partners and agree which questions to prioritize with the available resources.

• Are the questions aligned with the purpose of the evaluation?
• Are the focus and wording of questions appropriate for the culture and context of the HMRE program and community? For example, is it important to have strengths- versus deficit-oriented questions? Will answering the questions lead to actionable next steps?
• Are the questions attentive to the perspectives and experiences of the HMRE program recipients?
• Will the intent conveyed in the questions potentially limit or expand what can be learned from the evaluation?

STEP 5: Design the evaluation

Decide on the types and sources of information that will be collected, how the information will be collected, on what time frame, and how it will be analyzed. Consider the questions the evaluation seeks to answer, the types of evidence partners and community members with an interest in the HMRE program will value, and the cultural values of the HMRE program.

• What types of information will be collected to answer the evaluation questions? Do partners and other community members interested in the HMRE program value and use these measures and sources of information?
• Does the timeline allow time to build relationships with partners and key community members interested in the HMRE program? Does the timeline allow for engaging these individuals in the evaluation steps?
• Do elements of the evaluation design (e.g., randomization, quasi-experimental) align with the values, perspectives, and context of the community?
• How will the evaluation benefit the community served by the HMRE program?

STEP 6: Select and adapt instrumentation

Assess whether existing instruments, such as scales, are valid to use with the cultural group(s) in the HMRE program. Consider whether instruments can be adapted or translated to be culturally specific or whether developing original instruments would be more meaningful.

• What populations were used to validate existing instruments and in what context?
• Does the evaluation team have the resources and expertise to develop original instruments?
• What form of data collection instruments should be used with HMRE staff and participants? For example, are written surveys appropriate or is oral data collection preferable?
• Is the language and the order of questions or topics in data collection instruments appropriate for HMRE participants, the HMRE program, and the community context?

STEP 7: Collect the data

Learn how culture influences with whom to speak during data collection, in what order to speak with people, and the type of relationships necessary for information to be shared freely. Consider how evaluators’ identities and lived experiences may influence their perception and interpretation of all information. Determine whether data collectors can be trained in the relevant cultural dynamics and context of the HMRE program or whether lived experience is necessary for data collection activities.

• How can evaluators build relationships, establish trust, and promote valid understanding with individuals providing information?
• Who should collect the data? What training or experiences do data collectors need to understand the cultural context of the HMRE program?
• How can burden on participants be reduced (e.g., length of time commitment/questions, use of familiar software platforms)?

STEP 8: Analyze the data

Involve partners and other cultural interpreters from the community in data analysis to promote accurate understanding of the information within the cultural context. Go beyond average outcomes by considering meaningful ways to disaggregate data and explore what can be learned from positive and negative outliers and even unintended outcomes.

• Who will be involved in analyzing and interpreting the data, and how will they be engaged? Will sessions be held with program participants to interpret and make meaning of the data? Are trainings needed to promote capacity building and ethical use and interpretation of the evaluation data?
• What subgroup analysis will be most meaningful for the HMRE program?
• What structural, historical, or current contextual factors may contribute to differences observed for subgroups?

STEP 9: Disseminate and use the results

Integrate partner and community reviews, including reviews by HMRE program participants, to provide perspectives on findings before developing dissemination products. Consider which communication formats are most appropriate for sharing findings with a broad range of audiences, especially HMRE program participants and other community members with an interest in the HMRE program. When making dissemination decisions, prioritize options that have the potential to benefit the program and community that participated.

• How will the knowledge and products being disseminated benefit the HMRE program and community?
• Which dissemination formats and styles (e.g., brief, report, video, town hall) are most appropriate for different audiences and evaluation participants?
• Are the languages and terminology used in dissemination products appropriate for the intended audience(s)?
• What role should the evaluation team and other community partners have in disseminating evaluation findings and products?

Understanding approaches that complement CRE may help HMRE evaluation teams develop shared language and a strong foundation for new evaluation activities

Since its beginnings, the field of CRE has expanded and come to align with several related and complementary approaches in research and evaluation. Some of these approaches may be familiar to members of HMRE evaluation teams and include participatory research, collaborative evaluation, equitable evaluation, deliberate democratic evaluation, and critical qualitative inquiry. Independently, each approach provides a unique perspective that complements CRE. While it is outside the scope of this brief to detail each complementary approach, exploring the subtle differences and nuances they add to promoting a focus on culture and equity in examining a research or evaluation question may be of interest and support the use of shared language and terminology for embarking on new evaluation activities. See the appendix for approaches complementary to CRE.

Note that equity, or the quality of fairness and justice, is such a growing part of CRE that some scholars and organizations use culturally responsive and equitable evaluation (CREE) in the place of CRE. Beyond the CRE framework described here, CREE seeks to incorporate “cultural, structural, and contextual factors (e.g., historical, social, economic, racial, ethnic, gender) using a participatory process that shifts power to individuals most impacted.”²¹ As the evaluation field continues to incorporate CRE, it is important to note this intersection with other terminology in addition to those mentioned in the appendix.

Key points for integrating CRE in HMRE program evaluations

To fully benefit from CRE, HMRE evaluation teams should include program staff, evaluation partners, and other partners who collectively reflect the program’s culture and context and the skills necessary to complete the evaluation activities. The HMRE evaluation team should maintain an awareness of program and participant culture throughout the evaluation process. Resources, staff capacity, and other factors may limit the ability to employ the CRE framework thoroughly; however, recognizing these limitations can help prioritize areas where CRE may be most helpful for an HMRE program’s evaluation approach. By facilitating an understanding of what will be most useful to the HMRE program being evaluated, CRE can help HMRE evaluation teams understand the activities and curricula that work best for the families served in HMRE programs. CRE may also improve understanding of an HMRE program’s ability to produce desired outcomes across diverse groups of families. As HMRE evaluation teams form, partners should discuss the role they would like CRE to have in their evaluation activities. This brief introduces CRE, and HMRE program staff and evaluation partners interested in applying this approach are encouraged to visit the following additional resources.

Additional resources

To learn more about CRE and how it can be applied to strengthen the evaluation process, see:

• The Center for Culturally Responsive Evaluation and Assessment (CREA): CREA, located at the University of Illinois at Urbana-Champaign, consists of scholars and practitioners who integrate cultural responsiveness into evaluation, assessment, policy analysis, applied research, and action research.
• Considerations for Conducting Evaluation Using a Culturally Responsive and Racial Equity Lens: Public Policy Associates developed this resource, a practical guide to conducting evaluation from a culturally responsive and racial equity lens.
• Practical Strategies for Culturally Competent Evaluation: This resource for program staff and evaluators highlights the role of culture and shares important strategies to complement other evaluation resources offered by the Centers for Disease Control and Prevention’s Division for Heart Disease and Stroke Prevention.
• Practical Tools for Designing and Implementing Culturally Responsive and Inclusive Evaluations:
  Learning for Action created a series of checklists focusing on topics such as cultural humility to be used as tools to incorporate an equity lens in the evaluation process.
• Engaging Community Members in the Research Process to Enhance Healthy Marriage and Responsible Fatherhood Program Evaluations: This brief from the Fatherhood, Relationships, and Marriage—Illuminating the Next Generation of Research (FRAMING Research) project presents strategies that researchers can use to engage community members as co-creators in the research process.

Footnotes

^* Equally contributing authors
^** Equally contributing authors
^a The framework and guiding questions presented here are adapted from Hood, S., Hopson, R., & Kirkhart, K. (2015). Chapter 12: Culturally responsive evaluation: Theory, practice, and future implications. In Newcomer, K., Hatry, H., & Wholey, J. (4th ed.) Handbook of Practical Program Evaluation, p 287-296. https://nasaa-arts.org/wp-content/uploads/2017/11/CRE-Reading-1-Culturally-Responsive-Evaluation.pdf

APPENDIX: Approaches complementary to CRE