Outcomes and Outcome Domains Examined in HMRE Evaluation Studies

Sydney J. Briggs, April Wilson, Mindy E. Scott, and Deja Logan

Download brief

Overview

Healthy marriage and relationship education (HMRE) programs aim to promote healthy, stable relationships and marriages, thereby enhancing the quality and stability of participants’ lives.^a HMRE programs also promote relationship stability as a way to improve economic security for families.¹ Further, research demonstrates that the quality and stability of adults’ romantic relationships can translate into improved well-being for their children.² As such, the Administration for Children and Families (the primary funder of HMRE programs) regards HMRE programs as a strategy to promote economic self-sufficiency and strengthen families.¹

HMRE programs aim to impact the attitudes, knowledge, and behaviors of individuals, couples, families, and even communities.³ These programs typically teach adult and youth participants skills that research suggests are important for developing and maintaining healthy relationships. Such skills include communicating effectively, managing conflict, building intimacy, and identifying signs that a relationship may be dangerous or unhealthy.^b,4,5 To know whether these programs create the change they intend—on romantic relationships and, by extension, on family and child well-being—evaluation research assesses the impact of HMRE programs on measurable indicators across a range of relevant outcome domains.

This brief describes the types and prevalence of outcomes measured in evaluations of HMRE programs over the last decade at the individual, couple, family, and community levels. The brief ends with a discussion about the potential implications of these findings, as well as various considerations for evaluators and practitioners when selecting outcomes for HMRE evaluations.

Key Findings

MAST Center researchers reviewed 117 HMRE evaluation studies for this brief. In total, these evaluations assessed 693 separate outcomes, including outcomes on participants’ individual attitudes, beliefs, and knowledge, and their romantic and family relationships; in a few cases, evaluations examined how HMRE programs affect the broader community or society. Among the 693 outcomes identified, approximately:

• 27 percent (n=188) focused on the program participant as an individual
• 51 percent (n=354) focused on the participant’s romantic relationship
• 20 percent (n=135) focused on the participant’s family unit
• 2 percent (n=16) measured community- or society-level outcomes

Participant outcomes

We found that individual participant outcomes were assessed across four domains: relationship attitudes and knowledge, physical and psychological well-being, reproductive health attitudes and knowledge, and parenting attitudes and knowledge. Most individual participant outcomes focused on participants’ attitudes, beliefs, and knowledge about romantic relationships generally (55%), or their psychological well-being (31%). Fewer participant-level outcomes considered reproductive health and family planning (9%) or parenting attitudes (5%).

Romantic relationship outcomes

Romantic relationship outcomes were assessed across two domains: relationship quality and relationship status or stability. Relationship quality was the most frequently considered outcome domain (n=323; 91% of romantic relationship outcomes). Relationship quality is comprised of outcomes such as conflict and communication, global assessments of quality and satisfaction, and intimate partner violence, among others. Relatively few romantic relationship outcomes in these evaluations were focused on romantic relationship status and stability (9%).

Family outcomes

Family outcomes were assessed across six domains: parenting, coparenting, family economic outcomes, child development, child physical and psychological well-being, and other family outcomes. Parenting outcomes were the most common type of family-level outcome assessed (n=41; 30%) and child well-being outcomes were the least frequently assessed (n=5; 4%).

Community and society outcomes

Outcomes rarely focused on the participant’s social networks, communities, or societal contexts. As such, this level is not subdivided into additional domains.

Methods

The research team identified HMRE program evaluations and implementation studies conducted in the United States, written in English, and published from 2008 to 2019. We searched multiple research databases using the search terms “healthy marriage and relationship education,” “couple relationship education,” “marriage education,” and “relationship education.” In addition, we reviewed the bibliographies and references of meta-analyses and more recent articles to identify studies for inclusion. We also reviewed the Office of Planning, Research, and Evaluation (OPRE) website, as well as curriculum developer websites, to locate evaluation reports. In total, this review included 146 publications, of which 117 were program evaluations that met the parameters to be coded and analyzed for the present brief.

The research team compiled all outcomes measured in the 117 evaluations and then developed a coding scheme to organize various types of outcomes. Outcomes were coded according to an adapted socio-ecological framework and according to categories that emerged within each level of the framework. The outcome categories became domains and subdomains in the framework presented in this brief. To begin, we used an inductive approach to coding, creating categories from the outcomes rather than applying a predetermined set of categories to the outcomes.⁶ Then we assessed, organized, and defined the categories, which allowed us to categorize the outcomes consistently. In total, 693 outcomes were identified and coded. See Appendix A for a more detailed description of methods and Appendix B for a full list of evaluations included in this brief.

A Conceptual Framework for Organizing Healthy Marriage and Relationship Education Outcomes

As described above, HMRE programs aim to influence participant outcomes at multiple levels, including the attitudes, knowledge, and behavior of individuals, couples, families, and even communities. As such, a conceptual framework for organizing HMRE outcomes must show the connections between the multiple levels of outcomes that HMRE programs seek to influence. To organize outcomes, we selected an adaptation of Bronfenbrenner’s socio-ecological model, which focuses on the idea that individuals affect and are affected by their close relationships and their communities.^7,8 Our framework begins with outcomes focused on individual HMRE participants, then moves outward to those focused on participants’ relationships (both romantic and within their families), and finally, to a broader level of outcomes focused on participants’ social networks, community, and society (see Figure 1). Each level within the model influences, and is influenced by, each other level. This model allows us to organize HMRE outcomes, which often originate from what an HMRE program hopes to achieve, in a way that reflects the multiple systems at work in each couple’s and individual’s life.^c

We classified all outcomes using this framework by identifying the focus of an outcome:

Participant

Outcomes that assess an individual participant’s attitudes, beliefs, and well-being, such as attitudes toward parenting, beliefs about marriage, or depressive symptoms. Participant outcomes were organized into four domains, which are defined in Table 1.

• Relationship attitudes, beliefs, and knowledge
• Physical and psychological well-being
• Reproductive health attitudes, beliefs, knowledge, and behaviors
• Parenting attitudes, beliefs, and knowledge

Relationships

Outcomes in this level assess two categories of close relationships:

Romantic relationship: Outcomes that focus on specific romantic relationships, such as relationship status, communication between romantic partners, or partners’ feelings of support and affection. Romantic relationship outcomes were organized into two domains with seven subdomains, which are defined in Table 2.

• Romantic relationship status or stability
• Romantic relationship quality
  • Communication and conflict
  • Global rating of relationship quality or satisfaction
  • Intimate partner violence (IPV)
  • Commitment to and confidence in relationship
  • Other feelings about partner or relationship
  • Non-communication behaviors
  • Specific areas of (dis)satisfaction or (dis)agreement

Family: Outcomes that focus on relationships and behaviors within the family or the well-being of the family unit, such as a parent’s approach to discipline, the family’s level of economic stress, or the child’s level of anxiety.^d Family outcomes were organized into six domains, which are defined in Table 3.

• Parenting
• Coparenting
• Family economic outcomes
• Child development
• Child physical and psychological well-being
• Other family outcomes

Community and Society

Outcomes beyond the family, such as connections to social supports, use of community resources, the local divorce rate, or state-level child poverty rates. Community and society-level outcomes were not subdivided into domains.

Outcomes Considered for HMRE Evaluations

In total, the 117 evaluations reviewed for this brief assessed 693 separate outcomes. Among the 693 outcomes identified, 27 percent (n=188) focused on the program participant as an individual, 51 percent (n=354) focused on the participant’s romantic relationship, 20 percent (n=135) focused on the participant’s family unit, and 2 percent (n=16) measured community- or society-level outcomes.^e Key findings for each framework level are described below and results are presented in Tables 1-4. See Appendix C for a full table of the framework and counts by framework level, domain, and sub-domain.

Key findings: participants

Evaluations often included participant- (individual-) level outcomes (n=188). We grouped these outcomes into attitudes, beliefs, and knowledge about (a) romantic relationships and parenting; (b) physical and psychological well-being; and (c) parenting. We also grouped attitudes, beliefs, knowledge, and behaviors about (d) reproductive health and family planning (see Table 1).

Most participant-level outcomes considered individual participants’ attitudes, beliefs, and knowledge about romantic relationships or their psychological well-being. Among participant-level outcomes, 104 considered participants’ attitudes, beliefs, and knowledge about romantic relationships generally. This domain includes outcomes such as “faulty” relationships beliefs, attitudes toward dating violence, and confidence in managing conflict. Outcomes related to participants’ physical and psychological well-being were also commonly considered (n=59)—especially outcomes that considered whether participants experienced feelings of depression or anxiety. Mental health both affects and is affected by one’s romantic, parenting, and coparenting relationships, which may explain why many evaluations included this outcome.^9,10Additional outcomes of participant well-being included self-esteem, individual functioning, and quality of life.

Among participant-level outcomes, evaluations least frequently considered attitudes, beliefs, and knowledge about parenting or reproductive health. Instead, when evaluations included parenting outcomes, they tended to focus on parenting behaviors and interactions (part of the family area of the relationship level). The few outcomes related to family planning or reproductive health (n=16) focused on contraceptive use or beliefs around postponing sexual activity.

Key findings: romantic relationships

The romantic area of the relationship level contains outcomes related to both the quality and stability of participants’ current romantic relationships.

Romantic relationship outcomes were the most frequently considered HMRE outcomes— especially those that considered the quality of the relationship between the romantic partners. The emphasis on couples’ romantic relationships makes sense given that HMRE curricula most often target couples.¹¹ Approximately half of all outcomes (n=323) analyzed for this brief considered some aspect of romantic relationship quality. As such, we further sorted these measures across important sub-domains. Prior research has conceptualized and measured romantic relationship quality in many ways.^12,13 For this brief, we classified relationship quality into seven sub-domains that emerged from coding the outcomes. These sub-domains are frequently associated with healthy (or unhealthy) relationships and are shown in Table 2, along with their frequency as evaluation outcomes.¹⁴

• Among the sub-domains of relationship quality, the most commonly considered outcomes were communication and conflict (n=112; 35%) and global ratings of relationship quality and satisfaction (n=82; 25%).
• Even though HMRE programs are not considered intimate partner violence (IPV) interventions, and some programs screen out couples who indicate IPV,¹⁵ IPV was often assessed as an outcome (n=47; 15%).
• The skills and knowledge participants gain in HMRE programs may increase their confidence in their relationships; even the act of attending a program with a partner may enhance feelings of commitment. However, relatively few outcomes captured participants’ levels of commitment or confidence (n=34; 11%).
• Other aspects of relationship quality were less frequently considered, including other feelings about the partner, such as trust, affection, and friendship (n=27; 8%); noncommunication behaviors, such as infidelity and time spent together (n=10; 3%); and specific areas of (dis)satisfaction and (dis)agreement, such as finances, sex, and role orientation (n=11; 3%).

HMRE programs often seek to promote relationship stability among participants, yet relatively few (n=31) HMRE outcomes considered relationship status following program participation. Typically, evaluations that included relationship status outcomes examined marital stability or dissolution (i.e., separation or divorce), or whether participants continued living together or got married.

Key findings: family relationships

The family area of the relationship level includes outcomes on parenting, coparenting, family economic security, and a group of other outcomes that relate to family interaction but do not clearly fall into any outcome domain. Child outcomes are also included in the family level.

Family outcomes frequently assessed parenting and coparenting interactions. HMRE programs aim to improve outcomes for children,¹⁶ so it seems reasonable that many strive to improve parenting and coparenting behaviors, regardless of whether the parents are in a romantic relationship. The 41 outcomes focused on parenting typically included parental stress, parental engagement with their
children, and parents’ approaches to discipline. A separate domain of 23 coparenting outcomes focused on quality and conflict in coparenting relationships.

Family-level outcomes also considered family economic well-being. These outcomes (n=28) focused on families’ use of government assistance, their ability to pay for housing costs, and parents’ employment status. Including family economic outcomes in evaluations may make sense when programs include services in addition to relationship education, such as support and referrals from family coordinators in Building Strong Families, or employment services in programs in the Parents and Children Together evaluation.^17,18 The inclusion of family economic outcomes in HMRE evaluations is aligned with HMRE programs’ goal of supporting families’ self-sufficiency.¹

Although HMRE programs aim to improve child well-being, relatively few outcomes focused on children. As described above, children typically do not participate in HMRE programs, but programs often hope that children benefit indirectly through increased family stability, improved parenting practices, and parents’ enhanced relationships. Nonetheless, only 3.8 percent (n=26) of outcomes considered child well-being. Within the outcomes that focused on children, most (n=21) assessed child development, including internalizing behaviors, externalizing behaviors, and social competence. Fewer outcomes (n=5) considered children’s physical or psychological well-being.

Key findings: community and society

HMRE outcomes rarely considered social supports, other community factors, or broader societal trends. HMRE programs generally involve individual- or couple-level interventions; however, “advocates for these programs have stressed the eventual goal of society-wide change.”¹⁹ As such, a small minority of outcomes (n=16) assessed changes in whether participants talked about their relationship with others in their social networks, in state-level rates of divorce, or in household structure. Community- and society-focused outcomes tended to be restricted to evaluations, such as the Community Healthy Marriage Initiative, or to state-level analyses that assessed community- and societal-level effects of HMRE programs.^3,20

Implications of HMRE Evaluation Outcomes

This brief provides an inventory of the outcomes considered in evaluation studies of HMRE programs, as well as a framework for organizing HMRE outcomes. Knowing the types of outcomes HMRE evaluations measure helps evaluators and practitioners with the following:

• Assessing the extent to which the outcomes measured align with the diverse populations served in HMRE programs, including married adult couples, unmarried couples, single adults, youth and young adults, coparents, and stepfamilies
• Assessing whether the outcomes measured align with what programs and curricula focus on and aim to change
• Shedding light on the outcomes that evaluators prioritize or expect programs to change and identify the types of outcomes that are missing from evaluations

Evaluation outcome alignment with populations served in HMRE programs

HMRE programs serve increasingly diverse families with distinct needs.²⁰ Therefore, HMRE curricula are sometimes tailored for different family structures, such as unmarried couples and stepfamilies, as well as for different age groups, like youth and young adults.^11,21,22 When selecting outcomes to include in an HMRE evaluation, it is important to ensure alignment with the needs of the program population. If the populations served by a program shift, the outcomes measured also may need to change. For instance, outcomes focused on contraceptive use and beliefs around postponing sexual activity seem particularly relevant for youth and young adults. Similarly, evaluations of those not currently in a romantic relationship may include outcomes related to general romantic relationship attitudes and knowledge, such as healthy/unhealthy relationships beliefs, attitudes toward dating violence, and confidence in managing conflict.^23,24 Outcomes focused on the quality of coparenting relationships are relevant for both parents in a romantic relationship and those who are not (but not relevant if a program does not engage parents).

Implications for practitioners

To align evaluation outcomes with service populations, consider creating a participant profile with the relevant goals, values, beliefs, and desired outcomes of your population(s). Sit down with your staff and answer the following:

• • What are the characteristics of the population(s) you serve? (For example, gender, age, cultural norms, prior experiences of relationship violence, mental health or substance use challenges, etc.)
  • What do participants need/want/value?
  • What common challenges do participants face?

Once you better understand the participants you serve, you can align your program design, including outcomes, with that population(s).

Evaluation outcome alignment with program curricula and goals

When evaluators select outcomes, it is also important to consider curricula focus and program goals to ensure that the outcomes used in an evaluation align with what the program aims to accomplish. Although best practices in evaluation research promote this type of alignment,²⁵ in practice, outcomes also may be included in evaluations due to funding priorities, evaluators’ own assessment of important outcomes, and/or the ease of measuring these outcomes.

Fortunately, there is at least some alignment between the outcomes considered in HMRE evaluations and the topics most commonly addressed in HMRE curricula. Recent research from the MAST Center documents that the most common topics in HMRE curricula include (1) conflict management, resolution, and communication skills; (2) parenting and/ or the transition to parenthood; (3) recognizing relationship danger signs, warning signs, and destructive relationship patterns; (4) commitment; and (5) self-awareness.¹¹ Our review of outcomes in evaluations shows varying levels of alignment with these topics:

• Conflict management, resolution, and communication skills. In this category, there is strong alignment between the topics on which curricula focus and the outcomes assessed in evaluations. The most frequently assessed outcomes were related to communication and conflict (n=112).
• Parenting and/or the transition to parenthood. Less alignment exists in this category given that relatively few outcomes focused on parenting interactions (n=41) and very few focused on parenting knowledge (n=9).
• Recognizing relationship warning signs and destructive relationship patterns. Few outcomes assessed participants’ knowledge of relationship warning signs, but many assessed healthy communication skills, destructive conflict patterns (communication and conflict, n=112), and IPV (n=47). There is a possibility that evaluators (and/or programs) assume that changes in behavior also indicate an increased understanding of destructive relationships.
• Commitment. HMRE curricula emphasize relationship commitment and stability, but relatively few outcomes (n=34) focused on participants’ commitment to or confidence in their relationships.
• Self-awareness. Little alignment exists in this category, as self-awareness was not a common evaluation outcome.

Implications for practitioners

Talk to evaluators early about evaluation priorities and brainstorm outcomes to measure. Consider mapping these potential outcomes against the main objectives of your curriculum. This activity will help you identify gaps between what the curriculum is trying to accomplish and what outcomes should be measured. Remember, there can be competing considerations when it comes to selecting evaluation outcomes: funding priorities, evaluators’ assessment of what’s important, program goals, and measurement ease. It is important to strike a balance between these different priorities.

Outcomes prioritized and missing from HMRE evaluations

HMRE programs focus on supporting healthy romantic relationships and marriage, and this emphasis is reflected in the outcomes HMRE evaluations consider. Of the 693 outcomes included in these analyses, half (n=354) focused on current romantic relationships and were broadly categorized as outcomes related to relationship status or relationship quality.

The most common outcomes assessed within romantic relationship quality were communication and conflict. This finding may reflect the fact that communication and conflict behaviors are both strong predictors of relationship quality and stability, and are also relatively malleable aspects of romantic relationships.^5,11,26 However, previous research demonstrates a variety of factors that predict stable and satisfying relationships (e.g., support provision, gender role congruence, attitude and value similarity, economic stability, and mental health).^4,27 It may be worthwhile for future evaluations to consider some of these relationship features as outcomes, when appropriate to the aims of the program being evaluated.

A relatively small number of evaluations directly assessed parenting outcomes, despite the fact that improving child well-being is an important goal of HMRE programs.²⁸ Programs seek to improve child well-being both by teaching romantic relationship skills to parents with the aim of increasing family stability, and by directly supporting positive parenting practices and the transition to parenthood., Our review found that some parenting behaviors (such as parents’ engagement, daily routines, and discipline) were measured. However, few outcomes assessed parenting knowledge or skills, although research shows that increased parenting knowledge is associated with better parenting practices.^31-33

The most obvious gap found in our analysis was the near absence of outcomes that considered social networks, community impacts, or societal influences of HMRE. Research has found that a lack of social supports is associated with lower relationship quality.³⁴ Furthermore, a recent qualitative study found that alleviation of social poverty is an important benefit of HMRE programs.³⁵ Clearly assessing community- and society-level outcomes is important given that HMRE programs seek a high-level impact. However, it is likely that many evaluations excluded these outcomes because they are difficult to measure in program evaluations, which are designed to capture individual- or couple-level change following program participation. Additionally, community- and society-level outcomes are unlikely to be perceptible unless HMRE is taken to scale because the number of people who participate in HMRE programs will be too small to measure at a city or state level. Given that most programs do not aim to achieve outcomes at the community or society level, it also is reasonable that most evaluations did not focus on these outcomes.

Conclusion and Next Steps

This brief has described a framework for organizing the outcomes considered in evaluations of HMRE programs, the prevalence of outcomes within that framework, and implications for evaluators and practitioners. Another brief in the MAST Center’s series on HMRE evaluation will synthesize findings across evaluations and meta-analyses from the past 10 years to explore which outcomes improve when people participate in HMRE programs. This forthcoming brief will use the framework described here to summarize which outcomes improve (and which do not), which programs have demonstrated success in improving these outcomes, and which implementation features and program components are associated with improved outcomes.

Footnotes

^a In this brief we use the term “healthy marriage and relationship education (HMRE)” to refer to both federally and non-federally funded programs. As it is used here, HMRE is interchangeable with other labels and acronyms used in the field, such as marriage and relationship education (MRE), relationship education (RE), or couple’s relationship education (CRE).^{back to text}

^b See An Overview of Healthy Marriage and Relationship Education Curriculum for a discussion of topics most commonly addressed in HMRE curricula. ^{back to text}

^c Although multiple models or frameworks could be used to organize HMRE evaluation outcomes, we selected the socio-ecological framework because of its capacity to encompass a broad range of models, rather than using a framework drawn from a specific model, such as those used in the Supporting Healthy Marriage or Building Strong Families evaluations. ^{back to text}

^d Although HMRE programs do not typically provide direct services to children, programs often aim to improve parenting (or coparenting) practices and the family context for children. We include child outcomes within family outcomes: if children experience program effects, it is often via changes in their parents’ behaviors or their relationship with their parents. ^{back to text}

^e Of the 116 evaluations included in this brief, most measured more than one outcome and often included multiple outcomes within each of our levels (e.g., participant) or domains (e.g., physical and psychological well-being). As such, there are many more outcomes than evaluations in our analyses. The results reported in this brief represent the total counts of outcomes across all evaluations. Therefore, when we report 188 outcomes at the participant level, that indicates that, of the 693 outcomes coded, 188 were outcomes at the participant level, not that 188 evaluations included outcomes at the participant level. ^{back to text}