Motivating Our Students Through Experience implements framework and survey instrument to track scholar success

Since 1986, Los Angeles-based nonprofit Motivating Our Students Through Experience has offered mentoring and programs designed to help young women achieve their dreams of graduating from college. With a goal of having all of its high school seniors graduate from college within six years, MOSTe contributes to the development of women who are confident, career-focused agents of social change. In 2023, MOSTe and Measurement Resources Company partnered together to develop an Impact Measurement Framework and survey to track the nonprofit’s mission success over time.

“We had benchmark data confirming that our work is effective,” said Amy Ludwig, executive director, MOSTe, “but we didn’t have any day-to-day data to understand how our mentoring was helping our scholars. More recently, we had shifted our goals to include community building and mental health support, and we had no mechanism in place for measuring those activities.”

Today, MOSTe serves approximately 100 female K-12 students annually, across five middle schools and 30 high schools in Los Angeles County. The team also maintains regular contact with an additional 100 female alumni scholars currently attending college. Prior to engaging with Measurement Resources, MOSTe collected basic demographic data, with a fall, mid-year, and end-of-year questionnaire.

While the MOSTe team knew anecdotally how student participation was leading to program participants feeling more confident, more connected, and more able to reach their goals, the program had difficulty quantifying the change in those perceptions over time. Measurement Resources’ research team examined the sum of the organization’s inputs as a single human centered equation, and helped to clarify its short-, medium-, and long-term goals. They also identified the right metrics for tracking participants’ perceptions and growth in these areas.

“A common fear for many small nonprofits is bandwidth — having the bandwidth to administer and analyze the survey,” said Amy. “Measurement Resources was realistic about this and built measurement into existing touch points, making it all very manageable. We were able to hire an intern to analyze the data we collected, and now we have data points that both demonstrate our impact and identify opportunities to fine tune our questions going forward.”

Like most organizations, MOSTe pivoted its approach during the COVID-19 pandemic to provide more mental health supports. While programming is in person again, the nonprofit will continue to enhance and refine this focus on self-advocacy and community building. The mission has remained constant and the organization is financially secure with a solid mix of funding sources, but how the organization operates in this ever-shifting environment and make the best use of resources will continue to evolve. In 2024, MOSTe will continue to focus on rebuilding its post-pandemic student recruitment pipeline. Nationally, volunteering has also fallen off since the pandemic, as nonprofits are still trying to coax people out of their nests and back into the community space.

In 2023, MOSTe’s results using the new survey revealed many positive results. But the results also helped the staff realize that they need to better understand how students define community as a whole. To clarify this from a data collection standpoint, the team has decided to break this broad category out into smaller buckets, such as school, home, and family, and then separately measure attitudes about the MOSTe community. Additionally, MOSTe will iterate on questions around participant goal setting, to ensure that the responses reflect participants’ own perspectives and expectations of themselves, as opposed to external pressures and expectations.

“That’s what I liked about the language in the survey instrument,” said Amy. “It conveys that our students know how to find and access the resources they need. We’re trying to help build that muscle – so that our students know they are part of a community, and when the chips are down, they know what to do. That they have existing bridges and they know how to build more. So that’s where we’re focusing our growth as an organization. Not just that they are A students, but they are individuals who build community and make change, and that they are empowered to do that.”

For the 2022-2023 academic year, MOSTe’s new survey results revealed the following key success takeaways:

• Scholars’ Awareness of Resources: The average awareness of resources across multiple items increased by 7%, from 4.1 to 4.4 on a five-point scale. For high school scholars who scored above a predetermined success threshold cross all items increased from 71% to 85%.
• Scholars’ Engagement in Campus, Civic, or Leadership Activities: 83% of high school scholars spend some time each month on extracurricular engagement activities across all survey items in Q3 of the scholar check-in. The average engagement across all items increased by 5% over the academic year.
• Trajectory of Alumni: 57% of alumni have already obtained either an undergraduate or graduate degree. 41% are currently in the process of completing either an undergraduate or graduate program.

“It’s incredible to see what our students go on to do, despite the headwinds,” said Amy. “One of our students has just finished her first year of college. She’s on a full scholarship at Wesleyan University, and then this summer, she got a full scholarship to the Middlebury Language Schools to learn Arabic. We’re such a small organization, but this process was doable. And this data is what’s going to help us to continue to serve our students better, and help us secure the funding to do it.”