Chad Sansing
When we talk about internet health—or a healthy internet—we're talking about several interrelated issues: privacy and security, openness, digital inclusion, web literacy, decentralization, and how to leverage these issues for good on behalf of the internet and its users. The internet is like an ecosystem that needs all of us pulling together to maintain its health and sustainability.
And that ecosystem needs leaders. More specifically, a healthy internet needs open leaders.
Open leaders design and build projects that empower others to collaborate within inclusive communities.
That's the core belief of the Open Leadership and Events (OLE) team at the Mozilla Foundation. In our work to spread the principles, practices, and skills of open leadership, we aim to design and build programs that empower participants to take the lead on open projects about data, internet health, and the impact those things have on both their local and distributed communities.1 We serve leaders from around the world and across a wide range of open, for-profit and non-profit, and technical and non-technical projects.
Participants in OLE programs come from all over the open ecosystem; they include people working on open data, open educational resources (OER), open government, open hardware, open science, open software (F/OSS), privacy and security for open practitioners, and more. For example, in 2018, Mozilla's Global Sprint (a 2-day marathon of contribution to open projects) included projects and communities like these:
- Rust
- P5js
- Reading for Gender Bias
- Mission: Information
- Wikipathways
As we support a diverse set of open leaders like these—and collaborate with them to increase the breadth and depth of open's impact on the world—we're especially concerned with engagement and understanding the life-cycles of our participants' relationships with Mozilla and openness in general. We believe that one way to improve our work and better align it with participants' wants and needs is to follow their lead. Where do they begin their relationships with us? Why do they participate and what do they want from their participation? What are their typical next steps after a first engagement? In what kinds of participation do they engage over time, and do those engagements always deepen? Do people contribute more or less to us as we invest more in them? What are our returns on both high-touch and light-touch programs, and are those returns equitable for participants? What do they get from us in the value exchanges that underlay our community interactions? Do they consistently "graduate," so to speak, to run their own projects or programs after event X or training Y?
To put it another way: How can following our participants' patterns of engagement lead us to better understand and improve our work in supporting them?
To answer that question, we started building a participation index (PI) called the "Mountain of Engagement" (MoE). The MoE is meant to be both a methodology and measurement tool for defining and tracking meaningful interactions with our team so that we can follow individuals and groups of participants and surface patterns in their engagement with OLE programming. Those patterns help us identify our most and least successful programs. From there, we can make decisions about how to improve our work.
We want to help open leaders find helpful pathways to professional development and success in their own organizations, projects, and communities. We also want to improve our programs to make sure we are meeting community members' needs.
This is the story of how we developed the MoE to help us do those things. It's also the story of what we've learned so far and what we might do next. We hope that by following a similar methodology, you can develop a participation index unique to your organization that helps you strengthen its engagement with open leaders in your community.
Our MoE drew inspiration from the Total Engagement Index (TEI). In 2017, the advocacy team at the Mozilla Foundation developed the TEI and its dashboard in order to track how people on our mailing list interacted with emails and other campaign channels. Vojtech Sedlak and Brett Gaylor led the work at Mozilla in consultation with Harmony Labs. The MoE is also an extension of the well-known pyramid of engagement developed by Groundwire.2
The TEI grouped engagements into different bands or categories of action by depth of involvement. Each engagement in a particular band added a certain number of points to the TEI, and the advocacy team tracked the total engagement points it earned each month on an internal dashboard.
The TEI used these bands to group different kinds of engagements:
- Owning: Actions taken by allied individuals and organizations to promote internet health, like launching their own internet health campaign or project
- Leading: Actions taken to partner with Mozilla on internet health issues, like partnering on a campaign or event 3. Contributing: Actions taken to support Mozilla's internet health work, like donations, project contributions, and amplifying Mozilla content on social media
- Endorsing: Actions taken to spread Mozilla's internet health work, like signing petitions and liking and sharing updates on social media
- Observing: Actions taken to learn about Mozilla's internet health work, like visiting a campaign website or opening a campaign email.
An Observing engagement might have earned the team a fraction of a point, while each Leading or Owning might be worth a full point on its own. By totaling the scores of every engagement in a given month, the advocacy team could track an aggregate score representing its impact.
The TEI is no longer a primary inspiration for the foundation, but it gave the OLE team a framework for examining our own work and identifying the key forms of participation and engagement we want to track across our year-long cycle of leadership trainings and events from our Open Leaders program through the Global Sprint and MozFest. Our work also draws on research from Mozilla's Open Innovation team, which explores being "open by design" and fostering community interactions and value exchanges in open projects.3 Other key elements of the MoE come from team members' experiences with open science initiatives, working open workshops, web literacy trainings, research done for the Open Leadership Framework,4 and Mozilla's work to champion openness as an internet health issue.5
What follows is a description of the steps we took to develop our MoE.
This is an intentional, mindful methodology we developed especially for smaller projects and communities that need to track engagement differently than a traditional, technical analytics team does. We hope that by following a similar process, you can identify the types of engagement that matter most to you and your participants. Once you know which engagements matter most, you can work to improve them and better scaffold pathways between them.
To begin developing our MoE, first we asked: What do we do? We wanted to synthesize and capture a holistic view of our work taken from each team member's perspective to minimize the chance we'd overlook something important that we do, albeit infrequently or implicitly. We identified major programs like Open Leaders,6 the Global Sprint,[^mozilla-global-sprint], open science mini-grants,7 and MozFest,8 as well as less visible pieces of work like developing curriculum, maintaining a social media presence, researching open leadership practices, and speaking at conferences.
Then we asked: How do people engage with us? Looking at all we do, we listed the different types of engagements people could have with each area of work. For example, someone might be an attendee, facilitator (presenter), or wrangler (organizer) at MozFest. Someone else might follow one of our social media accounts, retweet or share a post, or clap for a blog.
Next, we asked: How might we band, group, or sort these types of engagements? We decided to use these tiers and descriptors:
- Leading: A high-touch relationship; we maintain relationships and co-branded events and trainings with alumni and allies to increase the impact, prestige, and reach of both parties' work.
- Collaborating: A high-touch relationship; we offer professional development through our own events in return for co-creation, localization, and spread.
- Participating: A high-touch relationship; we offer community management and professional development through our own trainings and events in return for soliciting ideas and learning through use.
- Endorsing: A low-touch relationship; we share information with people who gain social capital by spreading it and networking with others who share common interests.
- Learning: A low-touch relationship; we gift resources like open curriculum and get back aggregate data (like downloads, registrations, and views) showing people use our resources and pay attention to us.
After that, we asked: What does our Mountain of Engagement look like? Figure 1 shows the graphic we made to illustrate our MoE.
{insert Figure 1} Figure 1: The OLE team's MoE, CC BY 4.0 by Mozilla
Once we had our visualization of the MoE, we asked: How can we operationalize this or make this more useful? In response, we developed a summary document (Figure 2) that helped us connect each band to examples, scores, and the types of community interactions and value exchanges that might show up within each level of engagement.
{insert Figure 2} Figure 2: Summary chart of the draft OLE MoE, CC BY 4.0 by Mozilla
In the same document, we experimented with multipliers that aligned with our team and foundation's goals. For example, if were especially interested in engagements from particular places or groups of people, we might double their scores to draw our attention to their engagements and the pathways they took between them.
We wanted to know:
- What works and what doesn't work for different groups of participants.
- How participants move from one engagement to another or get stuck between them.
- How we might systematize the ways we recognize participants and invite them to deeper levels of engagement with us over the lifetime of our relationship with them.
With those prompts in mind, we asked ourselves: Where should we focus our attention? Given our capacities, goals, and interests, we decided to focus on engagements at the Participating, Collaborating, and Leading levels of the MoE. While we blog and tweet and send newsletters, most of our time is spent designing, implementing, and improving high-touch open leadership programs like Open Leaders, the Global Sprint, and MozFest. We worried that tracking tweets and retweets and likes and opens would distract us from supporting those programs and their participants, especially since we were piloting and testing this approach before importing it in to a customer relationship management (CRM) tool. Although we're curious about how our participants reach the Participating level in the first place, we think we can ask them about their journeys as they move up the MoE; we don't need to follow them from the first time they visit our website.
Furthermore, we don't compile an aggregate score like the TEI did; instead, we follow individuals' scores and reach out with specific communications and invitations to people and groups that cross different thresholds of engagement with us. For example, we might send an invitation to get involved with a program or event at the Collaborating level to anyone who earns 5 or more points at the Participating level.
By giving them most of our attention and support, we can empower them to co-create programming with us and then launch their own communities, organizations, and projects in support of internet health.
And that is where we are today.
We're listening to people who participate in programs like Open Leaders, the Global Sprint, open science mini-grants, and MozFest so we can improve those offerings, clarify the pathways between them, and empower alumni to launch and sustain their own open internet health projects in the future. Our ongoing question is: What do we do with the data we collect?
While you and your community, organization, and project might be more concerned with another part of the open ecosystem—such as data, government, or software—we hope that this process (and the questions it raises) will help you understand when, where, and how to empower your participants to broaden and deepen their engagement with you and the interests you share.
To summarize, here are the questions we asked ourselves while developing the MoE:
- What do we do?
- How do people engage with us?
- How might we band, group, or sort these types of engagements?
- What does our Mountain of Engagement (MoE) look like?
- How might we operationalize this or make it more useful?
- Where should we focus our attention?
- What do we do with the data we collect?
First, a quick note on data: you should follow all the laws, policies, and rules that cover data collection, retention, and use for you and your participants. We always encourage people to adopt data privacy policies that are as strong as Mozilla's.9
Let's take a look at some early data analysis we've done of participants' engagements with OLE programs during the first half (H1) of 2018. We want to share these observations to suggest how a MoE might help lead our team—and yours—answer new questions and develop new pathways and programs to support participants.
In looking at the 1,954 participants who completed an engagement with us between January and June, 2018, we found that:
- 65% came from outside the United States (US) and a "virtual" participant group (for which we have not geographic data).
- 61% completed an engagement at the Participating level of our MoE.
- 61% participated in the 2018 Global Sprint.
- 6% participated in cohort 5 of our Open Leaders program.
- 5% participated in our open science mini-grant application process.
Engagement data like this can lead us towards:
- Confirming or correcting our ratio of participants from the US and from outside the US so we can sustain or develop inclusive, global participation from diverse communities in our programming. We can also shift strategic focus to look at engagement from groups within the US whom we have not yet effectively or sufficiently invited and welcomed into these programs.
- Confirming or correcting our ratio of Participating engagements and engagements at other levels of the MoE to establish a baseline or benchmark for growth in Collaborating and Leading engagements.
- Examining the similarities and differences between audiences and opportunities across programs to apply best practices, as well as audience-specific invitations, to all of our trainings and events.
- Investigating why discrete projects—like developing the Open Leadership Framework— engage far fewer participants than on-going programs do.
We also looked at participation and scores from all participants who completed 2 or more engagements with us during the same time period. We found these outcomes:
- 6% of total participants engaged with 2 or more OLE programs at the Participating level or higher in H1 of 2018.
- 71% of these participants came from outside the US.
- 76% participated in the 2018 Global Sprint.
- 53% participated in cohort 5 of our Open Leaders program.
- 17% participated in our open science mini-grant application process.
- 4% participated in all 3 programs.
- 54% participated in the Global Sprint and Open Leaders.
- 7% participated in the Global Sprint and open science mini-grant application process.
- 1% participated in Open Leaders and the open science mini-grant application process.
Here we might ask ourselves how participants move between programs and how to encourage more consistent engagement across multiple programs. Is there a way to connect the open science mini-grant application to another program or sequence of programs? Are there pieces of our invitation to the Global Sprint and pieces of support for participants that we can adapt for other programs? What motivates participants to engage in multiple OLE programs? What makes the combination of Global Sprint and Open Leaders so popular with repeat participants?
With MozFest 2018 and the 2019 Global Sprint coming up later this year and early next, we can return to these questions with new data and a larger sample of participants and engagements to help us answer questions like these.
Looking back at the last few years of engagement data we have for our Open Leaders program, we found a drop off in how many past participants returned as mentors for new participants in the next cohort. Between rounds 1 and 3, that conversion rate went from 70% down to 55%. The number of open leaders willing to stick with us and to move from the participating band of our MoE to the collaborating band dropped by double digits.
However, by paying attention to that pathway of engagement in OLE programming between Participating (being a mentee) and Collaborating (becoming a mentor) on the MoE, we were able to identify this problem quickly and work to keep more participants engaged between rounds 3 and 4. By adding additional mentor training as a form of professional development and support for returning participants, we brought the percentage of past participants willing to stay on as collaborators back up to 72%. However, as we continued to develop and grow the program, that number dropped back down to 50% when we looked at participants from cohort 5 who returned as mentors for cohort 6.
Something is happening as we scale up the number of participants in the Open Leaders program. By using our MoE to focus our attention on key results like the pathway and conversion rate from participant to mentor, we can respond to issues like these and calibrate our work to fit our leaders' needs. We can also see how many participants submit a proposal to MozFest or return to the next Global Sprint and ask whether or not these opportunities take the place of continued engagement with Open Leaders for some participants.
Maybe becoming a mentor isn't the best fit for continued engagement after participating in Open Leaders. Or maybe becoming a mentor is the best fit for continued engagement after participating in a different offering. The MoE gives us the opportunity to consider such possibilities and to look for successes and challenges across our programs.
As we iterate on that mentor training further, our goal after cohort 6 is a 90% conversion rate. By refining our programs, we can make each one better at graduating participants to the next level of our MoE and then, ideally, out into the internet health movement as the leads of their own communities, organizations, and projects.
Paying attention to how people engage with us helps us improve our programs, clarify the pathways between them, and refine invitations we share to participate in them
You might use your own Mountain of Engagement—and the data you collect through it—to set similar goals for your community, organization, or project.
In pursuit of a world-class open leadership program, we let engagement lead the way. We want to empower participants to shape our work according to their needs and in support of a healthy internet for all.