Skip to content

Commit

Permalink
chore: remove references to old studies platforms and update some for…
Browse files Browse the repository at this point in the history 
… nimbus (#858)

* chore: remove references to old studies platforms and update some for nimbus

* lint/spelling
  • Loading branch information
@mikewilli
mikewilli authored Jan 22, 2025
1 parent 25c0c76 commit 14c3b83
Show file tree
Hide file tree
Showing 4 changed files with 52 additions and 225 deletions.
160 changes: 1 addition & 159 deletions src/cookbooks/client_guidelines.md
View file
Original file line number Diff line number Diff line change
@@ -1,161 +1,3 @@
# Client Implementation Guidelines for Experiments

**Note**: This guidance is useful for implementing Normandy experiments. To ship experiments with the Nimbus platform, please see the guidance for engineers at <https://experimenter.info/>.

There are three supported approaches for enabling experimental features for Firefox:

- [Firefox Prefs](#prefs)
- Prefs can be used to control features that **land in-tree**.
[Feature Gates](#feature-gates) provide a wrapper around prefs that can be used from JavaScript.
- [Firefox Extensions](#extensions) AKA "**Add-ons**".
- If the feature being tested should not land in the tree, or if it will ultimately ship as an extension, then an extension should be used.

New features go through the standard Firefox review, testing, and deployment processes, and are then enabled experimentally in the field using [Normandy][normandy-docs].

## Prefs

Firefox Preferences (AKA "prefs") are commonly used to enable and disable features. However, prefs are more complex to implement correctly than [feature gates](#feature-gates).

**Each pref should represent a different experimental treatment**. If your experimental feature requires multiple prefs, then Normandy does not currently support this but will soon. In the meantime, an [extension](#extensions) such as [multipreffer][multipreffer-docs] may be used.

There are three types of Prefs:

1. Built-in prefs - shipped with Firefox, in `firefox.js`.
2. `user branch` - set by the user, overriding built-in prefs.
3. `default branch` - Overrides both built-in and `user branch` prefs. Only persists until the browser session ends, next restart will revert to either built-in or `user branch` (if set).

[Normandy][normandy-docs] supports overriding both the `user` and `default` branches, although the latter is preferred as it does not permanently override user settings. `default` branch prefs are simple to reset since they do not persist past a restart.

**In order for features to be activated experimentally using `default branch` prefs**:

- The feature must not start up before `final-ui-startup` is observed.

For instance, to set an observer:

```js
Services.obs.addObserver(this, "final-ui-startup", true);
```

In this example, `this` would implement an `observe(subject, topic, data)` function which will be called when `final-ui-startup` is observed. See the [Observer documentation][observer-docs] for more information.

- It must be possible to enable/disable the feature at runtime, via a pref change.

This is similar to the observer pattern above:

```js
Services.prefs.addObserver("pref_name", this);
```

More information is available in the [Preference service documentation][pref-service-docs].

- Never use `Services.prefs.prefHasUserValue()`, or any other function specific to `user branch` prefs.

- Prefs should be set by default in `firefox.js`

If your feature cannot abide by one or more of these rules (for instance, it needs to run at startup and/or cannot be toggled at runtime) then experimental preferences can be set on the `user branch`. This is more complex than using the methods described above; user branch prefs override the users choice, which is a really complex thing to try to support when flipping prefs experimentally. We also need to be careful to back up and reset the pref, and then figure out how to resolve conflicts if the user has changed the pref in the meantime.

## Feature Gates

A new Feature Gate library for Firefox Desktop is now available.

**Each feature gate should represent a different experimental treatment**. If your experimental feature requires multiple flags, then Normandy will not be able to support this directly and an [extension](#extensions) may be used.

### Feature Gate caveats

The current Feature Gate library comes with a few caveats, and may not be appropriate for your situation:

- Only JS is supported.
- Always asynchronous.

Future versions of the Feature Gate API will include C++/Rust support and a synchronous API.

### Using the Feature Gate library

Read [the documentation][feature-gate-docs] to get started.

## Extensions

Firefox currently supports the [Web Extensions API](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions).

**If new WebExtension APIs are needed, they should land in-tree**. Extensions which are signed by Mozilla can load privileged code using the [WebExtension Experiments](https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html), but this is not preferred.

WebExtensions go through the same correctness and performance tests as other features. This is possible using the Mozilla tryserver by dropping your XPI into `testing/profiles/common/extensions` in `mozilla-central` and pushing to Tryserver - see the [Testing Extensions](#testing-extensions) section below.

NOTE - it is ideal to test against the version of Firefox which the extension will release against, but there is a [bug related to artifact builds on release channels][artifact-bug] which must be worked around. The workaround is pretty simple (modify an `artifacts.py` file), but this bug being resolved will make it much simpler.

**Each extension can represent a different experimental treatment (preferred), or the extension can choose the branch internally**.

### SHIELD studies

The previous version of the experiments program, SHIELD, always bundled privileged code with extensions and would do things such as mock UI features in Firefox.

This sort of approach is discouraged for new features - land these (or the necessary WebExtension APIs) in-tree instead.

For the moment, the [SHIELD Study Add-on Utilities](https://github.com/mozilla/shield-studies-addon-utils/) may be used if the extension needs to control the lifecycle of the study, but using one extension per experimental treatment makes this unnecessary and is preferred. The APIs provided by the SHIELD Study Add-on Utilities will be available as privileged APIs shipped with Firefox soon.

# Development and Testing

## Testing Built-in Features

Firefox features go through standard development and testing processes. See the [Firefox developer guide][firefox-dev-docs] for more information.

## Testing Extensions

Extensions do not need to go through the same process, but should take advantage of Mozilla CI and bug tracking systems:

1. Use the Mozilla CI to test changes (tryserver).
2. Performance tests (**this step is required**) - extension XPI files should be placed in `testing/profiles/common/extensions/`, which will cause test harnesses to load the XPI.
3. Custom unit/functional tests (AKA `xpcshell`/`mochitest`) may be placed in `testing/extensions`, although running these tests outside Mozilla CI is acceptable so these are **optional**.
4. Receive reviewer approval. A Firefox peer **must sign off** if this extension contains privileged code, aka WebExtension Experiments.

- Any [Firefox Peer][firefox-peer-list] should be able to do the review, or point you to someone who can.

5. Extension is signed.
6. Email to `[email protected]` is sent to request QA
7. QA approval signed off in Bugzilla.
8. Extension is shipped via [Normandy][normandy-docs].

## Example Extensions Testing Workflow

Note that for the below to work you only need [Mercurial][mercurial] installed, but if you want to do local testing you must be set up to [build Firefox][firefox-build-docs]. You don't need to build Firefox from source; [artifact builds][firefox-artifact-build] are sufficient.

In order to use Mozilla CI (AKA "[Tryserver][try-server-docs]"), you must have a full clone of the `mozilla-central` repository:

```bash
hg clone https://hg.mozilla.org/mozilla-central
cd mozilla-central
```

Copy in unsigned XPI, and commit it to your local Mercurial repo:

```bash
cp ~/src/my-extension.xpi testing/profiles/common/extensions/
hg add testing/profiles/common/extensions/my-extension.xpi
hg commit -m "Bug nnn - Testing my extension" testing/profiles/common/extensions/my-extension.xpi
```

Push to Try:

```bash
./mach try -p linux64,macosx64,win64 -b do -u none -t all --artifact
```

This will run Mozilla CI tests on all platforms

Note that you must have Level 1 commit access to use tryserver. If you are interested in interacting with Mozilla CI from Github (which only requires users to be in the Mozilla GitHub org), check out the [Taskcluster Integration proof-of-concept][taskcluster-integration-poc].

Also note that this requires an investment time to set up just as CircleCI or Travis-CI would, so it's not really appropriate for short-term projects. Use tryserver directly instead.

[feature-gate-docs]: https://firefox-source-docs.mozilla.org/toolkit/components/featuregates/featuregates/index.html
[firefox-dev-docs]: https://firefox-source-docs.mozilla.org/
[mercurial]: https://www.mercurial-scm.org/
[taskcluster-integration-poc]: https://github.com/biancadanforth/taskcluster-integration-poc/
[firefox-build-docs]: https://firefox-source-docs.mozilla.org/setup/index.html
[firefox-artifact-build]: https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html
[try-server-docs]: https://firefox-source-docs.mozilla.org/tools/try/
[observer-docs]: https://searchfox.org/mozilla-central/rev/59e797b66f5ce8a27ede0e7677688931be7aed20/xpcom/ds/nsIObserverService.idl#24-39
[normandy-docs]: https://github.com/mozilla/normandy
[pref-service-docs]: https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefService.idl
[multipreffer-docs]: https://github.com/nhnt11/multipreffer
[firefox-peer-list]: https://wiki.mozilla.org/Modules/All#Firefox
[artifact-bug]: https://bugzilla.mozilla.org/show_bug.cgi?id=1435403
To ship experiments with the Nimbus platform, please see the guidance for engineers at <https://experimenter.info/getting-started/engineers/for-engineers>.
23 changes: 13 additions & 10 deletions src/cookbooks/normandy_events.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
# Working with Normandy events

A common request is to count the number of users who have
enrolled or unenrolled from a SHIELD experiment.
<div class="warning">
Normandy and SHIELD are older experimentation platforms
at Mozilla, but are no longer in use. However, Normandy
events are still in use (as of January 2025) by Jetstream
for computing enrollments and exposures. Note that this
will likely change in the near future because these events
have been ported to Glean and legacy telemetry is being
deprecated across the platform.

For more up to date information on events used by Nimbus,
see <https://experimenter.info/telemetry>.

</div>

The [`events` table](../datasets/batch_view/events/reference.md)
includes Normandy enrollment and unenrollment events
Expand All @@ -15,12 +26,6 @@ or name (for add-on experiments).
Normandy events are described in detail in the
[Firefox source tree docs][normandy-doc].

Note that addon studies do not have branch information in the events table,
since addons, not Normandy, are responsible for branch assignment.
For studies built with the obsolete [add-on utilities][`addon-utils`],
branch assignments are published to the
[shield_study] dataset.

## Counting pref-flip enrollment events by branch

The `event_map_values` column of enroll events contains a `branch` key,
Expand Down Expand Up @@ -68,5 +73,3 @@ ORDER BY 1, 2, 3
```

[normandy-doc]: https://firefox-source-docs.mozilla.org/toolkit/components/normandy/normandy/data-collection.html#enrollment
[shield_study]: ../datasets/experiment_telemetry.md#telemetryshield_study
[`addon-utils`]: https://github.com/mozilla/shield-studies-addon-utils
82 changes: 31 additions & 51 deletions src/datasets/experiment_telemetry.md
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
# Analyzing data from SHIELD studies
# Accessing Experiment Telemetry

This article introduces the datasets that are useful for analyzing studies in Firefox.
After reading this article,
you should understand how to answer questions about
study enrollment,
identify telemetry from clients enrolled in an experiment,
and locate telemetry from add-on studies.
After reading this article, you should understand how to answer questions about
experiment enrollment, and identify telemetry from clients enrolled in an experiment.

## Table of contents

Expand All @@ -27,7 +24,34 @@ the slug for add-on experiments is defined in the recipe by a field named `name`

You can find the slug associated with an experiment in Experimenter.

## Tables
## Tables (Glean)

### `experiments` map (Glean)

Glean tables include a `ping_info` column with `experiments` mapping from
experiment slug to a struct containing information about the experiment,
including `branch`.

You can query for enrolled clients using a query like:

```sql
SELECT
-- ... some fields ...,
`mozfun.map.get_key`(ping_info.experiments, '{experiment_slug}').branch
FROM
`moz-fx-data-shared-prod.firefox_desktop.metrics`
WHERE
`mozfun.map.get_key`(ds.ping_info.experiments, '{experiment_slug}') IS NOT NULL
AND DATE(m.submission_timestamp) BETWEEN '<start_date>' AND '<end_date>'
AND normalized_channel = 'release'
-- AND ...other filters...
```

## Tables (Legacy Telemetry)

As of January 2025, legacy telemetry is still used for enrollment and exposure
events. However, while Glean adoption is in progress, these docs remain for
the time being as reference.

### `experiments` map (ping tables)

Expand Down Expand Up @@ -85,47 +109,3 @@ The event schema is described
[in the Firefox source tree](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/normandy/lib/TelemetryEvents.sys.mjs).

The `events` table is updated daily.

### `telemetry.shield_study_addon`

The `telemetry.shield_study_addon` table contains SHIELD telemetry from legacy add-on experiments,
i.e. key-value pairs sent with the
`browser.study.sendTelemetry()` method from the
[SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/)
library.

The `study_name` attribute of the `payload` column will contain the identifier
registered with the SHIELD add-on utilities.
This is set by the add-on; sometimes it takes the value of
`applications.gecko.id` from the add-on's `manifest.json`.
This is often not the same as the Normandy slug.

The schema for shield-study-addon pings is described in the
[`mozilla-pipeline-schemas` repository](https://github.com/mozilla-services/mozilla-pipeline-schemas/tree/master/schemas/telemetry/shield-study-addon).

The key-value pairs are present in `data` attribute of the `payload` column.

The `telemetry.shield_study_addon` table contains only full days of data.
If you need access to data with lower latency, you can use the "live" table
`telemetry_live.shield_study_addon_v4` which should have latency significantly
less than 1 hour.

### `telemetry.shield_study`

The `telemetry.shield_study` dataset includes
enrollment and unenrollment events for legacy add-on experiments only,
sent by the [SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/).

The `study_name` attribute of the `payload` column will contain the identifier
registered with the SHIELD add-on utilities.
This is set by the add-on; sometimes it takes the value of
`applications.gecko.id` from the add-on's `manifest.json`.
This is often not the same as the Normandy slug.

Normandy also emits its own enrollment and unenrollment events for these studies,
which are available in the `events` table.

The `telemetry.shield_study` table contains only full days of data.
If you need access to data with lower latency, you can use the "live" table
`telemetry_live.shield_study_v4` which should have latency significantly
less than 1 hour.
12 changes: 7 additions & 5 deletions src/tools/projects.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,11 @@ See also [`data-docs`][docs] for documentation on datasets.
| [Hardware Report][hwreport_gh] | Firefox Hardware Report, [available here][hwreport] |
| [St. Mocli][stmocli] | A command-line interface to [STMO][stmo] |
| [probe-scraper] | Scrape and publish Telemetry probe data from Firefox |
| [test-tube] | Compare data across branches in experiments |
| [experimenter] | A web application for managing experiments |
| [St. Moab][stmoab] | Automatically generate Redash dashboard for A/B experiments |
| [Jetstream] | Automated analysis for experiments |
| [metric-hub] | Semantic layer for metric definitions |

See also [What Data Tool Should I Use?][data-tools-wiki] for more information on Data Tools and their uses.

[tmo_gh]: https://github.com/mozilla/telemetry-dashboard
[glam]: https://github.com/mozilla/glam
Expand All @@ -91,16 +93,16 @@ See also [`data-docs`][docs] for documentation on datasets.
[redashstmo]: https://github.com/mozilla/redash-stmo
[taar]: https://github.com/mozilla/taar
[ensemble]: https://github.com/mozilla/ensemble
[shield]: https://wiki.mozilla.org/index.php?title=Firefox/Shield
[tmo]: https://telemetry.mozilla.org
[stmo]: https://sql.telemetry.mozilla.org
[hwreport_gh]: https://github.com/mozilla/firefox-hardware-report
[hwreport]: https://data.firefox.com/dashboard/hardware
[stmocli]: https://github.com/mozilla/stmocli
[probe-scraper]: https://github.com/mozilla/probe-scraper
[test-tube]: https://github.com/mozilla/firefox-test-tube
[experimenter]: https://github.com/mozilla/experimenter
[stmoab]: https://github.com/mozilla/stmoab
[jetstream]: https://github.com/mozilla/jetstream
[metric-hub]: https://github.com/mozilla/metric-hub
[data-tools-wiki]: https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/375750774/Data+Tools

## Legacy projects

Expand Down

0 comments on commit 14c3b83

Please sign in to comment.