Skip to content

Commit

Permalink
Update normalized-metrics.md
Browse files Browse the repository at this point in the history
  • Loading branch information
nick-statsig authored Dec 25, 2024
1 parent 9af4358 commit 779d5a9
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ With normal A/B tests the unit of randomization (e.g. UserID) matches the unit o
For example - you've added image support to a collaborative commenting feature in your product and want to A/B test it before rollout. You randomize it using businessID. You cannot randomize by userID, since you need everyone within a single business to either have this new feature or not. If you simply compared # of comments per businessID, this data would be skewed by large companies. A business with 1000 employees, but 10 comments would "contribute more" than a business with 5 employees who made 5 comments. Normalizing a metric in this case - is normalizing by users exposed to the experiment. In this instance if 1000 and 5 users were exposed from each business, the first business would have a comments/user rate of 0.01, while the second company would have a comments/user rate of 1. This is reasonable now to compare across companies of many different sizes.

## What it does
Under the covers, normalizing a metric simple creates a ratio metric. The numerator is metric you're normalizing. The denominator is a COUNT DISTINCT of the UnitID you're normalizing to.
Under the covers, normalizing a metric simply creates a ratio metric. The numerator is metric you're normalizing. The denominator is a COUNT DISTINCT of the UnitID you're normalizing to.
If you wanted to, you could also create this ratio metric yourself and use it in experiments - this is documented [here](https://docs.statsig.com/metrics/different-id).

## How to do it
Expand Down

0 comments on commit 779d5a9

Please sign in to comment.