Skip to content

Commit

Permalink
Update monitor-an-experiment.md
Browse files Browse the repository at this point in the history
  • Loading branch information
vmadhusudanan committed Jul 22, 2024
1 parent 5337899 commit ad8d093
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ To monitor the status of your experiment,
- **p-value < 0.001**: Alert (red) meaning that there is likely a problem with deduplication being applied more to one of the treatment groups and experiment results may not be trustworthy.

- **Pre-experimental Bias Check** In some cases, users in two experiment groups can have meaningfully different average behaviors before your experiment applies any intervention to them. If this difference is maintained after your experiment starts, it's possible that experiment analysis will attribute that pre-existing difference to your intervention. This can make a result seem more or less "good" or "bad" than it really is. CUPED is helpful in addressing this bias, but can't totally account for it. [Learn more](https://docs.statsig.com/stats-engine/pre-experiment-bias).
- **Outlier check** In some cases, your metrics can be materially influenced by a few outliers. This check will alert you when this is the case, so you can evaluate if using tools like Winsorization or Capping is appropriate. It will also make sure capping is not unduly influencing results if already applied.
- **Outlier Check** In some cases, your metrics can be materially influenced by a few outliers. This check will alert you when this is the case, so you can evaluate if using tools like Winsorization or Capping is appropriate. It will also make sure capping is not unduly influencing results if already applied.

0 comments on commit ad8d093

Please sign in to comment.