Skip to content

Commit

Permalink
Merge branch 'main' into add-scout-bot
Browse files Browse the repository at this point in the history
  • Loading branch information
vijaye-statsig authored Jul 29, 2024
2 parents 3d4eb4a + bb11f0f commit 669151f
Show file tree
Hide file tree
Showing 8 changed files with 44 additions and 9 deletions.
2 changes: 1 addition & 1 deletion docs/client/Templates/_GettingStarted.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ After installation, you will need to initialize the SDK using a Client SDK key f
These Client SDK Keys are intended to be embedded in client side applications. If need be, you can invalidate or create new SDK Keys for other applications/SDK integrations.

:::info
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you do accidentally expose it, you can create a new one in the Statsig console.
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console.
:::

In addition to the SDK key, you should also pass in a [StatsigUser](#statsig-user) for feature gate targeting and experimentation grouping purposes.
Expand Down
2 changes: 1 addition & 1 deletion docs/client/_client-sdk-template.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ After installation, you will need to initialize the SDK using a Client SDK key f
These Client SDK Keys are intended to be embedded in client side applications. If need be, you can invalidate or create new SDK Keys for other applications/SDK integrations.

:::info
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you do accidentally expose it, you can create a new one in the Statsig console.
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console.
:::

In addition to the SDK key, you should also pass in a [Statsig user](#statsig-user) for feature gate targeting and experimentation grouping purposes.
Expand Down
33 changes: 33 additions & 0 deletions docs/dynamic-config/enforce-schema.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: Using a schema
sidebar_label: Using a Schema
slug: /dynamic-config/enforce-schema
---

Dynamic configs support schemas using [JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) syntax to enforce a common convention between the return values for each rule you'll set.

:::info
Schemas are only enforced when editing dynamic configs through the console or API, and are not used at code runtime.
:::

For example, if you have a dynamic config that returns settings for a site banner, you might have a schema of:
```
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"title": {
"type": "string",
},
"description": {
"type": "string",
},
"cta": {
"type": "string",
}
},
"required": ["title", "description", "cta"],
}
```

Now, each of your rules must return an object including title, description, and CTA.
2 changes: 1 addition & 1 deletion docs/server/Templates/_GettingStarted.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ To work with the SDK, you will need a Statsig account. If you don't yet have an
After installation, you will need to initialize the SDK using a [Server Secret Key from the statsig console](https://console.statsig.com/api_keys).

:::info
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you do accidentally expose it, you can create a new one in the Statsig console.
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console.
:::

<>
Expand Down
2 changes: 1 addition & 1 deletion docs/server/_server-sdk-template.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ You could skip this for now, but you will need an SDK Key and some Feature Gates
After installation, you will need to initialize the SDK using a [Server Secret Key from the statsig console](https://console.statsig.com/api_keys).

:::info
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you do accidentally expose it, you can create a new one in the Statsig console.
Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console.
:::

There is also an optional parameter `options` that allows you to pass in a [StatsigOptions](#statsig-options) to customize the SDK.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ sidebar_label: Benjamini–Hochberg
slug: /stats-engine/methodologies/benjamini–hochberg-procedure
---

## Coming soon
## What it is
The Benjamini-Hochberg procedure is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. It is not as extreme as a [Bonferroni Correction](https://docs.statsig.com/stats-engine/methodologies/bonferroni-correction), because instead of controlling the chance of at least one false positive (Family Wise Error Rate), this controls the expected value of false positives when the null hypothesis has been rejected (False Discovery Rate).

Currently in beta. Reach out in Slack if you're having it turned on for your company now.
Like with other analysis settings, you can configure this per experiment (or configure global Experiment Settings to default it).
![image](https://github.com/user-attachments/assets/c865494e-0ae4-489c-a416-45848b4d10bc)

## Methodology

The Benjamini-Hochberg procedure is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. It is not as extreme as a [Bonferroni Correction](https://docs.statsig.com/stats-engine/methodologies/bonferroni-correction), because instead of controlling the chance of at least one false positive (Family Wise Error Rate), this controls the expected value of false positives when the null hypothesis has been rejected (False Discovery Rate).

## Methodology
The significance level is calculated by sorting p-values in ascending order and comparing with the a threshold. Each p-value’s threshold is the desired False Discovery Rate divided by the number of comparisons being evaluated multiplied by what rank a p-value is in the ordered list. The largest threshold value which is higher than its corresponding p-value is our new significance level ($\alpha$).

The Benjamini-Hochberg Correction can be applied based on:
Expand Down
1 change: 1 addition & 0 deletions docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,7 @@ module.exports = {
[
require.resolve("@easyops-cn/docusaurus-search-local"),
{
explicitSearchResultPath: true,
hashed: true,
indexBlog: false,
indexDocs: true,
Expand Down
1 change: 1 addition & 0 deletions sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ module.exports = {
},
items: [
"dynamic-config/working-with",
"dynamic-config/enforce-schema",
"dynamic-config/create-new",
"dynamic-config/add-rule",
],
Expand Down

0 comments on commit 669151f

Please sign in to comment.