diff --git a/README.md b/README.md index 7abf3a08..9f2acb09 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,8 @@ Custom Secret Scanning Patterns repository. +- Database Connection String (full string) + - Database Connection String (1) - Database Connection String (2) @@ -41,13 +43,25 @@ Custom Secret Scanning Patterns repository. - Database Connection String (3) - TSQL CREATE LOGIN/USER + +- SQLAlchemy Database Connection String + +- MongoDB connection string + +- JDBC Database Connection String ### [Generic Secrets / Passwords](./generic) -- Generic Passwords +- Generic Passwords (fewer FPs) + +- Generic Password with hex encoded secrets + +- Generic Password with Base64 encoded secrets + +- Generic Password with URI-safe Base64 encoded secrets - UUIDs @@ -85,6 +99,8 @@ Custom Secret Scanning Patterns repository. - Credit Cards - Discover - IBAN + +- Norwegian national identity number/D number ### [RSA Keys](./rsa) @@ -135,6 +151,8 @@ Custom Secret Scanning Patterns repository. - Okta token +- Okta API key (precise) + - DataDog API key - DataDog APP key @@ -163,9 +181,29 @@ Custom Secret Scanning Patterns repository. - Vercel OAuth client secrets -- MongoDB connection string - - UUIDv4 Bearer token (maybe Heroku) - Azure client secret + +- Google private key id (or older API key) + +- OpenStack password/API key + +- AlienVault OTX API key + +- Apollo.io API key + +- ClickUp API key + +- Amazon MWS Auth Token + +- Jenkins API token + +- AWS S3 presigned URL + +- Azure Access Key (legacy format) + +- Azure Shared Access Signature (SAS) Token + +- CircleCI API token \ No newline at end of file diff --git a/configs/README.md b/configs/README.md index a7295cea..5fbbc228 100644 --- a/configs/README.md +++ b/configs/README.md @@ -118,25 +118,29 @@ _version: v0.1_ ## YAML Static Password Fields **⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** -Pattern to find Static passwords in YAML configuration files +Pattern to find hardcoded passwords in YAML configuration files -_version: v0.1_ +_version: v0.2_ **Comments / Notes:** -- The hardcoded password is between 12 and 32 chars long +- Expect large numbers of false positives on variables containing 'key' or 'token' -- Some false positives in Code might appear +- The hardcoded password is any length -- The pattern only checks for certain key words to begin the pattern (`secret`, `password`, etc.) +- Some false positives in code might appear + +- The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.) + +- Does not allow for multline blocks
Pattern Format ```regex -[^\r\n'"]+ +[^\r\n`'"\x00-\x08]+ ```
@@ -145,7 +149,7 @@ _version: v0.1_ Start Pattern ```regex -(?:\n|\A)[ \t]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key)[ \t]*:[ \t]*['"]? +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*['"]? ```
@@ -166,31 +170,267 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - ^(?:keyPassphrase|password|key|[ \t]+|\$\{[A-Za-z0-9_-]+\}|(?:str|string|int|bool)( +#.*)?),?$ + ^(keyPassphrase|password|key|[ \t]+|\$\{[^}]+}|(str|string|int|bool)( +#.*)?),?$ + ``` +- Not Match: + + ```regex + ^(.* = )?(None|[Tt]rue|[Ff]alse|[Nn]ull|Default(Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ + ``` +- Not Match: + + ```regex + ^(((this|self|obj)\.)([A-Za-z_]+\,|[A-Za-z_].*)|\{\}|\[\]|[0-9a-zA-Z],|\{)$ + ``` +- Not Match: + + ```regex + ^(([a-zA-Z_]+(\(\))?\.)*[a-zA-Z_]+\(\)|\|\s*)$|\{\{[^}]+\}\}|\$\{\{ |^!Ref |^\$\(.*\)|^(https?|file)://|^(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)$|^=|\(\) => |\([^)]+\) => \{ + ``` +- Not Match: + + ```regex + ^\s*(typing\.)?([Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ + ``` + +
+ +## YAML hardcoded passwords (plain scalars) + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded passwords in YAML configuration files, using plain scalars + +_version: v0.1_ + +**Comments / Notes:** + + +- The hardcoded password is any length + +- Some false positives in code might appear + +- The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.) + +- Only allows for plain scalars, not quoted or multi-line, to better control false positives + + +
+Pattern Format + +```regex +[^\r\n`'"\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]* +``` + +
+End Pattern + +```regex +[\r\n]|\z +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + ^(keyPassphrase|password|key|[ \t]+|\$\{[^}]+}|(str|string|int|bool)( +#.*)?),?$ + ``` +- Not Match: + + ```regex + ^(.* = )?(None|[Tt]rue|[Ff]alse|[Nn]ull|Default(Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ + ``` +- Not Match: + + ```regex + ^(((this|self|obj)\.)([A-Za-z_]+\,|[A-Za-z_].*)|\{\}|\[\]|[0-9a-zA-Z],|\{)$ ``` - Not Match: ```regex - ^(?:.* = )?(?:None|[Tt]rue|[Ff]alse|[Nn]ull|Default(?:Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ + ^(([a-zA-Z_]+(\(\))?\.)*[a-zA-Z_]+\(\)|\|\s*)$|\{\{[^}]+\}\}|\$\{\{ |^!Ref |^\$\(.*\)|^(https?|file)://|^(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)$|^=|\(\) => |\([^)]+\) => \{ ``` - Not Match: ```regex - ^(?:(?:this|self|obj)\.)(?:[A-Za-z_]+\,|[A-Za-z_].*)$ + ^\s*(typing\.)?([Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ ``` + +
+ +## YAML hardcoded passwords (single quoted strings) + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded passwords in YAML configuration files, using single quotes + +_version: v0.1_ + +**Comments / Notes:** + + +- The hardcoded password is any length + +- Some false positives in code might appear + +- The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.) + +- Only allows for only single-quoted passwords, to better control false positives + + +
+Pattern Format + +```regex +[^\r\n'\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*' +``` + +
+End Pattern + +```regex +'([ \t]*[\r\n]|\z) +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + - Not Match: ```regex - ^(?:[a-zA-Z_]+(?:\(\))?\.)*[a-zA-Z_]+\(\)$ + \{\{[^{}]+\}\} ``` + +
+ +## YAML hardcoded passwords (double quoted strings) + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded passwords in YAML configuration files, using single quotes + +_version: v0.1_ + +**Comments / Notes:** + + +- The hardcoded password is any length + +- Some false positives in code might appear + +- The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.) + +- Only allows for only double-quoted passwords, to better control false positives + + +
+Pattern Format + +```regex +[^\r\n"\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*" +``` + +
+End Pattern + +```regex +"([ \t]*[\r\n]|\z) +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + - Not Match: ```regex - ^\s*(?:typing\.)?(?:[Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ + \{\{[^{}]+\}\} ```
+## YAML hardcoded passwords (multiline strings) + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded passwords in YAML configuration files, using multiline strings + +_version: v0.1_ + +**Comments / Notes:** + + +- The hardcoded password is any length + +- Some false positives in code or YAML files might appear, especially where the variable is called 'key' or 'token' + +- The pattern checks for certain key words to end the variable name (`secret`, `password`, etc.) + + +
+Pattern Format + +```regex +[^\x00-\x08]+? +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*[|>][+-]?[ \t]*(\n|\r\n)[ \t]+ +``` + +
+End Pattern + +```regex +\n\n|\r\n\r\n|(\n|\r\n)[ \t]+\S+:|(\n|\r\n)\.\.\.[ \t\n\r]|\z +``` + +
+ ## GitHub Actions SHA Checker @@ -320,4 +560,510 @@ _version: v0.1_ \" ``` + + +## .env file style secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Find .env file style secrets in configuration files such as .env, Dockerfile, shell scripts etc. + +_version: v0.1_ + +**Comments / Notes:** + + +- Looks for secrets in the format of `SECRET=secret` at the start of a line, possibly with an `ENV ` or `export ` prefix + +- Allows no whitespace in the secret, to cut false positives + +- Some false positives in code might appear, especially where the variable name is 'key' or 'token' + +- The pattern only checks for certain key words to begin the pattern (`secret`, `password`, etc.) + +- More restrictive than the Generic Passwords pattern, so less prone to false positives + +- Lower length limit of 8 to remove FPs + + +
+Pattern Format + +```regex +[^\r\n\x00-\x08'"#]{8,} +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)((export|ENV|ARG) )?[A-Z_]*(?:SECRET|PASS(?:WD|WOR[TD]|CODE|PHRASE)?|KEY|TOKEN)=['"]? +``` + +
+End Pattern + +```regex +[\r\n#]|['"]\s*[\r\n]|\z +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + ^\$(\{[^}]+\}|\([^)]+\)|[A-Za-z_]+|[0-9])$ + ``` +- Not Match: + + ```regex + ^(<[^>]+>|\[[^]+\]|\{[^}+\}|(your|my|the|a)_[a-z_]+|.*(passwor[t]|key|secret|token|密码).*|\.\.\.|xxx+|yyy+|zzz+|aaa+|bbb+|ccc+)$ + ``` +- Not Match: + + ```regex + ^(test|value)([._-][a-z_.-]+)?$ + ``` +- Not Match: + + ```regex + ^(?i)(true|false|y(es)?|no?|on|off|0|1|nill|null|none|(\\x[a-f0-9]{2})+)$ + ``` +- Not Match: + + ```regex + ^(/|file:///|https?://[A-Za-z]:/)[A-Za-z0-9._-]{3,}+(/[a-z._-]{1,}){2,}/?$ + ``` + +
+ +## YAML with Base64 encoded secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded Base64-encoded passwords in YAML configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- Some false positives in code might appear + + +
+Pattern Format + +```regex +(([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) +``` + +
+End Pattern + +```regex +['"\r\n]|\z +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ + ``` + +
+ +## YAML with URI-safe Base64 encoded secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded URI-safe Base64-encoded passwords in YAML configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- Some false positives in code might appear + +- This matches _- instead of +/, for URI-safe Base64 + + +
+Pattern Format + +```regex +(([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) +``` + +
+End Pattern + +```regex +['"\r\n]|\z +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ + ``` + +
+ +## YAML with hex token + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded hex-encoded tokens in YAML configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The hex token must be 32, 40 or 64 characters long, and contain numbers and letters + +- Some false positives in code might appear + + +
+Pattern Format + +```regex +[0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} +``` + +
+ +
+Start Pattern + +```regex +(?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) +``` + +
+End Pattern + +```regex +['"\r\n]|\z +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [a-f] + ``` + +
+ +## JSON with Base64 encoded secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded Base64-encoded passwords in JSON configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- This may match in code, such as Python, that resembles JSON + +- This will not match some isolated fragments of JSON, so be aware of that when testing it + + +
+Pattern Format + +```regex +(([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) +``` + +
+ +
+Start Pattern + +```regex +[{[,][ \t]*[ \t\r\n]*"(?i)[a-z_.-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" +``` + +
+End Pattern + +```regex +"[ \t\r\n]*[,}\]] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ + ``` + +
+ +## JSON with URI-safe Base64 encoded secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded URI-safe Base64-encoded passwords in JSON configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- This may match in code, such as Python, that resembles JSON + +- This will not match some isolated fragments of JSON, so be aware of that when testing it + +- This matches _- instead of +/, for URI-safe Base64 + + +
+Pattern Format + +```regex +(([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) +``` + +
+ +
+Start Pattern + +```regex +[{[,][ \t]*[ \t\r\n]*"(?i)[a-z_.-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" +``` + +
+End Pattern + +```regex +"[ \t\r\n]*[,}\]] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ + ``` + +
+ +## JSON with hex encoded secrets + +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** +Hardcoded hex-encoded tokens in JSON configuration files + +_version: v0.1_ + +**Comments / Notes:** + + +- The hex token must be 32, 40 or 64 characters long, and contain numbers and letters + +- This may match in code, such as Python, that resembles JSON + +- This will not match some isolated fragments of JSON, so be aware of that when testing it + + +
+Pattern Format + +```regex +[0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} +``` + +
+ +
+Start Pattern + +```regex +[{[,][ \t]*[ \t\r\n]*"(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" +``` + +
+End Pattern + +```regex +"[ \t\r\n]*[,}\]] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [a-f] + ``` +
\ No newline at end of file diff --git a/configs/__snapshots__/env_passwords.csv b/configs/__snapshots__/env_passwords.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/env_passwords.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/json_base64.csv b/configs/__snapshots__/json_base64.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/json_base64.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/json_base64_uri.csv b/configs/__snapshots__/json_base64_uri.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/json_base64_uri.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/json_token.csv b/configs/__snapshots__/json_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/json_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_base64.csv b/configs/__snapshots__/yaml_base64.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_base64.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_base64_uri.csv b/configs/__snapshots__/yaml_base64_uri.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_base64_uri.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_passwords_multiline.csv b/configs/__snapshots__/yaml_passwords_multiline.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_passwords_multiline.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_passwords_plain.csv b/configs/__snapshots__/yaml_passwords_plain.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_passwords_plain.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_passwords_single_quoted.csv b/configs/__snapshots__/yaml_passwords_single_quoted.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_passwords_single_quoted.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/__snapshots__/yaml_token.csv b/configs/__snapshots__/yaml_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/configs/__snapshots__/yaml_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/configs/patterns.yml b/configs/patterns.yml index 4a47f04d..8e347453 100644 --- a/configs/patterns.yml +++ b/configs/patterns.yml @@ -21,6 +21,7 @@ patterns: start_offset: 18 end_offset: 26 + - name: Hardcoded Spring SQL passwords description: | Hardcoded JDBC / Spring datasource passwords which typically are in property files or passed in at runtime @@ -60,6 +61,7 @@ patterns: start_offset: 27 end_offset: 35 + - name: Django Secret Key type: django_secret_key regex: @@ -84,26 +86,28 @@ patterns: - name: YAML Static Password Fields type: yaml_static_password_fields description: | - Pattern to find Static passwords in YAML configuration files + Pattern to find hardcoded passwords in YAML configuration files experimental: true regex: - version: 0.1 + version: 0.2 pattern: | - [^\r\n'"]+ + [^\r\n`'"\x00-\x08]+ start: | - (?:\n|\A)[ \t]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key)[ \t]*:[ \t]*['"]? + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*['"]? end: | ['"\r\n]|\z additional_not_match: - - ^(?:keyPassphrase|password|key|[ \t]+|\$\{[A-Za-z0-9_-]+\}|(?:str|string|int|bool)( +#.*)?),?$ - - ^(?:.* = )?(?:None|[Tt]rue|[Ff]alse|[Nn]ull|Default(?:Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ - - ^(?:(?:this|self|obj)\.)(?:[A-Za-z_]+\,|[A-Za-z_].*)$ - - ^(?:[a-zA-Z_]+(?:\(\))?\.)*[a-zA-Z_]+\(\)$ - - ^\s*(?:typing\.)?(?:[Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ - comments: - - "The hardcoded password is between 12 and 32 chars long" - - "Some false positives in Code might appear" - - "The pattern only checks for certain key words to begin the pattern (`secret`, `password`, etc.)" + - ^(keyPassphrase|password|key|[ \t]+|\$\{[^}]+}|(str|string|int|bool)( +#.*)?),?$ + - ^(.* = )?(None|[Tt]rue|[Ff]alse|[Nn]ull|Default(Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ + - ^(((this|self|obj)\.)([A-Za-z_]+\,|[A-Za-z_].*)|\{\}|\[\]|[0-9a-zA-Z],|\{)$ + - ^(([a-zA-Z_]+(\(\))?\.)*[a-zA-Z_]+\(\)|\|\s*)$|\{\{[^}]+\}\}|\$\{\{ |^!Ref |^\$\(.*\)|^(https?|file)://|^(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)$|^=|\(\) => |\([^)]+\) => \{ + - ^\s*(typing\.)?([Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ + comments: + - "Expect large numbers of false positives on variables containing 'key' or 'token'" + - "The hardcoded password is any length" + - "Some false positives in code might appear" + - "The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.)" + - "Does not allow for multline blocks" expected: - name: example.yml start_offset: 57 @@ -116,6 +120,117 @@ patterns: start_offset: 10 end_offset: 21 + + - name: YAML hardcoded passwords (plain scalars) + type: yaml_passwords_plain + description: | + Hardcoded passwords in YAML configuration files, using plain scalars + experimental: true + regex: + version: 0.1 + pattern: | + [^\r\n`'"\x00-\x08]+ + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]* + end: | + [\r\n]|\z + additional_not_match: + - ^(keyPassphrase|password|key|[ \t]+|\$\{[^}]+}|(str|string|int|bool)( +#.*)?),?$ + - ^(.* = )?(None|[Tt]rue|[Ff]alse|[Nn]ull|Default(Type)?|Event|[A-Z]+_KEY|VERSION|NAME|update|destroy|(?:dis|en)ableEventListeners|\.\.\.),?$ + - ^(((this|self|obj)\.)([A-Za-z_]+\,|[A-Za-z_].*)|\{\}|\[\]|[0-9a-zA-Z],|\{)$ + - ^(([a-zA-Z_]+(\(\))?\.)*[a-zA-Z_]+\(\)|\|\s*)$|\{\{[^}]+\}\}|\$\{\{ |^!Ref |^\$\(.*\)|^(https?|file)://|^(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)$|^=|\(\) => |\([^)]+\) => \{ + - ^\s*(typing\.)?([Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*$ + comments: + - "The hardcoded password is any length" + - "Some false positives in code might appear" + - "The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.)" + - "Only allows for plain scalars, not quoted or multi-line, to better control false positives" + test: + data: "password: password123" + start_offset: 10 + end_offset: 21 + + + - name: YAML hardcoded passwords (single quoted strings) + type: yaml_passwords_single_quoted + description: | + Hardcoded passwords in YAML configuration files, using single quotes + experimental: true + regex: + version: 0.1 + pattern: | + [^\r\n'\x00-\x08]+ + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*' + end: | + '([ \t]*[\r\n]|\z) + additional_not_match: + - \{\{[^{}]+\}\} + comments: + - "The hardcoded password is any length" + - "Some false positives in code might appear" + - "The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.)" + - "Only allows for only single-quoted passwords, to better control false positives" + test: + data: "password: 'password123'" + start_offset: 11 + end_offset: 22 + + + - name: YAML hardcoded passwords (double quoted strings) + type: yaml_passwords_single_quoted + description: | + Hardcoded passwords in YAML configuration files, using single quotes + experimental: true + regex: + version: 0.1 + pattern: | + [^\r\n"\x00-\x08]+ + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*" + end: | + "([ \t]*[\r\n]|\z) + additional_not_match: + - \{\{[^{}]+\}\} + comments: + - "The hardcoded password is any length" + - "Some false positives in code might appear" + - "The pattern only checks for certain key words to end the variable name (`secret`, `password`, etc.)" + - "Only allows for only double-quoted passwords, to better control false positives" + test: + data: | + password: "password123" + start_offset: 11 + end_offset: 22 + + + - name: YAML hardcoded passwords (multiline strings) + type: yaml_passwords_multiline + description: | + Hardcoded passwords in YAML configuration files, using multiline strings + experimental: true + regex: + version: 0.1 + pattern: | + [^\x00-\x08]+? + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*[|>][+-]?[ \t]*(\n|\r\n)[ \t]+ + end: | + \n\n|\r\n\r\n|(\n|\r\n)[ \t]+\S+:|(\n|\r\n)\.\.\.[ \t\n\r]|\z + comments: + - "The hardcoded password is any length" + - "Some false positives in code or YAML files might appear, especially where the variable is called 'key' or 'token'" + - "The pattern checks for certain key words to end the variable name (`secret`, `password`, etc.)" + test: + data: |- + password: | + password123 + andsomemore + other_field: foo + start_offset: 14 + end_offset: 56 + + - name: GitHub Actions SHA Checker type: github_actions_sha_checker comments: @@ -187,3 +302,207 @@ patterns: start_offset: 27 end_offset: 91 + + + - name: .env file style secrets + type: env_passwords + description: | + Find .env file style secrets in configuration files such as .env, Dockerfile, shell scripts etc. + experimental: true + regex: + version: 0.1 + pattern: | + [^\r\n\x00-\x08'"#]{8,} + start: | + (?:\n|\A)((export|ENV|ARG) )?[A-Z_]*(?:SECRET|PASS(?:WD|WOR[TD]|CODE|PHRASE)?|KEY|TOKEN)=['"]? + end: | + [\r\n#]|['"]\s*[\r\n]|\z + additional_not_match: + # variable, process substitution + - ^\$(\{[^}]+\}|\([^)]+\)|[A-Za-z_]+|[0-9])$ + # placeholders + - ^(<[^>]+>|\[[^]+\]|\{[^}+\}|(your|my|the|a)_[a-z_]+|.*(passwor[t]|key|secret|token|密码).*|\.\.\.|xxx+|yyy+|zzz+|aaa+|bbb+|ccc+)$ + # non-secret related "key" content + - ^(test|value)([._-][a-z_.-]+)?$ + # non-secret related content + - ^(?i)(true|false|y(es)?|no?|on|off|0|1|nill|null|none|(\\x[a-f0-9]{2})+)$ + # a path + - '^(/|file:///|https?://[A-Za-z]:/)[A-Za-z0-9._-]{3,}+(/[a-z._-]{1,}){2,}/?$' + comments: + - "Looks for secrets in the format of `SECRET=secret` at the start of a line, possibly with an `ENV ` or `export ` prefix" + - "Allows no whitespace in the secret, to cut false positives" + - "Some false positives in code might appear, especially where the variable name is 'key' or 'token'" + - "The pattern only checks for certain key words to begin the pattern (`secret`, `password`, etc.)" + - "More restrictive than the Generic Passwords pattern, so less prone to false positives" + - "Lower length limit of 8 to remove FPs" + test: + data: | + SECRET_KEY=password123 + start_offset: 11 + end_offset: 22 + + + - name: YAML with Base64 encoded secrets + type: yaml_base64 + description: | + Hardcoded Base64-encoded passwords in YAML configuration files + experimental: true + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) + end: | + ['"\r\n]|\z + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "Some false positives in code might appear" + test: + data: "password: AAAAAAAAAAAa00==" + start_offset: 10 + end_offset: 26 + + + - name: YAML with URI-safe Base64 encoded secrets + type: yaml_base64_uri + description: | + Hardcoded URI-safe Base64-encoded passwords in YAML configuration files + experimental: true + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) + end: | + ['"\r\n]|\z + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "Some false positives in code might appear" + - "This matches _- instead of +/, for URI-safe Base64" + test: + data: "password: AAAAAAAAAAAa00==" + start_offset: 10 + end_offset: 26 + + + - name: YAML with hex token + type: yaml_token + description: | + Hardcoded hex-encoded tokens in YAML configuration files + experimental: true + regex: + version: 0.1 + pattern: | + [0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} + start: | + (?:\n|\A)[ \t]*(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)[ \t]*:[ \t]*(['"]?|[|>]-?[ \t]*\n[ \t]*) + end: | + ['"\r\n]|\z + additional_match: + - '[0-9]' + - '[a-f]' + comments: + - "The hex token must be 32, 40 or 64 characters long, and contain numbers and letters" + - "Some false positives in code might appear" + test: + data: "password: 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + start_offset: 10 + end_offset: 74 + + + - name: JSON with Base64 encoded secrets + type: json_base64 + description: | + Hardcoded Base64-encoded passwords in JSON configuration files + experimental: true + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) + start: | + [{[,][ \t]*[ \t\r\n]*"(?i)[a-z_.-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" + end: | + "[ \t\r\n]*[,}\]] + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "This may match in code, such as Python, that resembles JSON" + - "This will not match some isolated fragments of JSON, so be aware of that when testing it" + test: + data: | + {"password": "AAAAAAAAAAAa00==", "user": "admin"} + start_offset: 14 + end_offset: 30 + + + - name: JSON with URI-safe Base64 encoded secrets + type: json_base64_uri + description: | + Hardcoded URI-safe Base64-encoded passwords in JSON configuration files + experimental: true + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) + start: | + [{[,][ \t]*[ \t\r\n]*"(?i)[a-z_.-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" + end: | + "[ \t\r\n]*[,}\]] + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "This may match in code, such as Python, that resembles JSON" + - "This will not match some isolated fragments of JSON, so be aware of that when testing it" + - "This matches _- instead of +/, for URI-safe Base64" + test: + data: | + {"password": "AAAAAAAAAAAa00==", "user": "admin"} + start_offset: 14 + end_offset: 30 + + + - name: JSON with hex encoded secrets + type: json_token + description: | + Hardcoded hex-encoded tokens in JSON configuration files + experimental: true + regex: + version: 0.1 + pattern: | + [0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} + start: | + [{[,][ \t]*[ \t\r\n]*"(?i)[a-z_-]*(?:secret|service_pass(wd|word|code|phrase)|pass(?:wd|word|code|phrase)?|key|token)"[ \t]*:[ \t]*" + end: | + "[ \t\r\n]*[,}\]] + additional_match: + - '[0-9]' + - '[a-f]' + comments: + - "The hex token must be 32, 40 or 64 characters long, and contain numbers and letters" + - "This may match in code, such as Python, that resembles JSON" + - "This will not match some isolated fragments of JSON, so be aware of that when testing it" + test: + data: | + {"password": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", "user": "admin"} + start_offset: 14 + end_offset: 78 diff --git a/database/README.md b/database/README.md index c84a0541..18da1416 100644 --- a/database/README.md +++ b/database/README.md @@ -5,6 +5,59 @@ # Database passwords +## Database Connection String (full string) + + +Database connection strings are used to connect to databases, often with embedded credentials. +_version: v0.1_ + + + +
+Pattern Format + +```regex +[^"'`\x00-\x08\r\n|]*([Ss]erver|[Pp]rovider|[Dd]atabase|[Uu]ser [Ii]d|[Dd]ata [Ss]ource|[Ee]ndpoint|[Dd]efault[Ee]nd[Pp]oints[Pp]rotocol|[Aa]ccountName|[Da]ata[Ss]ource|[Aa]uthentication|[Ll]ogin|[Ii]nitial[Cc]atalog|DB|Trusted_Connection|authenticationType|DSN|[Dd]ata[Ss]ource[Nn]ame|[Ii]ntegrated[Ss]ecurity|[Ll]ocation|[Ee]ncrypt|[Ss]ystem|[Pp]rotocol|[Hh]ost|[Pp]ort|SRVR|[Dd]river|Dbq|[Ss]sl[Mm]ode|SSL|[Uu]id|DBNAME|SystemDB|[Pp]ersist [Ss]ecurity [Ii]nfo|[Cc]onnection [Tt]ype|[Dd]ata[Ss]ource[Nn]ame|[Ee]xcel [Ff]ile|[Ss]erver [Nn]ame|URL)=[^"'`\x00-\x08\r\n|]* +``` + +
+ +
+Start Pattern + +```regex +\A|["'`]|"|[\r\n]|[=:-] +``` + +
+End Pattern + +```regex +\z|["'`]|"|[\r\n] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + (^|;)([Pp]assword|[Pp]wd|[Ss]hared[Ss]ecret[Vv]alue|[Aa]ccount[Kk]ey|PW|pw|[Cc]ipher [Kk]ey|OAuth Access Token Secret)= + ``` +- Not Match: + + ```regex + (^|;)([Pp]assword|[Pp]wd|[Ss]hared[Ss]ecret[Vv]alue|[Aa]ccount[Kk]ey|PW|pw|[Cc]ipher [Kk]ey|OAuth Access Token Secret)=(%s\b|\{\{[^}]+\}\}|\{[0-9]+\}|\$?\{[^}]+\}|\[[A-Z_]+\]|['"`]|$) + ``` + +
+ ## Database Connection String (1) @@ -207,4 +260,156 @@ _version: v0.1_ \' ``` + + +## SQLAlchemy Database Connection String + + +SQLAlchemy connection strings are used to connect to databases, often with embedded credentials. +_version: v0.1_ + + + +
+Pattern Format + +```regex +[^$/?#@\s][^/?#@\s\x00-\x08]* +``` + +
+ +
+Start Pattern + +```regex +(\A|\b)mysql\+[a-z]+://[^/?#:@\s\x00-\x08]*: +``` + +
+End Pattern + +```regex +@ +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + (?i)^[[{(<]?(?:password|passwd|secret)[\]})>]?$ + ``` +- Not Match: + + ```regex + ^\$?\{[^}+]\}i\}$ + ``` +- Not Match: + + ```regex + ^%(?:\.\*)?s$ + ``` + +
+ +## MongoDB connection string + + + +_version: v0.2_ + + + +
+Pattern Format + +```regex +mongodb(\+[a-z]+)?://[^'"`<>/:@\s\x00-\x08]+:[^'"`<>/@\s\x00-\x08]+@[^?'"`\s\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +\A|\b +``` + +
+End Pattern + +```regex +\z|\s|['"`?] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + (?i):(test|a|my)?[_-]?pass(word)?@ + ``` +- Not Match: + + ```regex + :%(?:\.\*)?[sv]@ + ``` +- Not Match: + + ```regex + :\$?\{[^}+]\}@ + ``` +- Not Match: + + ```regex + ^mongodb\+srv://b\*b%40f3tt%3D:%244to%40L8%3DMC@test3.test.build.10gen.cc/mydb%3F\?replicaSet=repl0 + ``` + +
+ +## JDBC Database Connection String + + + +_version: v0.1_ + + + +
+Pattern Format + +```regex +jdbc:[^:\x00-\x08]+:\/\/[^\/\x00-\x08]+\/[^?\x00-\x08]+\?user=[^&\x00-\x08]+&password=[^\s'"`<{$%*\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +\A|\b +``` + +
+End Pattern + +```regex +\z|\s|['"`<] +``` +
\ No newline at end of file diff --git a/database/__snapshots__/database_connection_string_full.csv b/database/__snapshots__/database_connection_string_full.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/database/__snapshots__/database_connection_string_full.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/database/__snapshots__/jdbc_connection_string.csv b/database/__snapshots__/jdbc_connection_string.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/database/__snapshots__/jdbc_connection_string.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/database/__snapshots__/mongodb_connection_string.csv b/database/__snapshots__/mongodb_connection_string.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/database/__snapshots__/mongodb_connection_string.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/database/__snapshots__/sqlalchemy_connection_string.csv b/database/__snapshots__/sqlalchemy_connection_string.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/database/__snapshots__/sqlalchemy_connection_string.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/database/patterns.yml b/database/patterns.yml index a67da06a..c8185f68 100644 --- a/database/patterns.yml +++ b/database/patterns.yml @@ -8,6 +8,27 @@ name: Database passwords # We don't account for connection strings encoded in UTF-16, but that will be a rare case, if it happens at all patterns: + - name: Database Connection String (full string) + type: database_connection_string_full + description: "Database connection strings are used to connect to databases, often with embedded credentials." + regex: + pattern: | + [^"'`\x00-\x08\r\n|]*([Ss]erver|[Pp]rovider|[Dd]atabase|[Uu]ser [Ii]d|[Dd]ata [Ss]ource|[Ee]ndpoint|[Dd]efault[Ee]nd[Pp]oints[Pp]rotocol|[Aa]ccountName|[Da]ata[Ss]ource|[Aa]uthentication|[Ll]ogin|[Ii]nitial[Cc]atalog|DB|Trusted_Connection|authenticationType|DSN|[Dd]ata[Ss]ource[Nn]ame|[Ii]ntegrated[Ss]ecurity|[Ll]ocation|[Ee]ncrypt|[Ss]ystem|[Pp]rotocol|[Hh]ost|[Pp]ort|SRVR|[Dd]river|Dbq|[Ss]sl[Mm]ode|SSL|[Uu]id|DBNAME|SystemDB|[Pp]ersist [Ss]ecurity [Ii]nfo|[Cc]onnection [Tt]ype|[Dd]ata[Ss]ource[Nn]ame|[Ee]xcel [Ff]ile|[Ss]erver [Nn]ame|URL)=[^"'`\x00-\x08\r\n|]* + start: | + \A|["'`]|"|[\r\n]|[=:-] + end: | + \z|["'`]|"|[\r\n] + additional_match: + - (^|;)([Pp]assword|[Pp]wd|[Ss]hared[Ss]ecret[Vv]alue|[Aa]ccount[Kk]ey|PW|pw|[Cc]ipher [Kk]ey|OAuth Access Token Secret)= + additional_not_match: + # placeholders, variable substitutions, etc. + - (^|;)([Pp]assword|[Pp]wd|[Ss]hared[Ss]ecret[Vv]alue|[Aa]ccount[Kk]ey|PW|pw|[Cc]ipher [Kk]ey|OAuth Access Token Secret)=(%s\b|\{\{[^}]+\}\}|\{[0-9]+\}|\$?\{[^}]+\}|\[[A-Z_]+\]|['"`]|$) + test: + data: connection_string = "Data Source=eee,1433;Initial Catalog=bbb;Integrated Security=False;User ID=ccc@aaa;Password=ddd;" + start_offset: 21 + end_offset: 117 + + - name: Database Connection String (1) type: database_connection_string_1 description: "Database connection strings are used to connect to databases, often with embedded credentials." @@ -113,5 +134,56 @@ patterns: comments: - This is specific to Microsoft SQL Server TSQL syntax + - name: SQLAlchemy Database Connection String + type: sqlalchemy_connection_string + description: SQLAlchemy connection strings are used to connect to databases, often with embedded credentials. + regex: + pattern: | + [^$/?#@\s][^/?#@\s\x00-\x08]* + start: | + (\A|\b)mysql\+[a-z]+://[^/?#:@\s\x00-\x08]*: + end: | + @ + additional_not_match: + - (?i)^[[{(<]?(?:password|passwd|secret)[\]})>]?$ + - ^\$?\{[^}+]\}i\}$ + - ^%(?:\.\*)?s$ + test: + data: | + engine = create_engine('mysql+mysqlconnector://scott:tiger@localhost/foo') + start_offset: 53 + end_offset: 58 + - name: MongoDB connection string + type: mongodb_connection_string + regex: + version: 0.2 + pattern: | + mongodb(\+[a-z]+)?://[^'"`<>/:@\s\x00-\x08]+:[^'"`<>/@\s\x00-\x08]+@[^?'"`\s\x00-\x08]+ + start: | + \A|\b + end: | + \z|\s|['"`?] + additional_not_match: + - (?i):(test|a|my)?[_-]?pass(word)?@ + - :%(?:\.\*)?[sv]@ + - :\$?\{[^}+]\}@ + - ^mongodb\+srv://b\*b%40f3tt%3D:%244to%40L8%3DMC@test3.test.build.10gen.cc/mydb%3F\?replicaSet=repl0 + test: + data: mongodb+srv://foo:bar@mongodb.example.invalid/ + - name: JDBC Database Connection String + type: jdbc_connection_string + regex: + version: 0.1 + pattern: | + jdbc:[^:\x00-\x08]+:\/\/[^\/\x00-\x08]+\/[^?\x00-\x08]+\?user=[^&\x00-\x08]+&password=[^\s'"`<{$%*\x00-\x08]+ + start: | + \A|\b + end: | + \z|\s|['"`<] + test: + data: | + jdbc:mysql://localhost:3306/mydb?user=root&password=toor + start_offset: 0 + end_offset: 56 diff --git a/generic/README.md b/generic/README.md index dd195f4d..5061513a 100644 --- a/generic/README.md +++ b/generic/README.md @@ -7,13 +7,15 @@ ## Generic Passwords +**⚠️ WARNING: THIS RULE IS EXPERIMENTAL AND MIGHT CAUSE A HIGH FALSE POSITIVE RATE (test before commiting to org level) ⚠️** - -_version: v0.4_ +_version: v0.5_ **Comments / Notes:** +- Likely to cause large numbers of false positives - use with caution + - `password`, `secret`, `key`, or password like prefix (fuzzy) - Delimiters like `=` or `:` (with padding) @@ -36,7 +38,7 @@ _version: v0.4_ Start Pattern ```regex -(?:\A|[^a-zA-Z0-9])(?i)(?:api|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret)([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? +(?:\A|[^a-zA-Z0-9])(?i)[a-z0-9_.-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? ```
@@ -57,7 +59,7 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - ^(?i)(?:[a-z0-9_.]*,\s*)?(?:str\()?[[<(]?(?:(?:(?:user|key)_?)?(?:[a-zA-Z0-9._]+[_.])?(?:the )?(?:pass?(wo?r?d|code|phrase)|pass|pwd|secret|token|tok|redacted|placeholder|dummy|pw|thephrase)|write|read|on|off|true|false|none|null|nil|undefined|eof|ignore|eol|git|yes|no|y|n),?[\]>)]?(?:\)\s*\{)?\\?( or )?$ + ^(?i)_?\)?((a-zA-Z0-9._]+[_.])?(?:the )?(?:pass?(wo?r?d|code|phrase)|pass|pwd|secret|token|key|tok|pw)|redacted|placeholder|dummy|thephrase|write|read|on|off|true|false|none|value|null( \? )?|nil|undefined|eof|ignore|eol|git|yes|no|y|n|f[0-9]{1,2}|[a-zA-Z]),?\s*\){0,2}[\]>)]?(?:\)\s*\{)?\\?(( or | \|\| ).*)?$ ``` - Not Match: @@ -67,17 +69,287 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - ^\s*(?:\.\.\.|\\|\\n|\\0|[,()[\]{}`.]\\?|-[)(]|0x[A-Fa-f0-9]+|[0-9]{1,4}|(?:~|/tmp|\.\.|\.)|\\{1,2}w\+/g,( \\?)?|%[sr]|geheim\$parole|\([Oo]ptional\).*|\$?(?:\{\{?[^}]+\}\}?|\(\(?[^)]+\)\)?|\[\[?[^\]+]\]\]?))?,?\s*(?:\s*(?:/\*|#|//).*)?$ + ^\s*(?:\.\.\.|\\|\\n|\\0|\?|\$\(|[,()[\]{}`.]\\?|-[)(]|\\f21b|0x[A-Fa-f0-9]+|[0-9]{1,4}|(?:~|/tmp|\.\.|\.|(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9]+(\.(pem|crt|key|cer|pub|der)|_rsa))|\\{1,2}w\+/g,( \\?)?|%[sr]|geheim\$parole|\([Oo]ptional\).*|\$?(?:\{\{?[^}]+\}\}?|\(\(?[^)]+\)\)?|\[\[?[^\]+]\]\]?)|(before|hover|focus)(,| \{))?,?\s*(?:\s*(?:/\*|#|//).*)?$ + ``` +- Not Match: + + ```regex + ^(?:function\s*\([^)]*\)\s*{\s*.*|\([^)]*\)\s*=>\s*(?:{\s*|[^;)]+[;)])|(?:new |\([A-Za-z]+\)\s*)?[a-zA-Z0-9_.]+\s*\(.*|(?:public|private) [A-Za-z0-9_]+ \{|[A-Za-z0-9_.-]+\s*\) \{)$|\{\{[^}]+\}\}|\$\{\{|\{\}$|\[\]$|(0x)?%[0-9]+x|%[dusx]\.$ + ``` +- Not Match: + + ```regex + ^\s*(?:(?:self|this)\.[a-zA-Z_][a-zA-Z0-9_.]+[,[]?|[a-zA-Z0-9_.]+\[(?:[a-zA-Z0-9_.]+)?\]?|\$(?:[1-9]|[A-Za-z0-9_]+)\{?|os\.environ\[[^\]]\]|process\.env\.[A-Z0-9_]+)\s*(?:,|\|\||&&)?\s*$|`(%s|\+) + ``` + +
+ +## Generic Passwords (fewer FPs) + + + +_version: v0.1_ + +**Comments / Notes:** + + +- Expect many false positives if run against vendored-in code, such as JS libraries - use with caution + +- `password`, `secret`, `key`, `token` etc. password-like prefix (fuzzy) + +- Delimiters like `=` or `:` (with padding) + +- Matches fewer characters (A-Za-z0-9_/+.!-), and requires matching quotes around the string + +- Attempts to remove variables, placeholders, and common configuration constants such as 'read' and 'write' + + +
+Pattern Format + +```regex +((?i)[a-z0-9_.-]*(api|auth[a-z]*|jwt|mysql|db)[_.-]?)?((?i)pass?(wo?r?d|code|phrase)|secret|token|key)([_-][A-Za-z0-9]+){0,4}_{0,2}(["'`]|[ \t]+As[ \t]+String)?[\t ]*(?:=|>|:{1,3}=|\|\|:|<=|=>|:|\?=)[\t ]*([br]?"[A-Za-z0-9_/+.!-]+"|[br]?'[A-Za-z0-9_/+.!-]+') +``` + +
+ +
+Start Pattern + +```regex +\A|[^0-9A-Za-z] +``` + +
+End Pattern + +```regex +\z|[^A-Za-z0-9] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + ^[A-Za-z0-9_.-]*(key|KEY|[Tt]oken|TOKEN)(_[a-zA-Z]+)?['"]?\s*([:=]|=>)\s*["']([Ee]mploye[er]|[Ss]taff|([Ss]earch)?[Rr]esult|[a-z][a-zA-Z]+CSX[A-Z][A-Za-z]+|[A-Za-z]*[Bb]ase(64|32|58)|object|claret|assigns?|clean|contains|error|expand|generate|hoist|indent(ation)?|invert|jumps?|pairs?|param(eter)?s?|pop|rewrite|temp(orary)?|token(s|i[sz]e)?|type|((un)?(quote|shift|wrap|finished))|[a-z]{2,10}([A-Z][a-z]{1,15}){1,6}|(compile|is|has|make|add|each|check|close|cache|format|tag|get|set)([A-Z][A-Za-z]+)?|gadget|classic|(try_)?(base|mode|grade|model)|words|identifier|[a-z.-]+\.(jpe?g|(x|ht)ml|txt|docx?|xlsx?|pdf|png)|enabled|name\.invalidPattern|\.|\.data-api|expect|file|config|ansi|Default(Type)?|Cache-Control|((notD|d)eepE|e)qual|name|NAME|package|version|VERSION|start|end|step|async|Event|throws|ok|notOK|verbose|push(Result)?|slimAssertions|(p|notP)ropEqual|((notS|s)trict|not)Equal|value|prev|next|year|key[0-9]?|destroy|[a-z]+EventListeners|timeout|str(ing)?|hmac|uuid|update|find|true|false|val|VAL|REDACTED|redacted|nop|F[0-9]{1,2}|[A-Za-z0-9]|[Nn][ui]ll|[Nn]one|[a-z_]+\.((tf)?state|id|key)|(hibernate|ws|err|i18n|employee|bs|org|com|sun|java)(\.[a-zA-Z0-9_]+){1,4}|[A-Z_]+_KEY)["']$ + ``` +- Not Match: + + ```regex + (?i)(token|key)[_-](name|format|type|enabled|success|type|method)\b ``` - Not Match: ```regex - ^(?:function\s*\([^)]*\)\s*{\s*.*|\([^)]*\)\s*=>\s*(?:{\s*|[^;)]+[;)])|(?:new )?[a-zA-Z0-9_.]+\(.*|(?:public|private) [A-Za-z0-9_]+ \{)$ + ^(?i)token(_[A-Z]+)?['"]?\s*[:=]\s*['"](barline|parenthesis|qualified|suport|symbol|statementEnd|singleLineTitle|character|pageBreak|operator|optionalTitle|option|zupfnoter|chordname|macro|error|escape|indent|term|titleUnderline|tag|link|literal|(other|table)Block|list|value|control|set|support|injections|array|doc|source|heading|tokens|storage|empty|newline|empty_line|keyword|(line)?comment|meta|[lr]?paren|class|punctuation|regexp?|constant|string|entity|invalid|support|variable|multiline|language|paren|markup|singleline|nospell|text|array|doc|source|heading|tokens)(\.{1,2}[A-Za-z0-9_-]+){0,6}[!.]?["']$ ``` - Not Match: ```regex - ^\s*(?:(?:self|this)\.[a-zA-Z_][a-zA-Z0-9_.]+[,[]?|[a-zA-Z0-9_.]+\[(?:[a-zA-Z0-9_.]+)?\]?|\$(?:[1-9]|[A-Za-z0-9_]+)\{?|os\.environ\[[^\]]\]|process\.env\.[A-Z0-9_]+)\s*(?:,|\|\||&&)?\s*$ + ^KEY_[A-Z]+[0-9]{0,3}: 'k[a-zA-Z0-9]{1,6}'$ + ``` +- Not Match: + + ```regex + ['"` ](/dev/u?random|(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)|https?://.*|file://.*)['"`]$ + ``` + +
+ +## Generic Password with hex encoded secrets + + + +_version: v0.1_ + +**Comments / Notes:** + + +- `password`, `secret`, `key`, or password like prefix (fuzzy) + +- Delimiters like `=` or `:` (with padding) + +- Has to be a token-like value - a 32, 40 or 64 character hex string + + +
+Pattern Format + +```regex +[0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} +``` + +
+ +
+Start Pattern + +```regex +(?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? +``` + +
+End Pattern + +```regex +(\z|[\r\n'"]) +``` + +
+ +## Generic Password with Base64 encoded secrets + + + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- `password`, `secret`, `key`, or password like prefix (fuzzy) + +- Delimiters like `=` or `:` (with padding) + + +
+Pattern Format + +```regex +(([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) +``` + +
+ +
+Start Pattern + +```regex +(?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? +``` + +
+End Pattern + +```regex +(\z|[\r\n'"]) +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ + ``` + +
+ +## Generic Password with URI-safe Base64 encoded secrets + + + +_version: v0.1_ + +**Comments / Notes:** + + +- The Base64 must contain numbers, upper case and lower case and be at least 12 characters long + +- `password`, `secret`, `key`, or password like prefix (fuzzy) + +- Delimiters like `=` or `:` (with padding) + +- This matches _- instead of +/, for URI-safe Base64 + + +
+Pattern Format + +```regex +(([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) +``` + +
+ +
+Start Pattern + +```regex +(?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? +``` + +
+End Pattern + +```regex +(\z|[\r\n'"]) +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [0-9] + ``` + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` + +- Match: + + ```regex + ^.{12,}$ ```
@@ -156,13 +428,15 @@ _version: v0.2_ - As used in an Authorization header - We try to remove common placeholders + +- Lower length limit of 12 to remove common false positives on "Token ", since most words are below 12 characters in length
Pattern Format ```regex -[a-zA-Z0-9_.=/+:-]+ +[a-zA-Z0-9_.=/+:-]{12,} ```
@@ -171,7 +445,7 @@ _version: v0.2_ Start Pattern ```regex -\b([Bb]earer|[Tt]oken)[ ]+ +(Authorization: |['"])([Bb]earer |[Tt]oken (token=)?) ```
@@ -192,7 +466,7 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - ^(?:letmein|Oracle|SuperSecretString|foo|ababbdbbebbbebdbbe5538003023|XYZ_INVALID_ACCESTOKEN_XYZ|QQ==|Shizuku|mF_9.B5f-4.1JqM|h480djs93hd8|SlAV32hkKG)$ + ^(?:letmein|Oracle|SuperSecretString|foo|ababbdbbebbbebdbbe5538003023|XYZ_INVALID_ACCESTOKEN_XYZ|QQ==|Shizuku|mF_9.B5f-4.1JqM|h480djs93hd8|SlAV32hkKG|YmVlcDpib29w)$ ``` - Not Match: @@ -207,12 +481,7 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - (?i)x{5} - ``` -- Not Match: - - ```regex - ^(?i)(x+|y+|z+|a+|\.+|.*\.\.\.)$ + (^(?i)(x+|y+|z+|a+|\.+|.*\.\.\.)$|(?i)x{5}) ```
diff --git a/generic/__snapshots__/generic_passwords.csv b/generic/__snapshots__/generic_passwords.csv index 384a50bb..b0aaa6f8 100644 --- a/generic/__snapshots__/generic_passwords.csv +++ b/generic/__snapshots__/generic_passwords.csv @@ -1 +1,57 @@ secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column +"generic_passwords","Generic Passwords","7c7853e3659d1c01e65f3cb460ac07d079288bfa5bc21aae3d31fe01a0814278","configs/application.properties","31","31","28","34", +"generic_passwords","Generic Passwords","dbfdae0d66bb259d57896c533fa8d1d2bb5f3e685c3d1a32fbddc0109ebb13f8","common/top-passwords-shortlist.txt","24","24","10","16", +"generic_passwords","Generic Passwords","8799e334b94c1c08dbc46b3887d86c1fb12a0080634d8ee72084ca536cb2342c","common/top-passwords-shortlist.txt","19","19","10","18", +"generic_passwords","Generic Passwords","d90886c29bdf153471809be45596c83863e30586f10b7f38f157b965f8904981","common/top-passwords-shortlist.txt","23","23","10","16", +"generic_passwords","Generic Passwords","9e6bbe76d8fafd22ff9293a04934361d2fcd709081272a005ab83ca5e66d6d16","database/tsql_create_user.sql","1","1","36","46", +"generic_passwords","Generic Passwords","067d2f107733c6928d1699ae06855f1895747e6c0709cbc9807ecfe6c0c1c2dd","common/top-passwords-shortlist.txt","31","31","10","16", +"generic_passwords","Generic Passwords","49924b9a100a8aa9dab6014398ed23d567f59036f14de87175387fbc02dba7cc","database/docker_sqlserver.yml","14","14","122","133", +"generic_passwords","Generic Passwords","3820fe31b7bd9e2090f4a1f49c66f36006f8eedfb6552af35fe627bc070f27ae","configs/application.properties","29","29","28","35", +"generic_passwords","Generic Passwords","66483b17db44f612efcb123bd998ed1f091b778aeb072f89f4b49bad370a7809","common/top-passwords-shortlist.txt","27","27","10","17", +"generic_passwords","Generic Passwords","8d67877bcf2b6cba2f619aae5a3c6eff8210b74a26fa0466147622fc65237e2b","configs/application.properties","43","43","28","35", +"generic_passwords","Generic Passwords","2eca97ad9e30f2ed064b3e6ecb2cd2e0ab8a05b5bbd54842df31abaa7477e989","configs/mysql/run-mysql.sh","5","5","28","42", +"generic_passwords","Generic Passwords","2eca97ad9e30f2ed064b3e6ecb2cd2e0ab8a05b5bbd54842df31abaa7477e989","docker/mysql/run-mysql.sh","5","5","28","42", +"generic_passwords","Generic Passwords","ffd0f84644936fceee82fffb1bb6056922126e05be040c84007c37a26fe6d9f2","common/top-passwords-shortlist.txt","4","4","10","17", +"generic_passwords","Generic Passwords","5f07dd6c436eefbd1ee0a62f2353dab5523c2a6a7e007dd4dc04ad3cb225348e","configs/application.properties","50","50","28","35", +"generic_passwords","Generic Passwords","90ff950e50d23ac032cf5705fc9b783fcf9c9a3d324b0c5e845a79da54d42ead","configs/application.properties","12","12","28","45", +"generic_passwords","Generic Passwords","4813494d137e1631bba301d5acab6e7bb7aa74ce1185d456565ef51d737677b2","configs/application.properties","10","10","30","34", +"generic_passwords","Generic Passwords","81ea24e895ec1e9ec08a8763a773b4253e1bc682bea01cb2a4e99b81e866c97d","common/top-passwords-shortlist.txt","14","14","10","16", +"generic_passwords","Generic Passwords","544e62cee8033709e389e5b2755343d0d0fa8c4850215cfb6331717e80d1aea3","jwt/owasp-juice-shop.ts","207","207","20","52", +"generic_passwords","Generic Passwords","9584c5ca00f5e717366a1c441d851c3daf00e4dd190b85e1cb0cb0d36af688de","configs/application.properties","36","36","28","35", +"generic_passwords","Generic Passwords","24795a7bb3efee18ee90ad0dfbf6cf7f444de648e49a39b2657ab36dd2a8cbc4","common/top-passwords-shortlist.txt","32","32","10","16", +"generic_passwords","Generic Passwords","066b91577bc547e21aa329c74d74b0e53e29534d4cc0ad455abba050121a9557","common/top-passwords-shortlist.txt","22","22","10","17", +"generic_passwords","Generic Passwords","6ab5c67640a9b1f875050b158cdbb1d081ee76ff8e703279062ce8652a372332","configs/postgres/docker-compose.yml","12","12","95","102", +"generic_passwords","Generic Passwords","b34e5475b55d84c7de90e07142d9427477c757a46a8e8712f7bc2f92dd6c6473","jwt/owasp-juice-shop.ts","43","43","20","29", +"generic_passwords","Generic Passwords","e4ad93ca07acb8d908a3aa41e920ea4f4ef4f26e7f86cf8291c5db289780a5ae","common/top-passwords-shortlist.txt","18","18","10","18", +"generic_passwords","Generic Passwords","9f214b49fafc432e116884a5244442203267f200e72c6ad3ad4dbfd7b0f089c3","jwt/owasp-juice-shop.ts","146","146","22","37", +"generic_passwords","Generic Passwords","1e744b6a3177a1165c1d67ccfe8989267e8364189d9124aa87a756aa219db83c","configs/example.yml","5","5","15","38", +"generic_passwords","Generic Passwords","fc93cb07e1ad92898527100e58a1cf1d1e7f65e9a266a6f87f3c84feb541c7b3","jwt/example.txt","2","2","12","15", +"generic_passwords","Generic Passwords","e864c10ad586803d73b4e7a684ec1c8e2c2c9cd0a8718d2924bb50afe77c8d4b","common/top-passwords-shortlist.txt","28","28","10","15", +"generic_passwords","Generic Passwords","b4b215eb04965939aab0e90f828c2e00cb17f147926b3f3de28901e3e02013ef","common/top-passwords-shortlist.txt","21","21","10","16", +"generic_passwords","Generic Passwords","bdc7f0fb11236645c4c4d7aba2afab3649d5df400779061f1d034803a811c622","common/top-passwords-shortlist.txt","16","16","10","20", +"generic_passwords","Generic Passwords","50b15954a84bfb3594a8ef96c10dd2fe25c27341db7c443ac1220ecb00f3f16a","common/top-passwords-shortlist.txt","36","36","10","16", +"generic_passwords","Generic Passwords","20fdf64da3cd2c78ec3c033d2ac628bacf701711fa99435ee37bef0304800dc5","common/top-passwords-shortlist.txt","15","15","10","17", +"generic_passwords","Generic Passwords","e2587f6c678061df35dbff7fee253348085045f1a4d68549f0e4261330af080c","configs/mysql/docker-compose.yml","8","8","28","51", +"generic_passwords","Generic Passwords","e2587f6c678061df35dbff7fee253348085045f1a4d68549f0e4261330af080c","docker/mysql/docker-compose.yml","8","8","28","51", +"generic_passwords","Generic Passwords","592c75c07fbc94b2eb5301bf67264dbd659a0f9c519b100d1f61db03e24dce77","common/top-passwords-shortlist.txt","2","2","10","15", +"generic_passwords","Generic Passwords","406c0a9e14e3d00a364c559acde4192c6059163f7ef33e0820207e369338fa7b","common/top-passwords-shortlist.txt","29","29","10","17", +"generic_passwords","Generic Passwords","1c20528f953116dfa0b8755308b7ee5ab14e5a2efdef2f90e65ce33526757351","common/top-passwords-shortlist.txt","6","6","10","16", +"generic_passwords","Generic Passwords","1c8bfe8f801d79745c4631d09fff36c82aa37fc4cce4fc946683d7b336b63032","common/top-passwords-shortlist.txt","13","13","10","17", +"generic_passwords","Generic Passwords","09d73fcb2ef34f11781d8f35d88428b9b90c3431dc82417ad161655cf1c465ed","common/top-passwords-shortlist.txt","17","17","10","16", +"generic_passwords","Generic Passwords","2b08a09b43e6d9e6a51a2ad5fcdfc6532e95da6d72cd0ef5f65d8e6ae86ca3e5","common/top-passwords-shortlist.txt","34","34","10","16", +"generic_passwords","Generic Passwords","ddd9a2b20bba26222d4d886042639f855ddce06f41a8c5df391b79fd15367ccc","common/top-passwords-shortlist.txt","33","33","10","16", +"generic_passwords","Generic Passwords","8bb0cf6eb9b17d0f7d22b456f121257dc1254e1f01665370476383ea776df414","common/top-passwords-shortlist.txt","7","7","10","17", +"generic_passwords","Generic Passwords","cb1b7759c2c93af5a9359769ec4fa443112b4b7c145e99767c16620aaa3c2c6c","configs/mysql/docker-compose.yml","11","11","23","36", +"generic_passwords","Generic Passwords","cb1b7759c2c93af5a9359769ec4fa443112b4b7c145e99767c16620aaa3c2c6c","docker/mysql/docker-compose.yml","11","11","23","36", +"generic_passwords","Generic Passwords","181b99e27e562b1237d009179d010d5774a853606551d7ac983b8c3daa9c5163","generic/passwords.js","3","3","13","34", +"generic_passwords","Generic Passwords","ae9216f925f35e1784e790ba36020358acd29bbc0a8087ce5caf72d31560372f","common/top-passwords-shortlist.txt","26","26","10","15", +"generic_passwords","Generic Passwords","b45f549d9212dac4edc9a16b6d18246fb5dae1c9c0d1d830a75b54573c06ee12","common/top-passwords-shortlist.txt","12","12","10","16", +"generic_passwords","Generic Passwords","bef57ec7f53a6d40beb640a780a639c83bc29ac8a9816f1fc6c5c6dcd93c4721","common/top-passwords-shortlist.txt","8","8","10","16", +"generic_passwords","Generic Passwords","f081f7b8d4310e67a7572f60b6070a3034d5f1ae1465b3fe4f8dafca9213a0e3","common/top-passwords-shortlist.txt","30","30","10","18", +"generic_passwords","Generic Passwords","04d116726bdc3b35faf64ccd34af64c5308b3433e35b33db06219f071275fde8","common/top-passwords-shortlist.txt","20","20","10","18", +"generic_passwords","Generic Passwords","1fb8edf1e0156e7eb349146b4c1f91ea88821c269bb290afeb21b90ef1bb379c","generic/passwords.js","1","1","15","26", +"generic_passwords","Generic Passwords","d451823f5c7ea01b991edb52a7890624607a24f1b8e00f2e89f6807433f079ec","configs/application.properties","14","14","29","49", +"generic_passwords","Generic Passwords","65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5","common/top-passwords-shortlist.txt","11","11","10","16", +"generic_passwords","Generic Passwords","0fb769916f2a1adbca016f75e580284fd7bc26ab6e91e094d1096681445f62d3","generic/passwords.js","7","7","16","31", +"generic_passwords","Generic Passwords","464c7a646393b68d1a42076c010b5aae418d8d322f233ca0b8cd8e2c6bcd9676","common/top-passwords-shortlist.txt","10","10","10","14", +"generic_passwords","Generic Passwords","c471039f93e753cbc8c09509e9f937af8633c7a595267536714db11916779c81","common/top-passwords-shortlist.txt","3","3","10","19", diff --git a/generic/__snapshots__/generic_passwords_base64.csv b/generic/__snapshots__/generic_passwords_base64.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/generic/__snapshots__/generic_passwords_base64.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/generic/__snapshots__/generic_passwords_base64_uri.csv b/generic/__snapshots__/generic_passwords_base64_uri.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/generic/__snapshots__/generic_passwords_base64_uri.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/generic/__snapshots__/generic_passwords_fewer_fps.csv b/generic/__snapshots__/generic_passwords_fewer_fps.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/generic/__snapshots__/generic_passwords_fewer_fps.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/generic/__snapshots__/generic_passwords_hex.csv b/generic/__snapshots__/generic_passwords_hex.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/generic/__snapshots__/generic_passwords_hex.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/generic/patterns.yml b/generic/patterns.yml index afe9aa3b..409432d2 100644 --- a/generic/patterns.yml +++ b/generic/patterns.yml @@ -3,34 +3,33 @@ name: Generic Secrets / Passwords patterns: - name: Generic Passwords type: generic_passwords + experimental: true regex: - version: 0.4 + version: 0.5 pattern: | [a-zA-Z0-9!.,$%&*+?^_`{|}()[\]\\/~-][a-zA-Z0-9\t !.,$%&*+?^_`{|}()[\]\\/~-]* start: | - (?:\A|[^a-zA-Z0-9])(?i)(?:api|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret)([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? + (?:\A|[^a-zA-Z0-9])(?i)[a-z0-9_.-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? end: | (\z|[\r\n'"]) additional_not_match: # something that means "password" or a placeholder name - either a variable or a placeholder # a literal value or config switch - - ^(?i)(?:[a-z0-9_.]*,\s*)?(?:str\()?[[<(]?(?:(?:(?:user|key)_?)?(?:[a-zA-Z0-9._]+[_.])?(?:the )?(?:pass?(wo?r?d|code|phrase)|pass|pwd|secret|token|tok|redacted|placeholder|dummy|pw|thephrase)|write|read|on|off|true|false|none|null|nil|undefined|eof|ignore|eol|git|yes|no|y|n),?[\]>)]?(?:\)\s*\{)?\\?( or )?$ + - ^(?i)_?\)?((a-zA-Z0-9._]+[_.])?(?:the )?(?:pass?(wo?r?d|code|phrase)|pass|pwd|secret|token|key|tok|pw)|redacted|placeholder|dummy|thephrase|write|read|on|off|true|false|none|value|null( \? )?|nil|undefined|eof|ignore|eol|git|yes|no|y|n|f[0-9]{1,2}|[a-zA-Z]),?\s*\){0,2}[\]>)]?(?:\)\s*\{)?\\?(( or | \|\| ).*)?$ # Python type hints, Swift typing - ^\s*(?:(?:typing\.)?(?:(?:[Tt]uple|[Ll]ist|[Dd]ict|Callable|Iterable|Sequence|Optional|Union)\[.*|(?:int|str|float|(?:typing.)?Any|None|bytes|bool|ReadableBuffer)\s*(?:[,|].*)?|(?:Int|Swift\.Int|Int32)\.*))\s*$ # ..., \, , \n, \0, ',' and other single chars, smilies, hex, digits, nothing at all, # directories, regex, format string placeholder, urllib demo passphrase, "optional" in docs, a variable substitution, or surrounded by brackets of various kinds # all with possible ',' and surrounding whitespace, possibly with a following comment - - ^\s*(?:\.\.\.|\\|\\n|\\0|[,()[\]{}`.]\\?|-[)(]|0x[A-Fa-f0-9]+|[0-9]{1,4}|(?:~|/tmp|\.\.|\.)|\\{1,2}w\+/g,( \\?)?|%[sr]|geheim\$parole|\([Oo]ptional\).*|\$?(?:\{\{?[^}]+\}\}?|\(\(?[^)]+\)\)?|\[\[?[^\]+]\]\]?))?,?\s*(?:\s*(?:/\*|#|//).*)?$ + - ^\s*(?:\.\.\.|\\|\\n|\\0|\?|\$\(|[,()[\]{}`.]\\?|-[)(]|\\f21b|0x[A-Fa-f0-9]+|[0-9]{1,4}|(?:~|/tmp|\.\.|\.|(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9]+(\.(pem|crt|key|cer|pub|der)|_rsa))|\\{1,2}w\+/g,( \\?)?|%[sr]|geheim\$parole|\([Oo]ptional\).*|\$?(?:\{\{?[^}]+\}\}?|\(\(?[^)]+\)\)?|\[\[?[^\]+]\]\]?)|(before|hover|focus)(,| \{))?,?\s*(?:\s*(?:/\*|#|//).*)?$ # function definitions, e.g. Javascript, function calls or variable declaration - - ^(?:function\s*\([^)]*\)\s*{\s*.*|\([^)]*\)\s*=>\s*(?:{\s*|[^;)]+[;)])|(?:new )?[a-zA-Z0-9_.]+\(.*|(?:public|private) [A-Za-z0-9_]+ \{)$ + - ^(?:function\s*\([^)]*\)\s*{\s*.*|\([^)]*\)\s*=>\s*(?:{\s*|[^;)]+[;)])|(?:new |\([A-Za-z]+\)\s*)?[a-zA-Z0-9_.]+\s*\(.*|(?:public|private) [A-Za-z0-9_]+ \{|[A-Za-z0-9_.-]+\s*\) \{)$|\{\{[^}]+\}\}|\$\{\{|\{\}$|\[\]$|(0x)?%[0-9]+x|%[dusx]\.$ # reference to a member variable, index into a variable, bash variables, perl hash key index, environment vars - - ^\s*(?:(?:self|this)\.[a-zA-Z_][a-zA-Z0-9_.]+[,[]?|[a-zA-Z0-9_.]+\[(?:[a-zA-Z0-9_.]+)?\]?|\$(?:[1-9]|[A-Za-z0-9_]+)\{?|os\.environ\[[^\]]\]|process\.env\.[A-Z0-9_]+)\s*(?:,|\|\||&&)?\s*$ - + - ^\s*(?:(?:self|this)\.[a-zA-Z_][a-zA-Z0-9_.]+[,[]?|[a-zA-Z0-9_.]+\[(?:[a-zA-Z0-9_.]+)?\]?|\$(?:[1-9]|[A-Za-z0-9_]+)\{?|os\.environ\[[^\]]\]|process\.env\.[A-Z0-9_]+)\s*(?:,|\|\||&&)?\s*$|`(%s|\+) test: data: password=Password123 start_offset: 9 end_offset: -1 - expected: - name: passwords.js start_offset: 14 @@ -41,13 +40,117 @@ patterns: - name: passwords.js start_offset: 97 end_offset: 112 - comments: + - "Likely to cause large numbers of false positives - use with caution" - "`password`, `secret`, `key`, or password like prefix (fuzzy)" - "Delimiters like `=` or `:` (with padding)" - "String with a number of chars until a breaking char" - "Not matching variables, placeholders or common configuration constants such as 'read' and 'write'" + + - name: Generic Passwords (fewer FPs) + type: generic_passwords_fewer_fps + regex: + version: 0.1 + pattern: | + ((?i)[a-z0-9_.-]*(api|auth[a-z]*|jwt|mysql|db)[_.-]?)?((?i)pass?(wo?r?d|code|phrase)|secret|token|key)([_-][A-Za-z0-9]+){0,4}_{0,2}(["'`]|[ \t]+As[ \t]+String)?[\t ]*(?:=|>|:{1,3}=|\|\|:|<=|=>|:|\?=)[\t ]*([br]?"[A-Za-z0-9_/+.!-]+"|[br]?'[A-Za-z0-9_/+.!-]+') + start: | + \A|[^0-9A-Za-z] + end: | + \z|[^A-Za-z0-9] + additional_not_match: + - |- + ^[A-Za-z0-9_.-]*(key|KEY|[Tt]oken|TOKEN)(_[a-zA-Z]+)?['"]?\s*([:=]|=>)\s*["']([Ee]mploye[er]|[Ss]taff|([Ss]earch)?[Rr]esult|[a-z][a-zA-Z]+CSX[A-Z][A-Za-z]+|[A-Za-z]*[Bb]ase(64|32|58)|object|claret|assigns?|clean|contains|error|expand|generate|hoist|indent(ation)?|invert|jumps?|pairs?|param(eter)?s?|pop|rewrite|temp(orary)?|token(s|i[sz]e)?|type|((un)?(quote|shift|wrap|finished))|[a-z]{2,10}([A-Z][a-z]{1,15}){1,6}|(compile|is|has|make|add|each|check|close|cache|format|tag|get|set)([A-Z][A-Za-z]+)?|gadget|classic|(try_)?(base|mode|grade|model)|words|identifier|[a-z.-]+\.(jpe?g|(x|ht)ml|txt|docx?|xlsx?|pdf|png)|enabled|name\.invalidPattern|\.|\.data-api|expect|file|config|ansi|Default(Type)?|Cache-Control|((notD|d)eepE|e)qual|name|NAME|package|version|VERSION|start|end|step|async|Event|throws|ok|notOK|verbose|push(Result)?|slimAssertions|(p|notP)ropEqual|((notS|s)trict|not)Equal|value|prev|next|year|key[0-9]?|destroy|[a-z]+EventListeners|timeout|str(ing)?|hmac|uuid|update|find|true|false|val|VAL|REDACTED|redacted|nop|F[0-9]{1,2}|[A-Za-z0-9]|[Nn][ui]ll|[Nn]one|[a-z_]+\.((tf)?state|id|key)|(hibernate|ws|err|i18n|employee|bs|org|com|sun|java)(\.[a-zA-Z0-9_]+){1,4}|[A-Z_]+_KEY)["']$ + - |- + (?i)(token|key)[_-](name|format|type|enabled|success|type|method)\b + - |- + ^(?i)token(_[A-Z]+)?['"]?\s*[:=]\s*['"](barline|parenthesis|qualified|suport|symbol|statementEnd|singleLineTitle|character|pageBreak|operator|optionalTitle|option|zupfnoter|chordname|macro|error|escape|indent|term|titleUnderline|tag|link|literal|(other|table)Block|list|value|control|set|support|injections|array|doc|source|heading|tokens|storage|empty|newline|empty_line|keyword|(line)?comment|meta|[lr]?paren|class|punctuation|regexp?|constant|string|entity|invalid|support|variable|multiline|language|paren|markup|singleline|nospell|text|array|doc|source|heading|tokens)(\.{1,2}[A-Za-z0-9_-]+){0,6}[!.]?["']$ + - "^KEY_[A-Z]+[0-9]{0,3}: 'k[a-zA-Z0-9]{1,6}'$" + - |- + ['"` ](/dev/u?random|(/[a-zA-Z0-9./_-]+/)?[a-zA-Z0-9_-]{5,}(\.(pem|crt|key|cer|pub|der)|_rsa)|https?://.*|file://.*)['"`]$ + test: + data: password="Password123" + start_offset: 0 + end_offset: 22 + comments: + - "Expect many false positives if run against vendored-in code, such as JS libraries - use with caution" + - "`password`, `secret`, `key`, `token` etc. password-like prefix (fuzzy)" + - "Delimiters like `=` or `:` (with padding)" + - "Matches fewer characters (A-Za-z0-9_/+.!-), and requires matching quotes around the string" + - "Attempts to remove variables, placeholders, and common configuration constants such as 'read' and 'write'" + + + - name: Generic Password with hex encoded secrets + type: generic_passwords_hex + regex: + version: 0.1 + pattern: | + [0-9a-f]{32}|[0-9a-f]{40}|[0-9a-f]{64} + start: | + (?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? + end: | + (\z|[\r\n'"]) + test: + data: password=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef + start_offset: 9 + end_offset: -1 + comments: + - "`password`, `secret`, `key`, or password like prefix (fuzzy)" + - "Delimiters like `=` or `:` (with padding)" + - "Has to be a token-like value - a 32, 40 or 64 character hex string" + + + - name: Generic Password with Base64 encoded secrets + type: generic_passwords_base64 + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9+/]){4})*([A-Za-z0-9+/]{4}|[A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==) + start: | + (?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? + end: | + (\z|[\r\n'"]) + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + test: + data: password="AAAAAAAAAAAa00==" + start_offset: 10 + end_offset: 26 + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "`password`, `secret`, `key`, or password like prefix (fuzzy)" + - "Delimiters like `=` or `:` (with padding)" + + + - name: Generic Password with URI-safe Base64 encoded secrets + type: generic_passwords_base64_uri + regex: + version: 0.1 + pattern: | + (([A-Za-z0-9_-]){4})*([A-Za-z0-9_-]{4}|[A-Za-z0-9_-]{3}=|[A-Za-z0-9_-]{2}==) + start: | + (?:\A|[^a-zA-Z0-9])(?i)[a-z0-9._-]*(?:api|auth[a-z]+|jwt|mysql|db)?[_.-]?(?:pass?(?:wo?r?d|code|phrase)|secret|key|token)([_-][a-z0-9]+){0,3}([ \t]+As[ \t]+String)?[\t ]*(={1,3}|:)[\t ]*(?:["']|b["'])? + end: | + (\z|[\r\n'"]) + additional_match: + - '[0-9]' + - '[A-Z]' + - '[a-z]' + - '^.{12,}$' + test: + data: password="AAAAAAAAAAAa00==" + start_offset: 10 + end_offset: 26 + comments: + - "The Base64 must contain numbers, upper case and lower case and be at least 12 characters long" + - "`password`, `secret`, `key`, or password like prefix (fuzzy)" + - "Delimiters like `=` or `:` (with padding)" + - "This matches _- instead of +/, for URI-safe Base64" + + - name: UUIDs type: uuids regex: @@ -63,12 +166,10 @@ patterns: - ^00000000-0000-0000-0000-000000000000$ - ^(?i)00010203-0405-0607-0809-0a0b0c0d0e0f$ - ^(?i)12345678-1234-1234-1234-123456789abc$ - test: - data: 10203040-5060-7080-90a0-b0c0d0e0f000 start_offset: 0 end_offset: -1 - expected: - name: uuids.txt start_offset: 12 @@ -80,22 +181,22 @@ patterns: start_offset: 131 end_offset: 167 + - name: Bearer Tokens type: bearer_tokens regex: version: 0.2 pattern: | - [a-zA-Z0-9_.=/+:-]+ + [a-zA-Z0-9_.=/+:-]{12,} start: | - \b([Bb]earer|[Tt]oken)[ ]+ + (Authorization: |['"])([Bb]earer |[Tt]oken (token=)?) end: | \z|[\s'"] additional_not_match: - - ^(?:letmein|Oracle|SuperSecretString|foo|ababbdbbebbbebdbbe5538003023|XYZ_INVALID_ACCESTOKEN_XYZ|QQ==|Shizuku|mF_9.B5f-4.1JqM|h480djs93hd8|SlAV32hkKG)$ + - ^(?:letmein|Oracle|SuperSecretString|foo|ababbdbbebbbebdbbe5538003023|XYZ_INVALID_ACCESTOKEN_XYZ|QQ==|Shizuku|mF_9.B5f-4.1JqM|h480djs93hd8|SlAV32hkKG|YmVlcDpib29w)$ - ^(?i)(?:dummy|fake|bearer|auth|invalid|your|my|the|undefined|github|oidc|database)(?:_api)?(?:_?token|key|secret)?$ - ^(?i)(?:[a-z0-9]|XYZ|ABC|123|.*_token)$ - - (?i)x{5} - - ^(?i)(x+|y+|z+|a+|\.+|.*\.\.\.)$ + - (^(?i)(x+|y+|z+|a+|\.+|.*\.\.\.)$|(?i)x{5}) expected: - name: bearer.txt start_offset: 45 @@ -108,8 +209,10 @@ patterns: start_offset: 23 end_offset: 42 comments: - - "As used in an Authorization header" - - "We try to remove common placeholders" + - As used in an Authorization header + - We try to remove common placeholders + - Lower length limit of 12 to remove common false positives on "Token ", since most words are below 12 characters in length + - name: OAuth client secret and ID pair type: oauth_client_secret diff --git a/pii/README.md b/pii/README.md index df6363f7..ce0c639c 100644 --- a/pii/README.md +++ b/pii/README.md @@ -178,4 +178,59 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc ^(?:MG4600005030071289421016045|ML13ML0160120102600100668497|MZ59000301080016367102371|NE58NE0380100100130305000268|SN08SN0100152000048500003035|TD8960002000010271091600153|TG53TG0090604310346500400070|KM4600005000010010904400137|HN54PISA00000000000000123124|NI92BAMC000000000000000003123123|MN580050099123456789)$ ``` + + +## Norwegian national identity number/D number + + + +_version: v0.1_ + +**Comments / Notes:** + + +- With no validation of the checksum this can cause a lot of false positives + +- The example test data does not have a valid checksum - it is one of the examples used with one digit in the checksum changed + +- You can test using the correct checksum, but it is used as a NOT match here to prevent false positives on other test data + + +
+Pattern Format + +```regex +(([04][1-9]|[15][0-9]|[26][0-9])(0[1-9]|1[0-2])|[37]0(0[469]|11)|[37][01](0[13578]|1[02]))[0-9]{2} ?[0-9]{3} ?[0-9]{2} +``` + +
+ +
+Start Pattern + +```regex +\A|[^0-9A-Za-z_.+/\\-] +``` + +
+End Pattern + +```regex +\z|[^0-9A-Za-z_.+/\\=-] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + +- Not Match: + + ```regex + 1111111111[123]|11112222333|01123456978|410185 ?123 ?45|220676 ?123 ?45|01010202010|01010101023 + ``` +
\ No newline at end of file diff --git a/pii/__snapshots__/credit_card_discovery.csv b/pii/__snapshots__/credit_card_discovery.csv index 3f703dee..384a50bb 100644 --- a/pii/__snapshots__/credit_card_discovery.csv +++ b/pii/__snapshots__/credit_card_discovery.csv @@ -1,3 +1 @@ secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column -"credit_card_discovery","Credit Card (Discovery)","19ff47cc8024c133d5845d3f8938caca289929031e7d508c3adf7adff177f0c2","pii/credit-cards.txt","46","46","1","17", -"credit_card_discovery","Credit Card (Discovery)","d8086d483c15c711ebba19f966b97d3c2adcba74025ff8d7e07c3698c9531deb","pii/credit-cards.txt","47","47","1","17", diff --git a/pii/__snapshots__/credit_card_mastercard.csv b/pii/__snapshots__/credit_card_mastercard.csv index 18501dd1..384a50bb 100644 --- a/pii/__snapshots__/credit_card_mastercard.csv +++ b/pii/__snapshots__/credit_card_mastercard.csv @@ -1,6 +1 @@ secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column -"credit_card_mastercard","Credit Card (MasterCard)","3aebd9be84d81a1c5676e31adda86653aa5e7da0a6bda5ea02fec6526c19bb11","pii/credit-cards.txt","14","14","1","17", -"credit_card_mastercard","Credit Card (MasterCard)","334b48feec00e7ec9808e50f1f05efd5eb1089abce7430e3a5b2b79dd2da1b73","pii/credit-cards.txt","13","13","1","17", -"credit_card_mastercard","Credit Card (MasterCard)","be6a3050f2611250d0b50c1fa6aff7d23925922a35520c2d0a51a262ba894f8e","pii/credit-cards.txt","12","12","1","17", -"credit_card_mastercard","Credit Card (MasterCard)","304945e91de3deff52a61d08733141d72dd42ec9d47972f1060534d54c0c7f90","pii/credit-cards.txt","11","11","1","17", -"credit_card_mastercard","Credit Card (MasterCard)","2f725bbd1f405a1ed0336abaf85ddfeb6902a9984a76fd877c3b5cc3b5085a82","pii/credit-cards.txt","10","10","1","17", diff --git a/pii/__snapshots__/credit_card_visa.csv b/pii/__snapshots__/credit_card_visa.csv index 8b2d9414..384a50bb 100644 --- a/pii/__snapshots__/credit_card_visa.csv +++ b/pii/__snapshots__/credit_card_visa.csv @@ -1,4 +1 @@ secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column -"credit_card_visa","Credit Card (Visa)","253da55d8a05e361b2e41c28728129909ed9a235632f2c9100ea5e827e396db7","pii/credit-cards.txt","21","21","1","14", -"credit_card_visa","Credit Card (Visa)","dd13cdf9af9dd3baf46ce96aecd7163cabf381ccb21e63f15f0fa10b1c663fa9","pii/credit-cards.txt","20","20","1","17", -"credit_card_visa","Credit Card (Visa)","9bbef19476623ca56c17da75fd57734dbf82530686043a6e491c6d71befe8f6e","pii/credit-cards.txt","19","19","1","17", diff --git a/pii/__snapshots__/no_national_id_number.csv b/pii/__snapshots__/no_national_id_number.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/pii/__snapshots__/no_national_id_number.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/pii/patterns.yml b/pii/patterns.yml index a31642f0..f0924045 100644 --- a/pii/patterns.yml +++ b/pii/patterns.yml @@ -237,3 +237,23 @@ patterns: - Examples include YouTube playlist IDs and Italian tax codes - If this is a problem, try the individual country IBAN patterns, based on the published IBAN structures on https://iban.com/structure + - name: Norwegian national identity number/D number + type: no_national_id_number + regex: + pattern: | + (([04][1-9]|[15][0-9]|[26][0-9])(0[1-9]|1[0-2])|[37]0(0[469]|11)|[37][01](0[13578]|1[02]))[0-9]{2} ?[0-9]{3} ?[0-9]{2} + start: | + \A|[^0-9A-Za-z_.+/\\-] + end: | + \z|[^0-9A-Za-z_.+/\\=-] + additional_not_match: + - 1111111111[123]|11112222333|01123456978|410185 ?123 ?45|220676 ?123 ?45|01010202010|01010101023 + test: + data: | + 41018512346 + start_offset: 0 + end_offset: 11 + comments: + - With no validation of the checksum this can cause a lot of false positives + - The example test data does not have a valid checksum - it is one of the examples used with one digit in the checksum changed + - You can test using the correct checksum, but it is used as a NOT match here to prevent false positives on other test data diff --git a/update_markdown.sh b/update_markdown.sh new file mode 100755 index 00000000..b94659cf --- /dev/null +++ b/update_markdown.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +if [ -z "$SECRET_SCANNING_TOOLS_PATH" ]; then + export SECRET_SCANNING_TOOLS_PATH="${HOME}"/secret-scanning-tools + echo "Defaulting to SECRET_SCANNING_TOOLS_PATH=${SECRET_SCANNING_TOOLS_PATH}" +fi + +CUSTOM_PATTERNS_PATH=$PWD "${SECRET_SCANNING_TOOLS_PATH}"/examples/update_custom_patterns_readme.sh diff --git a/vendors/README.md b/vendors/README.md index b44c3c63..eed07b4c 100644 --- a/vendors/README.md +++ b/vendors/README.md @@ -98,7 +98,7 @@ _version: v0.1_ Start Pattern ```regex -(?:\A|[\r\n])(?:\[auth\][^[]*\ntoken\s*=|(?:export )?SENTRY_AUTH_TOKEN\s*=|sentry-cli [^\r\n]*--auth-token |auth\.token\s*=)\s*['"`]? +(?:(?:\A|[\r\n])\[auth\][^[]*\ntoken\s*=|(?:\A|\b)SENTRY_AUTH_TOKEN\s*=|(?:\A|\b)sentry-cli [^\r\n]*--auth-token |(?:\A|\b)auth\.token\s*=)\s*['"`]? ```
@@ -146,7 +146,7 @@ _version: v0.1_ Start Pattern ```regex -(?:\A|[\r\n])(?:\[auth\][^[]*\napi_key\s*=|(?:export )?SENTRY_API_KEY\s*=|sentry-cli [^\r\n]*--api-key |auth\.api_key\s*=)\s*['"`]? +(?:(?:\A|[\r\n])\[auth\][^[]*\napi_key\s*=|(?:\A|\b)SENTRY_API_KEY\s*=|(?:\A|\b)sentry-cli [^\r\n]*--api-key |(?:\A|\b)auth\.api_key\s*=)\s*['"`]? ```
@@ -387,6 +387,45 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc
+## Okta API key (precise) + + + +_version: v0.1_ + +**Comments / Notes:** + + +- Uses surrounding context to reduce false positives + +- Either `SSWS ` then the token, or a variable starting `okta` followed by an assignment operator, then the token + + +
+Pattern Format + +```regex +0{2}[0-9A-Za-z_-]{40} +``` + +
+ +
+Start Pattern + +```regex +(\bSSWS\s{1,5}|(?i)okta[_-]?(api[_-]?)?(token|key|secret)\s{0,28}([:=]|[=-]>|to|[!=]={1,2}|<>)\s{0,28}['"`]?) +``` + +
+End Pattern + +```regex +\z|[^0-9A-Za-z_+/=-] +``` + +
+ ## DataDog API key @@ -564,7 +603,7 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc ## LaunchDarkly API key - +LaunchDarkly API or SDK key _version: v0.1_ @@ -573,7 +612,7 @@ _version: v0.1_ Pattern Format ```regex -api-[a-f0-9-]{8}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{12} +(api|sdk)-[a-f0-9-]{8}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{12} ``` @@ -600,7 +639,13 @@ api-[a-f0-9-]{8}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{12} _version: v0.1_ +**Comments / Notes:** + +- Looks for surrounding context to confirm this is a PagerDuty API key, not some other 20-byte alphanumeric string + +- The `Token token=` prefix is used in an Authorization header; it's possible that a different vendor could use a similar key and this same prefix, causing results that are a different vendor's key +
Pattern Format @@ -615,7 +660,7 @@ _version: v0.1_ Start Pattern ```regex -(\A|\b)(?i)pd_(service|api)_key['"`]?\s*([:=]|[?:]=|[=-]>|,)\s*['"`]? +(\A|\b)(?i)((pd|pagerduty)_(service|api)_key['"`]?\s*([:=]|[?:]=|[=-]>)\s*['"`]?|Token token=) ```
@@ -627,6 +672,32 @@ _version: v0.1_
+
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [A-Z] + ``` + +- Match: + + ```regex + [a-z] + ``` +- Not Match: + + ```regex + ^(pagerduty|pd)_(service|api)_ + ``` + +
+ ## Flickr OAuth token @@ -924,7 +995,7 @@ _version: v0.1_ -## MongoDB connection string +## UUIDv4 Bearer token (maybe Heroku) @@ -936,7 +1007,7 @@ _version: v0.1_ Pattern Format ```regex -mongodb\+srv://[^'"<>/:@\s\x00-\x08]+:[^'"<>/@\s\x00-\x08]+@[^/\s\x00-\x08]+\S* +[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12} ``` @@ -945,14 +1016,140 @@ mongodb\+srv://[^'"<>/:@\s\x00-\x08]+:[^'"<>/@\s\x00-\x08]+@[^/\s\x00-\x08]+\S* Start Pattern ```regex -\A|\b +(\b|\A)(?i)Bearer[ ] ```
End Pattern ```regex -\z|\s|['"`] +\b|\z +``` + +
+ +## Azure client secret + + + +_version: v0.1_ + + + +
+Pattern Format + +```regex +[a-zA-Z0-9~_.-]{34} +``` + +
+ +
+Start Pattern + +```regex +(?i)(client|azure[a-z_.-]{0,10})(key|secret|password|pwd|token)[a-z_.-]{0,9}\b['"`]?(\s{0,5}[\]\)])?\s{0,3}([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s{0,5}([[{])?['"`]? +``` + +
+End Pattern + +```regex +\z|["'`]|\s +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + [A-Z][a-z]|[A-Z][a-z] + ``` + +- Match: + + ```regex + [0-9][A-Za-z]|[A-Za-z][0-9] + ``` + +- Match: + + ```regex + [.~_-][A-Za-z0-9]|[A-Za-z0-9][.~_-] + ``` + +
+ +## Google private key id (or older API key) + + + +_version: v0.1_ + + + +
+Pattern Format + +```regex +[a-fA-F0-9]{40} +``` + +
+ +
+Start Pattern + +```regex +(?i)(private_key_id|google_api_key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +\b|\z +``` + +
+ +## OpenStack password/API key + + +OpenStack password or API key +_version: v0.1_ + + + +
+Pattern Format + +```regex +[^'",\r\n \t\x00-\x08]+ +``` + +
+ +
+Start Pattern + +```regex +(?i)OPEN_?STACK_(PASSWORD|API_?KEY)[_A-Z]*['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +['"\r\n,]|\z ```
@@ -966,30 +1163,68 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Not Match: ```regex - ^mongodb\+srv://(test-)?user:(test-)?pass(word)?@ + ^(ENV|[a-z_]+)\[$ ``` - Not Match: ```regex - ^mongodb\+srv://%s:%s@ + ^<%=.*%>$ ``` - Not Match: ```regex - ^mongodb\+srv://auser:apass@ + ^([a-z_]+\.api_?key|self\.[a-z_]+|os\.environ\.get\()$ ``` - Not Match: ```regex - ^mongodb\+srv://b\*b%40f3tt%3D:%244to%40L8%3DMC@test3.test.build.10gen.cc/mydb%3F\?replicaSet=repl0 + ^(\$\{?[A-Z]+\}?||\s+)$ + ``` +- Not Match: + + ```regex + ^(@?[a-z_]+\[:.*\]|@[a-z_]+)$ ``` -## UUIDv4 Bearer token (maybe Heroku) +## AlienVault OTX API key + + +AlienVault OTX API key +_version: v0.1_ + + + +
+Pattern Format + +```regex +[a-f0-9]{64}|[a-f0-9]{40} +``` + +
+ +
+Start Pattern + +```regex +(?i)ALIENVAULT(_?OTX)?(_?API)?_?KEY['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +['"`\r\n,]|\z +``` + +
+## Apollo.io API key +Apollo.io API key _version: v0.1_ @@ -998,7 +1233,7 @@ _version: v0.1_ Pattern Format ```regex -[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12} +service:[A-Za-z0-9-]+:[^\s'"`,\x00-\x08\x7f-\xff]+ ``` @@ -1007,7 +1242,40 @@ _version: v0.1_ Start Pattern ```regex -(\b|\A)(?i)Bearer[ ] +(?i)key['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +['"`,]|\z|\s +``` + +
+ +## ClickUp API key + + +ClickUp API key +_version: v0.1_ + + + +
+Pattern Format + +```regex +pk_[0-9]{6,8}_[A-Z0-9]{32} +``` + +
+ +
+Start Pattern + +```regex +\b|\A ```
@@ -1019,10 +1287,43 @@ _version: v0.1_
-## Azure client secret +## Amazon MWS Auth Token + + +Amazon MWS Auth Token +_version: v0.1_ +
+Pattern Format + +```regex +amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} +``` + +
+ +
+Start Pattern + +```regex +\b|\A +``` + +
+End Pattern + +```regex +\b|\z +``` + +
+ +## Jenkins API token + + +Jenkins API token _version: v0.1_ @@ -1031,7 +1332,7 @@ _version: v0.1_ Pattern Format ```regex -[a-zA-Z0-9~_.-]{34} +[a-f0-9]{32,64} ``` @@ -1040,14 +1341,88 @@ _version: v0.1_ Start Pattern ```regex -(?i)(client|azure[a-z_.-]{0,10})(key|secret|password|pwd|token)[a-z_.-]{0,9}\b['"`]?(\s{0,5}[\]\)])?\s{0,3}([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s{0,5}([[{])?['"`]? +(?i)jenkins_?(api[_-]?)?(token|secret|key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? ```
End Pattern ```regex -\z|["'`]|\s +['"`\r\n,]|\z +``` + +
+ +## AWS S3 presigned URL + + +AWS S3 presigned URL +_version: v0.1_ + + + +
+Pattern Format + +```regex +https://[a-z-]+\.s3\.amazonaws\.com/[^?\s'"`\r\n]+\?[^\s'"`\r\n]+&X-Amz-Signature=[^\s'"`\r\n]+ +``` + +
+ +
+Start Pattern + +```regex +\b|\A +``` + +
+End Pattern + +```regex +['"`\r\n,]|\z +``` + +
+ +## Azure Access Key (legacy format) + + +Azure Access Key in context in a variable assignment - legacy key format without internal identifiable features +_version: v0.1_ + +**Comments / Notes:** + + +- This is a legacy format for Azure Access Keys. The key is base64 encoded and encodes a fixed length key, so we know its length and that it always end in `==`. + +- The key lacks internal identifiable features, which are used in modern keys issued by these Azure services. + +- The use of `+` instead of `{86}` in the regex pattern is due to limitations of secret scanning - make sure you use the "additional match" to constrain the length + + +
+Pattern Format + +```regex +[A-Za-z0-9/+]+== +``` + +
+ +
+Start Pattern + +```regex +(\A|\b)(?i)(AZURE|ACCOUNT)(_?ACCESS|_?STORAGE(_?ACCOUNT)?)?_?KEY['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +['"`\r\n,]|\z ```
@@ -1062,19 +1437,117 @@ Add these additional matches to the [Secret Scanning Custom Pattern](https://doc - Match: ```regex - [A-Z][a-z]|[A-Z][a-z] + ^[A-Za-z0-9/+]{86}==$ + ``` + + + +## Azure Shared Access Signature (SAS) Token + + +Azure Shared Access Signature (SAS) Token +_version: v0.1_ + +**Comments / Notes:** + + +- This is a Shared Access Signature (SAS) token for Azure services. See [these examples](https://learn.microsoft.com/en-us/rest/api/storageservices/service-sas-examples) + +- The token is a URL query string parameter, and the signature is a base64 encoded HMAC-SHA256 hash, so is a fixed length in plain text and always ends in = + +- When encoded in a URL, the `+` character is replaced with `%2B`, the `/` character is replaced with `%2F`, and the `=` character is replaced with `%3D` + +- Because of the variable length of the characters (beacuse of the URL encoding), we use `{43,}` to match the signature + +- We ignore `https://files.oaiusercontent.com/` because they are URLs for images generated by ChatGPT + + +
+Pattern Format + +```regex +(https://[^?]+\?)?[^\s?/]*\bsig=([A-Za-z0-9]|%2[bfBF]){43,}%3[dD][^\s?/]* +``` + +
+ +
+Start Pattern + +```regex +\b|\A +``` + +
+End Pattern + +```regex +\z|\s|['"`] +``` + +
+ +
+Additional Matches + +Add these additional matches to the [Secret Scanning Custom Pattern](https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/defining-custom-patterns-for-secret-scanning#example-of-a-custom-pattern-specified-using-additional-requirements). + + + +- Match: + + ```regex + (^|&)sv=[0-9]{4}-[0-9]{2}-[0-9]{2} ``` - Match: ```regex - [0-9][A-Za-z]|[A-Za-z][0-9] + (^|&)se=[0-9]{4}-[0-9]{2}-[0-9]{2} ``` - Match: ```regex - [.~_-][A-Za-z0-9]|[A-Za-z0-9][.~_-] + (^|&)st=[0-9]{4}-[0-9]{2}-[0-9]{2} ``` +- Not Match: + + ```regex + ^https://files\.oaiusercontent\.com/ + ``` + +
+ +## CircleCI API token + + +CircleCI API token +_version: v0.1_ + + + +
+Pattern Format + +```regex +[a-f0-9]{40} +``` + +
+ +
+Start Pattern + +```regex +(?i)circle[_-]?(ci[_.-]?)?(api[_.-]?)?(token|key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? +``` + +
+End Pattern + +```regex +['"`\r\n,]|\z +```
\ No newline at end of file diff --git a/vendors/__snapshots__/alienvault_otx_api_key.csv b/vendors/__snapshots__/alienvault_otx_api_key.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/alienvault_otx_api_key.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/amazon_mws_auth_token.csv b/vendors/__snapshots__/amazon_mws_auth_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/amazon_mws_auth_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/apollo_io_api_key.csv b/vendors/__snapshots__/apollo_io_api_key.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/apollo_io_api_key.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/aws_s3_presigned_url.csv b/vendors/__snapshots__/aws_s3_presigned_url.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/aws_s3_presigned_url.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/azure_access_key_legacy.csv b/vendors/__snapshots__/azure_access_key_legacy.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/azure_access_key_legacy.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/azure_sas_token.csv b/vendors/__snapshots__/azure_sas_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/azure_sas_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/circleci_api_token.csv b/vendors/__snapshots__/circleci_api_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/circleci_api_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/clickup_api_key.csv b/vendors/__snapshots__/clickup_api_key.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/clickup_api_key.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/google_private_key_id_or_old_api_token.csv b/vendors/__snapshots__/google_private_key_id_or_old_api_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/google_private_key_id_or_old_api_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/jenkins_api_token.csv b/vendors/__snapshots__/jenkins_api_token.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/jenkins_api_token.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/okta_token_precise.csv b/vendors/__snapshots__/okta_token_precise.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/okta_token_precise.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/__snapshots__/openstack_password_or_key.csv b/vendors/__snapshots__/openstack_password_or_key.csv new file mode 100644 index 00000000..384a50bb --- /dev/null +++ b/vendors/__snapshots__/openstack_password_or_key.csv @@ -0,0 +1 @@ +secret_type,secret_type_display_name,secret,path,start_line,end_line,start_column,end_column diff --git a/vendors/patterns.yml b/vendors/patterns.yml index 0902683f..a6bc114a 100644 --- a/vendors/patterns.yml +++ b/vendors/patterns.yml @@ -58,7 +58,7 @@ patterns: pattern: | [a-fA-F0-9]{64} start: | - (?:\A|[\r\n])(?:\[auth\][^[]*\ntoken\s*=|(?:export )?SENTRY_AUTH_TOKEN\s*=|sentry-cli [^\r\n]*--auth-token |auth\.token\s*=)\s*['"`]? + (?:(?:\A|[\r\n])\[auth\][^[]*\ntoken\s*=|(?:\A|\b)SENTRY_AUTH_TOKEN\s*=|(?:\A|\b)sentry-cli [^\r\n]*--auth-token |(?:\A|\b)auth\.token\s*=)\s*['"`]? end: | \z|\s|['"`] additional_match: @@ -88,7 +88,7 @@ patterns: pattern: | [a-fA-F0-9]{32} start: | - (?:\A|[\r\n])(?:\[auth\][^[]*\napi_key\s*=|(?:export )?SENTRY_API_KEY\s*=|sentry-cli [^\r\n]*--api-key |auth\.api_key\s*=)\s*['"`]? + (?:(?:\A|[\r\n])\[auth\][^[]*\napi_key\s*=|(?:\A|\b)SENTRY_API_KEY\s*=|(?:\A|\b)sentry-cli [^\r\n]*--api-key |(?:\A|\b)auth\.api_key\s*=)\s*['"`]? end: | \z|\s|['"`] additional_match: @@ -227,10 +227,26 @@ patterns: - name: okta.txt start_offset: 180 end_offset: 222 - comments: - Okta token, starting with `00` and 40 random alphanumeric with _ and - + - name: Okta API key (precise) + type: okta_token_precise + regex: + version: 0.1 + pattern: | + 0{2}[0-9A-Za-z_-]{40} + start: | + (\bSSWS\s{1,5}|(?i)okta[_-]?(api[_-]?)?(token|key|secret)\s{0,28}([:=]|[=-]>|to|[!=]={1,2}|<>)\s{0,28}['"`]?) + end: | + \z|[^0-9A-Za-z_+/=-] + test: + data: SSWS 00RCsRGyrF9QzBe0rnWmw221UeclqRXFQGEJFdhAFM + start_offset: 5 + comments: + - Uses surrounding context to reduce false positives + - Either `SSWS ` then the token, or a variable starting `okta` followed by an assignment operator, then the token + - name: DataDog API key type: datadog_api_key regex: @@ -328,10 +344,11 @@ patterns: - name: LaunchDarkly API key type: launchdarkly_api_key + description: LaunchDarkly API or SDK key regex: version: 0.1 pattern: | - api-[a-f0-9-]{8}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{12} + (api|sdk)-[a-f0-9-]{8}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{4}-[a-f0-9-]{12} start: | \A|\b end: | @@ -346,12 +363,20 @@ patterns: pattern: | [A-Za-z0-9_-]{20} start: | - (\A|\b)(?i)pd_(service|api)_key['"`]?\s*([:=]|[?:]=|[=-]>|,)\s*['"`]? + (\A|\b)(?i)((pd|pagerduty)_(service|api)_key['"`]?\s*([:=]|[?:]=|[=-]>)\s*['"`]?|Token token=) end: | \z|\b + additional_match: + - '[A-Z]' + - '[a-z]' + additional_not_match: + - ^(pagerduty|pd)_(service|api)_ test: - data: pd_api_key=AAAAAAAAAAAAAAAAAAAA + data: pd_api_key=AAAAAAAAAAAAAAAAAAAa start_offset: 11 + comments: + - "Looks for surrounding context to confirm this is a PagerDuty API key, not some other 20-byte alphanumeric string" + - "The `Token token=` prefix is used in an Authorization header; it's possible that a different vendor could use a similar key and this same prefix, causing results that are a different vendor's key" - name: Flickr OAuth token type: flickr_oauth_token @@ -484,24 +509,6 @@ patterns: start_offset: 57 end_offset: 81 - - name: MongoDB connection string - type: mongodb_connection_string - regex: - version: 0.1 - pattern: | - mongodb\+srv://[^'"<>/:@\s\x00-\x08]+:[^'"<>/@\s\x00-\x08]+@[^/\s\x00-\x08]+\S* - start: | - \A|\b - end: | - \z|\s|['"`] - additional_not_match: - - ^mongodb\+srv://(test-)?user:(test-)?pass(word)?@ - - ^mongodb\+srv://%s:%s@ - - ^mongodb\+srv://auser:apass@ - - ^mongodb\+srv://b\*b%40f3tt%3D:%244to%40L8%3DMC@test3.test.build.10gen.cc/mydb%3F\?replicaSet=repl0 - test: - data: mongodb+srv://foo:bar@mongodb.example.invalid/ - - name: UUIDv4 Bearer token (maybe Heroku) type: uuidv4_bearer_token regex: @@ -534,3 +541,203 @@ patterns: data: ClientSecret = "kc3KP75a~edN1cV.rs~_r6F-O7-C-55-Ts" start_offset: 16 end_offset: 50 + + - name: Google private key id (or older API key) + type: google_private_key_id_or_old_api_token + regex: + version: 0.1 + pattern: | + [a-fA-F0-9]{40} + start: | + (?i)(private_key_id|google_api_key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + \b|\z + test: + data: | + google_api_key: 0123456789abcdef0123456789abcdef01234567 + start_offset: 16 + end_offset: 56 + + - name: OpenStack password/API key + type: openstack_password_or_key + description: OpenStack password or API key + regex: + pattern: | + [^'",\r\n \t\x00-\x08]+ + start: | + (?i)OPEN_?STACK_(PASSWORD|API_?KEY)[_A-Z]*['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"\r\n,]|\z + additional_not_match: + - ^(ENV|[a-z_]+)\[$ + - ^<%=.*%>$ + - ^([a-z_]+\.api_?key|self\.[a-z_]+|os\.environ\.get\()$ + - ^(\$\{?[A-Z]+\}?||\s+)$ + - ^(@?[a-z_]+\[:.*\]|@[a-z_]+)$ + test: + data: | + OPENSTACK_PASSWORD: "password1234isnotagoodpassword" + start_offset: 21 + end_offset: 51 + + - name: AlienVault OTX API key + type: alienvault_otx_api_key + description: AlienVault OTX API key + regex: + pattern: | + [a-f0-9]{64}|[a-f0-9]{40} + start: | + (?i)ALIENVAULT(_?OTX)?(_?API)?_?KEY['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"`\r\n,]|\z + test: + data: | + ALIENVAULT_OTX_API_KEY: "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" + start_offset: 25 + end_offset: 89 + + - name: Apollo.io API key + type: apollo_io_api_key + description: Apollo.io API key + regex: + pattern: | + service:[A-Za-z0-9-]+:[^\s'"`,\x00-\x08\x7f-\xff]+ + start: | + (?i)key['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"`,]|\z|\s + test: + data: | + key: service:my-apollo-service:aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa + start_offset: 5 + end_offset: 67 + + - name: ClickUp API key + type: clickup_api_key + description: ClickUp API key + regex: + pattern: | + pk_[0-9]{6,8}_[A-Z0-9]{32} + start: | + \b|\A + end: | + \b|\z + test: + data: | + CLICKUP_API_KEY: "pk_1234567_ABCDEFGHIJKLMNOPQRSTUVWXYZ123456" + start_offset: 18 + end_offset: 61 + + - name: Amazon MWS Auth Token + type: amazon_mws_auth_token + description: Amazon MWS Auth Token + regex: + pattern: | + amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} + start: | + \b|\A + end: | + \b|\z + test: + data: | + amzn.mws.auth.token: "amzn.mws.12345678-1234-1234-1234-1234567890ab" + start_offset: 22 + end_offset: 67 + + - name: Jenkins API token + type: jenkins_api_token + description: Jenkins API token + regex: + pattern: | + [a-f0-9]{32,64} + start: | + (?i)jenkins_?(api[_-]?)?(token|secret|key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"`\r\n,]|\z + test: + data: | + jenkins_api_token="0123456789abcdef0123456789abcdef" + start_offset: 19 + end_offset: 51 + + - name: AWS S3 presigned URL + type: aws_s3_presigned_url + description: AWS S3 presigned URL + regex: + pattern: | + https://[a-z-]+\.s3\.amazonaws\.com/[^?\s'"`\r\n]+\?[^\s'"`\r\n]+&X-Amz-Signature=[^\s'"`\r\n]+ + start: | + \b|\A + end: | + ['"`\r\n,]|\z + test: + data: | + https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/000/596/original/Rules_Writers_Guide_to_Snort_3_Rules.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIXACIED2SPMSC7GA%2F20211011%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211011T165700Z&X-Amz-Expires=172800&X-Amz-SignedHeaders=host&X-Amz-Signature=1d5134d3a84db0f7b1ece95fcf98387a0cc32ff131b6c95e6a80d3cd4b616a5c + start_offset: 0 + end_offset: 402 + + - name: Azure Access Key (legacy format) + type: azure_access_key_legacy + description: Azure Access Key in context in a variable assignment - legacy key format without internal identifiable features + regex: + pattern: | + [A-Za-z0-9/+]+== + start: | + (\A|\b)(?i)(AZURE|ACCOUNT)(_?ACCESS|_?STORAGE(_?ACCOUNT)?)?_?KEY['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"`\r\n,]|\z + additional_match: + - ^[A-Za-z0-9/+]{86}==$ + test: + data: | + AZURE_ACCESS_KEY: "abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn==" + start_offset: 19 + end_offset: 107 + comments: + - This is a legacy format for Azure Access Keys. The key is base64 encoded and encodes a fixed length key, so we know its length and that it always end in `==`. + - The key lacks internal identifiable features, which are used in modern keys issued by these Azure services. + - The use of `+` instead of `{86}` in the regex pattern is due to limitations of secret scanning - make sure you use the "additional match" to constrain the length + + - name: Azure Shared Access Signature (SAS) Token + type: azure_sas_token + description: Azure Shared Access Signature (SAS) Token + regex: + pattern: | + (https://[^?]+\?)?[^\s?/]*\bsig=([A-Za-z0-9]|%2[bfBF]){43,}%3[dD][^\s?/]* + start: | + \b|\A + end: | + \z|\s|['"`] + additional_match: + - '(^|&)sv=[0-9]{4}-[0-9]{2}-[0-9]{2}' + - '(^|&)se=[0-9]{4}-[0-9]{2}-[0-9]{2}' + - '(^|&)st=[0-9]{4}-[0-9]{2}-[0-9]{2}' + additional_not_match: + - '^https://files\.oaiusercontent\.com/' + test: + data: | + AZURE_SAS_TOKEN: "?sv=2019-12-12&ss=bfqt&srt=sco&sp=rwdlacupx&se=2021-10-11T16:57:00Z&st=2021-10-11T08:57:00Z&spr=https&sig=abcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn%3D" + start_offset: 19 + end_offset: 214 + comments: + - This is a Shared Access Signature (SAS) token for Azure services. See [these examples](https://learn.microsoft.com/en-us/rest/api/storageservices/service-sas-examples) + - The token is a URL query string parameter, and the signature is a base64 encoded HMAC-SHA256 hash, so is a fixed length in plain text and always ends in = + - When encoded in a URL, the `+` character is replaced with `%2B`, the `/` character is replaced with `%2F`, and the `=` character is replaced with `%3D` + - Because of the variable length of the characters (beacuse of the URL encoding), we use `{43,}` to match the signature + - We ignore `https://files.oaiusercontent.com/` because they are URLs for images generated by ChatGPT + + - name: CircleCI API token + type: circleci_api_token + description: CircleCI API token + regex: + pattern: | + [a-f0-9]{40} + start: | + (?i)circle[_-]?(ci[_.-]?)?(api[_.-]?)?(token|key)['"`]?(\s*[\]\)])?\s*([:,=]|[=-]>|to|[!=]={1,2}|<>)?\s*([[{])?['"`]? + end: | + ['"`\r\n,]|\z + test: + data: | + circle_ci_api_token: "0123456789abcdef0123456789abcdef01234567" + start_offset: 22 + end_offset: 62