Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/master' into pr/binyamin2/183
Browse files Browse the repository at this point in the history
  • Loading branch information
Baruch Odem committed Sep 28, 2023
2 parents 6848741 + 45e3309 commit 0bbecf6
Show file tree
Hide file tree
Showing 30 changed files with 1,333 additions and 513 deletions.
19 changes: 15 additions & 4 deletions .2ms.yml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,18 @@
log-level: info
ignore-result:
- c5748512948b492f5c07849ae2e69e7e831d36d3
- 9c6853ebe9b5e20774224ba6e5ea739191330e53
- 4d358e6dd9e2f21c647dad571f13dde3fd77d107
- 4f1d13e1bbebef31175ffe9a8d752609b9edc174
- 65706aeda7939dca8035f4b0a3446babffc7fcef
- d696fe501f3860f76cf768c7ebbccc416db6e4d2
- d766d69fed184582fc0cba1515f9beef7901e7a1
- deddd58b4aa4999419d6b9046dffa9fffdfd4860
- e7bf294c124122a6cf919edbffa40bf6572927b6
- 0b217706e100e9a05bbaa8427070d181bb2e2465
- 1e68cf841873862527a00ee5ef8d7957e319b6bc
- 0b217706e100e9a05bbaa8427070d181bb2e2465
- 4d358e6dd9e2f21c647dad571f13dde3fd77d107
- 7c73d41f23ba8e59a1c8d744594dbb54f87197b4
- 8f3e5dec63edc317daa4bff154939555cd35b2c3
- 9c6853ebe9b5e20774224ba6e5ea739191330e53
- 59f8916ff79257c8f86207d6e89767cc8e156814
- a3a83b7224e7e98e3cca6bd2cd138dbca831e06d
- ba1f0517b77a5b451d1d55078218cd23d96b686e
- c5748512948b492f5c07849ae2e69e7e831d36d3
2 changes: 1 addition & 1 deletion .ci/check_new_rules.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ func fetchGitleaksRules(version string) ([]byte, error) {
}

func fetchOurRules() ([]byte, error) {
content, err := os.ReadFile("secrets/secrets.go")
content, err := os.ReadFile("secrets/rules/rules.go")
if err != nil {
return nil, fmt.Errorf("failed to read our file content: %w", err)
}
Expand Down
4 changes: 3 additions & 1 deletion .ci/update-readme.sh
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
help_message=$(go run .)

echo '```' >output.txt
echo "" >output.txt
echo '```' >>output.txt
echo "$help_message" >>output.txt
echo '```' >>output.txt
echo "" >>output.txt

sed -i '/<!-- command-line:start -->/,/<!-- command-line:end -->/{
/<!-- command-line:start -->/{
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pr-validation.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
run: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.52.0 golangci-lint run -v -E gofmt --timeout=5m --out-format github-actions

- name: Run 2ms Scan
run: docker run -v $(pwd):/repo checkmarx/2ms:2.8.1 git /repo --config /repo/.2ms.yml
run: go run . git . --config .2ms.yml

- name: Go Test
run: go test -v ./...
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/release.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Release
on:
workflow_dispatch:
push:
branches: [ master ]
branches: [master]

jobs:
test:
Expand All @@ -26,9 +26,6 @@ jobs:
- name: Go Linter
run: docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.52.0 golangci-lint run -v -E gofmt --timeout=5m

- name: Run 2ms Scan
run: docker run -v $(pwd):/repo checkmarx/2ms:2.8.1 git /repo --config /repo/.2ms.yml

- name: Unit Tests
run: go test ./...

Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/gosec.yml → .github/workflows/security.yml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: gosec
name: Security Scans

on:
push:
Expand All @@ -22,8 +22,18 @@ jobs:
- name: Run Gosec Security Scanner
uses: securego/gosec@master
with:
args: "-no-fail -fmt sarif -out results.sarif ./..."
args: "-no-fail -fmt sarif -out results.sarif -exclude-dir=.ci -exclude-dir=tests ./..."
- name: Upload Gosec Results
uses: github/codeql-action/upload-sarif@v2
with:
sarif_file: results.sarif

secret-scanning:
runs-on: ubuntu-latest
steps:
- name: Checkout Source
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Run 2ms Scan
run: docker run -v $(pwd):/repo checkmarx/2ms:latest git /repo --config /repo/.2ms.yml
14 changes: 14 additions & 0 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Welcome to the 2ms club!

> [!NOTE]
> This is the first version of the document, we will rewrite it on the fly.
## Test

Along with the regular unit tests, we also have a set of other tests:

- `tests/cli` - e2e tests that build the CLI, run it, and check the output.
To skip these tests, run `go test -short ./...`.
- `tests/lint` - linter, to verify we are not using our forbidden functions (for example, using `fmt.Print` instead of `log.Info`)
- `.ci/check_new_rules.go` - compares the list of rules in the [latest _gitleaks_ release](https://github.com/gitleaks/gitleaks/releases/latest) with our list of rules, and fails if there are rules in the release that are not in our list.
- `.ci/update-readme.sh` - auto update the `help` message in the [README.md](README.md#command-line-interface) file.
110 changes: 67 additions & 43 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ If you wish to compile the project from its source use the following commands
```bash
git clone https://github.com/checkmarx/2ms.git
cd 2ms
go build -o dist/2ms main.go
go build -o dist/2ms main.go
./dist/2ms
```

Expand All @@ -49,7 +49,7 @@ go build -o dist/2ms main.go
We publish container image releases of `2ms` to [checkmarx/2ms](https://hub.docker.com/r/checkmarx/2ms) . To run `2ms` from a docker container use the following command:

```
docker run checkmarx/2ms
docker run checkmarx/2ms
```

You may also mount a local directory with the `-v <local-dir-path>:<container-dir-path>` argument. For instance:
Expand All @@ -71,7 +71,7 @@ on:
pull_request:
workflow_dispatch:
push:
branches: [ main ]
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
Expand All @@ -85,7 +85,7 @@ jobs:
# ...

- name: Run 2ms Scan
run: docker run -v $(pwd):/repo checkmarx/2ms:2.8.1 git /repo
run: docker run -v $(pwd):/repo checkmarx/2ms:2.8.1 git /repo
```
- In this example we've pinned the version to `2.8.1`. Make sure to check out if there's a newer version
Expand All @@ -96,6 +96,7 @@ jobs:
We've built `2ms` command line interface to be as self-descriptive as possible. This is the help message that you will see if you executed `2ms` without args:

<!-- command-line:start -->

```
2ms Secrets Detection: A tool to detect secrets in public websites and communication services.

Expand All @@ -116,30 +117,54 @@ Additional Commands:
rules List all rules

Flags:
--config string config file path
-h, --help help for 2ms
--ignore-result strings ignore specific result by id
--ignore-rule strings ignore rules by name or tag
--log-level string log level (trace, debug, info, warn, error, fatal) (default "info")
--regex stringArray custom regexes to apply to the scan, must be valid Go regex
--report-path strings path to generate report files. The output format will be determined by the file extension (.json, .yaml, .sarif)
--rule strings select rules by name or tag to apply to this scan
--stdout-format string stdout output format, available formats are: json, yaml, sarif (default "yaml")
-v, --version version for 2ms
--add-special-rule strings special (non-default) rules to apply.
This list is not affected by the --rule and --ignore-rule flags.
--config string config file path
-h, --help help for 2ms
--ignore-on-exit ignoreOnExit defines which kind of non-zero exits code should be ignored
accepts: all, results, errors, none
example: if 'results' is set, only engine errors will make 2ms exit code different from 0 (default none)
--ignore-result strings ignore specific result by id
--ignore-rule strings ignore rules by name or tag
--log-level string log level (trace, debug, info, warn, error, fatal) (default "info")
--max-target-megabytes int files larger than this will be skipped.
Omit or set to 0 to disable this check.
--regex stringArray custom regexes to apply to the scan, must be valid Go regex
--report-path strings path to generate report files. The output format will be determined by the file extension (.json, .yaml, .sarif)
--rule strings select rules by name or tag to apply to this scan
--stdout-format string stdout output format, available formats are: json, yaml, sarif (default "yaml")
-v, --version version for 2ms

Use "2ms [command] --help" for more information about a command.
```
<!-- command-line:end -->
## Special Rules
Special rules are rules that are not part of the default ruleset, usually because they are too noisy or too specific. You can use the `--add-special-rule` flag to add special rules by rule ID.
For example:
```
2ms git . --add-special-rule hardcoded-password
```
### List of Special Rules
| Rule ID | Description |
| -------------------- | -------------------------------------------------------------------------------------------------- |
| `hardcoded-password` | Detects strings that assigned to variables that contain the word `password`, `access`, `key`, etc. |
## Custom Regex Rules
You may specify one or more custom regex rules with the optional argument `--regex`. The value provided will be parsed as a regular expression and will be matched against the target items.
my-file.txt
```
password=1234567
username=admin
password=1234567
username=admin
```
```
Expand All @@ -161,12 +186,12 @@ scans a [Confluence](https://www.atlassian.com/software/confluence) instance
```
| Flag | Value | Default | Description |
|--------------|--------|--------------------------------|----------------------------------------------------------------------------------|
| `--url` | string | - | Confluence instance URL in the form of `https://<company id>.atlassian.net/wiki` |
| ------------ | ------ | ------------------------------ | -------------------------------------------------------------------------------- |
| `--url` | string | - | Confluence instance URL in the form of `https://<company id>.atlassian.net/wiki` |
| `--history` | - | not scanning history revisions | Scans pages history revisions |
| `--spaces` | string | all spaces | The names or IDs of the Confluence spaces to scan |
| `--token` | string | - | The Confluence API token for authentication |
| `--username` | string | - | Confluence user name or email for authentication |
| `--spaces` | string | all spaces | The names or IDs of the Confluence spaces to scan |
| `--token` | string | - | The Confluence API token for authentication |
| `--username` | string | - | Confluence user name or email for authentication |
For example:
Expand All @@ -183,36 +208,36 @@ For example:
Scans [Paligo](https://paligo.net/) content management system instance.
| Flag | Value | Default | Description |
|--------------|--------|---------------------------------|--------------------------------------------------|
| `--instance` | string | - | Instance name |
| `--token` | string | - | API token for authentication |
| `--username` | string | - | Confluence user name or email for authentication |
| ------------ | ------ | ------------------------------- | ------------------------------------------------ |
| `--instance` | string | - | Instance name |
| `--token` | string | - | API token for authentication |
| `--username` | string | - | Confluence user name or email for authentication |
| `--folder` | string | scanning all instance's folders | Folder ID |
| `--auth` | string | - | Base64 auth header encoded username:password |
| `--auth` | string | - | Base64 auth header encoded username:password |
### Discord
Scans [Discord](https://discord.com/) chat application history.
| Flag | Value | Default | Description |
|--------------------|----------|----------------------------------|--------------------------------------------------------------------------------------------------------|
| `--token` | string | - | Discord token |
| `--channel` | strings | all channels will be scanned | Discord channel IDs to scan |
| `--messages-count` | int | 0 = all messages will be scanned | Confluence user name or email for authentication |
| ------------------ | -------- | -------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `--token` | string | - | Discord token |
| `--channel` | strings | all channels will be scanned | Discord channel IDs to scan |
| `--messages-count` | int | 0 = all messages will be scanned | Confluence user name or email for authentication |
| `--duration` | duration | 14 days | The time interval to scan from the current time. For example, 24h for 24 hours or 336h0m0s for 14 days |
| `--server` | strings | - | Discord servers IDs to scan |
| `--server` | strings | - | Discord servers IDs to scan |
### Slack
Scans [Slack](https://slack.com/) chat application history.
| Flag | Value | Default | Description |
|--------------------|----------|----------------------------------|--------------------------------------------------------------------------------------------------------|
| `--token` | string | - | Slack token |
| `--channel` | strings | all channels will be scanned | Slack channel IDs to scan |
| `--messages-count` | int | 0 = all messages will be scanned | Confluence user name or email for authentication |
| ------------------ | -------- | -------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `--token` | string | - | Slack token |
| `--channel` | strings | all channels will be scanned | Slack channel IDs to scan |
| `--messages-count` | int | 0 = all messages will be scanned | Confluence user name or email for authentication |
| `--duration` | duration | 14 days | The time interval to scan from the current time. For example, 24h for 24 hours or 336h0m0s for 14 days |
| `--team` | string | - | Slack team name or ID |
| `--team` | string | - | Slack team name or ID |
### Git Repository
Expand All @@ -223,8 +248,8 @@ Scans a local git repository
```
| Flag | Value | Default | Description |
|------------------|-------|----------------------------------------|----------------------------------------------------------|
| `--all-branches` | - | false - only current checked in branch | scan all branches |
| ---------------- | ----- | -------------------------------------- | -------------------------------------------------------- |
| `--all-branches` | - | false - only current checked in branch | scan all branches |
| `--depth` | int | no limit | limit the number of historical commits to scan from HEAD |
For example
Expand All @@ -244,12 +269,11 @@ Scans a local repository
```
| Flag | Value | Default | Description |
|--------------------|---------|---------|--------------------------------------------------------|
| `--path` | string | - | Local directory path |
| `--project-name` | string | - | Project name to differentiate between filesystem scans |
| ------------------ | ------- | ------- | ------------------------------------------------------ |
| `--path` | string | - | Local directory path |
| `--project-name` | string | - | Project name to differentiate between filesystem scans |
| `--ignore-pattern` | strings | - | Patterns to ignore |
## Configuration File
You can pass `--config [path to config file]` argument to specify a configuration file. The configuration file format can be in YAML or JSON.
Expand Down Expand Up @@ -294,7 +318,7 @@ docker run -v $(pwd)/.2ms.yml:/app/.2ms.yml checkmarx/2ms confluence --url https

## Contributing

`2ms` is extendable with the concept of plugins. We designed it like this so anyone can easily contribute, improve and extend `2ms`
`2ms` is extendable with the concept of plugins. We designed it like this so anyone can easily contribute, improve and extend `2ms`. Read more about contributing in our [CONTRIBUTING.md](CONTRIBUTING.md) file.

## Contact

Expand Down
Loading

0 comments on commit 0bbecf6

Please sign in to comment.