Adding_Detectors_external.md

Secret Detectors

Secret Detectors have these two major functions:

1. Given some bytes, extract possible secrets, typically using a regex.
2. Validate the secrets against the target API, typically using a HTTP client.

The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.

Table of Contents

• Secret Detectors
  • Table of Contents
  • Getting Started
    • Sourcing Guidelines
    • Development Guidelines
    • Development Dependencies
    • Creating a new Secret Scanner
  • Addendum
    • Using a test secret file
    • Adding Protos in Windows

Getting Started

Sourcing Guidelines

We are interested in detectors for services that meet at least one of these criteria

• host data (they store any sort of data provided)
• have paid services (having a free or trial tier is okay though)

If you think that something should be included outside of these guidelines, please let us know.

Development Guidelines

• When reasonable, favor using the net/http library to make requests instead of bringing in another library.
• Use the common.SaneHttpClient for the http.Client whenever possible.

Development Dependencies

• Go 1.17+
• Make

Creating a new Secret Scanner

1. Identify the Secret Detector name from the /proto/detectors.proto DetectorType enum. If necessary, run make protos when adding new ones.

2. Generate the Secret Detector

   go run hack/generate/generate.go detector <DetectorType enum name>

3. Complete the secret detector.

   The previous step templated a boilerplate + some example code as a package in the pkg/detectors folder for you to work on. The secret detector can be completed with these general steps:

   1. Create a test secrets file, and export the variable
   2. Update the pattern regex and keywords. Try iterating with regex101.com.
   3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
   4. Update the tests with these test cases at minimum:
      1. Found and verified (using a credential loaded from GCP Secrets)
      2. Found and unverified
      3. Not found
      4. Any false positive cases that you come across
   5. Create a pull request for review.

Addendum

Using a test secret file

1. Create a file called .env with this env file format:

   SECRET_TYPE_ONE=value
   SECRET_TYPE_ONE_INACTIVE=v@lue

2. Export the TEST_SECRET_FILE variable, pointing to the env file:

   export TEST_SECRET_FILE=".env"

Now, the detector test should attempt to load the given env key from that file.

Adding Protos in Windows

1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
3. Open Ubuntu cli and install dos2unix.

   sudo apt install dos2unix

4. Identify the trufflehog local directory and convert scripts/gen_proto.sh file in Unix format.

   dos2unix ./scripts/gen_proto.sh

5. Open /proto/detectors.proto file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.

   make protos