Add documentation and set of guidelines for writing detectors.

PiperOrigin-RevId: 666316412
Change-Id: I63ac57b5700bab97025d79a384f855ebdde63c5a

docs/howto/common-patterns.md

@@ -0,0 +1,197 @@
+# Common detector patterns
+
+### Running only for a specific service
+
+*Use case: I want my detector to only run for web applications or for
+application X.*
+
+There exist currently two way in Tsunami to filter the service type:
+
+1. Using annotations (preferred)
+
+The
+[`@ForWebService`](https://github.com/google/tsunami-security-scanner/blob/master/plugin/src/main/java/com/google/tsunami/plugin/annotations/ForWebService.java)
+and
+[`@ForServiceName({"X", "Y"})`](https://github.com/google/tsunami-security-scanner/blob/master/plugin/src/main/java/com/google/tsunami/plugin/annotations/ForServiceName.java)
+annotations can be used to instruct the core engine of Tsunami to only run this
+plugin if the service was a web service or a service with name `X` or `Y`. The
+name of the service is obtained during the discovery phase. It currently is the
+exact same service name as NMAP would report (e.g. `http`, `https`, `ssh`).
+
+2. Using filtering (web service only)
+
+The
+[`NetworkServiceUtils.isWebService()`](https://github.com/google/tsunami-security-scanner/blob/483f9ea5b7c69e8802353e0dcd293c2f35eaa4aa/common/src/main/java/com/google/tsunami/common/data/NetworkServiceUtils.java#L69)
+can be used when performing filtering to ensure only `NetworkService` that were
+identified as web service will be processed.
+
+Example usage:
+
+```java
+someNetworkServiceCollection.stream()
+  .filter(NetworkServiceUtils::isWebService)
+  // {...}
+```
+
+### Building URLs
+
+*Use case: My detector targets a web application. How do I build the URL?*
+
+When writing your plugins, there are a few things that you should NOT have to
+care about and that the Tsunami core engine should resolve for you:
+
+- Is the service using HTTP or HTTPS?
+- How do I construct the URL from the `NetworkService`?
+- What if NMAP fails to identify the service as HTTP and I still want to build
+  an URL?
+
+All of these concerns are addressed in the core engine of Tsunami and you can
+simply use the URL building API:
+[`NetworkServiceUtils.buildWebApplicationRootUrl()`](https://github.com/google/tsunami-security-scanner/blob/483f9ea5b7c69e8802353e0dcd293c2f35eaa4aa/common/src/main/java/com/google/tsunami/common/data/NetworkServiceUtils.java#L173)
+
+#### DO
+
+```java
+String myUrl = NetworkServiceUtils.buildWebApplicationRootUrl(networkService)
+  + MY_VULNERABLE_ENDPOINT;
+```
+
+#### DO NOT
+
+The following **SHOULD NOT BE USED**:
+
+1. Defining a `buildTarget` intermediate function is redundant and most of the
+time not necessary:
+
+```java
+ private static StringBuilder buildTarget(NetworkService networkService) {
+    StringBuilder targetUrlBuilder = new StringBuilder();
+    if (NetworkServiceUtils.isWebService(networkService)) {
+      targetUrlBuilder.append(NetworkServiceUtils.buildWebApplicationRootUrl(networkService));
+    } else {
+      targetUrlBuilder
+          .append("http://")
+          .append(toUriAuthority(networkService.getNetworkEndpoint()))
+          .append("/");
+    }
+    targetUrlBuilder.append(MY_VULNERABLE_ENDPOINT);
+    return targetUrlBuilder;
+  }
+```
+
+2. Using `String.Format` does not make use of the information obtained during
+the discovery phase and is error prone:
+
+```java
+var uriAuthority = NetworkEndpointUtils.toUriAuthority(networkService.getNetworkEndpoint());
+var loginPageUrl = String.format("http://%s/%s", uriAuthority, MY_VULNERABLE_ENDPOINT);
+```
+
+### Adding command line arguments consumed by the detector
+
+*Use case: I need command line arguments for my detector*
+
+Tsunami uses [jCommander](https://jcommander.org/) for command line argument
+parsing. In order to add new CLI arguments for your plugin, first define the
+data class for holding all the arguments. You can follow the jCommander tutorial
+to learn more about this tool.
+
+For example:
+
+```java
+@Parameters(separators = "=")
+public final class MyPluginArgs implements CliOption {
+  @Parameter(names = "--param", description = "Description for param.")
+  public String param;
+
+  @Override
+  public void validate() {
+    // Validate the command line value.
+  }
+}
+```
+
+Then, the CLI flags will be parsed and an instance of this class will be created
+by the main scanner at start-up time. In order to use this class in your plugin,
+you can directly inject this data class into your plugin's constructor.
+
+```java
+public final class MyVulnDetector implements VulnDetector {
+  private final MyPluginArgs args;
+
+  @Inject
+  MyVulnDetector(MyPluginArgs args) {
+    this.args = checkNotNull(args);
+  }
+
+  // {...}
+}
+```
+
+### Adding configuration properties consumed by the detector
+
+*Use case: How do I add configurable option for my detector?*
+
+Tsunami supports loading configs from a YAML file and uses
+[snakeyaml](https://bitbucket.org/asomov/snakeyaml/wiki/Documentation) to parse
+the YAML config files. In order to add configuration properties to your plugin,
+first you need to define a data class for holding all the configuration values.
+
+NOTE: Currently Tsunami only supports standard Java data types for
+configurations like `String`, numbers (`int`, `long`, `float`, `double`, etc),
+`List` and `Map`.
+
+```java
+// All config classes must be annotated by this ConfigProperties annotation in
+// order for Tsunami to recognize the config class.
+@ConfigProperties(prefix = "my.plugin.configs")
+public class MyPluginConfigs {
+  String stringValue;
+  long longValue;
+  List<String> listValues;
+  Map<String, String> mapValues;
+}
+```
+
+Then, similar to the command line arguments, you can inject an instance of this
+data class into your plugin's constructor.
+
+```java
+public final class MyVulnDetector implements VulnDetector {
+  private final MyPluginConfigs configs;
+
+  @Inject
+  MyVulnDetector(MyPluginConfigs configs) {
+    this.configs = checkNotNull(configs);
+  }
+
+  // {...}
+}
+```
+
+The scanner will parse the configuration file when it starts, create an instance
+of the data class from the config data, and inject the instance into your
+plugin.
+
+Following is an example config file for the previously defined `MyPluginConfigs`
+object.
+
+```yaml
+my:
+  plugin:
+    configs:
+      # Config name can be exact match.
+      stringValue: "example value"
+      # Or matching via snake_case.
+      long_value: 123
+      list_values:
+      - "a"
+      - "b"
+      - "c"
+      mapValues:
+        key1: "value1"
+        key2: "value2"
+```
+
+To use the configuration file, you need to set the `tsunami.config.location`
+option when calling Tsunami.

docs/howto/index.md

@@ -1,4 +1,8 @@
 # Tsunami documentation
 
 - [How tsunami works]({{ site.baseurl }}/howto/orchestration)
-- [Howto]({{ site.baseurl }}/howto/howto)
+- Guides
+  * [How to build and run Tsumami]({{ site.baseurl }}/howto/howto)
+  * [How to write a detector]({{ site.baseurl }}/howto/new-detector)
+  * [Common detector patterns]({{ site.baseurl }}/howto/common-patterns)
+  (i.e. "How do I do that?!")

docs/howto/new-detector.md

@@ -0,0 +1,173 @@
+# Writing a Tsunami detector
+
+## Overview
+
+Each Tsunami detector needs the following pieces which we will create in this
+tutorial:
+
+*   A plugin name that is unique among all enabled Tsunami plugins.
+*   A set of build rules for [Gradle](https://gradle.org/) (external build)
+*   A `VulnDetector` that implements the vulnerability detection logic.
+*   A `PluginBootstrapModule` that provides necessary Guice bindings for the
+    detector.
+*   An optional `CliOption` that captures all the supported command line flags
+    for the detector.
+*   An optional `ConfigProperties` that captures all the supported configuration
+    for the detector.
+
+## 1. Fork the examples
+
+Tsunami provides a few example implementations of a `VulnDetector` plugin. The
+examples live in the
+[examples directory](https://github.com/google/tsunami-security-scanner-plugins/tree/master/examples)
+
+*   Update Java package names. The example `VulnDetector` plugin is defined
+    under `com.google.tsunami.plugins.example` package. Refactor the package and
+    class name according to your detector implementation.
+*   Give a meaningful description to the Gradle build rule at `build.gradle`. We
+    will work on the Gradle build rule later.
+*   Rewrite the `README.md` file to have a good explanation of your
+    `VulnDetector` plugin.
+
+## 2. Putting the detector together
+
+### 2.a - `PluginInfo` annotation
+
+All Tsunami plugins must be annotated by the `PluginInfo` annotation. Otherwise
+it cannot be identified by Tsunami scanner at runtime. This annotation provides
+the general information about the plugin to the core scanner.
+
+Following is an example usage of the `PluginInfo` annotation from our
+`WordPressInstallPageDetector`.
+
+```java
+@PluginInfo(
+    // VULN_DETECTION PluginType is required for a VulnDetector plugin.
+    type = PluginType.VULN_DETECTION,
+    // This gives a human readable name for your VulnDetector. Can be different
+    // from your class name.
+    name = "WordPressInstallPageDetector",
+    // The current version of your plugin.
+    version = "0.1",
+    // A detailed description about what this plugin does.
+    description =
+        "This detector checks whether a WordPress install is unfinished. An unfinished WordPress"
+            + " installation exposes the /wp-admin/install.php page, which allows attacker to set"
+            + " the admin password and possibly compromise the system.",
+    // The author of this plugin.
+    author = "Tsunami Team ([email protected])",
+    // The guice module that bootstraps this plugin.
+    bootstrapModule = WordPressInstallPageDetectorBootstrapModule.class)
+```
+
+### 2.b - Define the `VulnDetector` plugin
+
+Each vulnerability detector plugin is an implementation of the `VulnDetector`
+interface. For this step we only explain the placeholder code, later you'll need
+to provide implementations for the class itself.
+
+Following is an example placeholder code from the
+`WordPressInstallPageDetector`:
+
+```java
+// ...
+// annotations
+// ...
+public final class WordPressInstallPageDetector implements VulnDetector {
+  // See https://github.com/google/flogger for details.
+  private static final GoogleLogger logger = GoogleLogger.forEnclosingClass();
+
+  // Tsunami uses Guice (https://github.com/google/guice) to manage the
+  // dependencies.
+  @Inject
+  WordPressInstallPageDetector(
+    // Tsunami provides a UtcClock for production and FakeUtcClock for
+    // testing purposes.
+    @UtcClock Clock utcClock,
+    // You can also inject other useful dependencies to your plugin code, e.g.
+    // inject HttpClient if you need to interact with a web server.
+    HttpClient httpClient) {
+  }
+
+  // The entrypoint of the VulnDetector. We will explain this later.
+  @Override
+  public DetectionReportList detect(
+    TargetInfo targetInfo, ImmutableList<NetworkService> matchedServices) {
+    // implement me.
+  }
+}
+```
+
+### 2.c - Implement the main detection logic
+
+The main logic of the detection is expected to happen in the `detect` method,
+which expects two arguments:
+
+1.  [The `TargetInfo` proto](https://github.com/google/tsunami-security-scanner/blob/master/proto/reconnaissance.proto).
+   This proto contains information that were gathered during the fingerprinting
+   and discovery phase.
+1.  [The `NetworkService` list](https://github.com/google/tsunami-security-scanner/blob/master/proto/network_service.proto).
+    This list contains all the identified network services that match the
+    service filtering annotations.
+
+The main detection logic usually iterates over all the elements of the
+`matchedServices` parameter and checks whether the `NetworkService` is
+vulnerable to the vulnerability your plugin checks for. If any service is
+vulnerable, you'll need to build a `DetectionReport` proto that explains the
+identified vulnerability.
+
+Following is an example implementation from our `WordPressInstallPageDetector`:
+
+```java
+@Override
+public DetectionReportList detect(
+    TargetInfo targetInfo, ImmutableList<NetworkService> matchedServices) {
+  return DetectionReportList.newBuilder()
+      .addAllDetectionReports(
+          matchedServices.stream()
+              // The WordPressInstallPageDetector only works for web services.
+              .filter(NetworkServiceUtils::isWebService)
+              // Detection logic that checks whether a web service exposes
+              // a wordpress installation page. Omitted here for simplicity.
+              .filter(this::isServiceVulnerable)
+              // Build a DetectionReport when the web service is vulnerable.
+              .map(networkService -> buildDetectionReport(targetInfo, networkService))
+              .collect(toImmutableList()))
+      .build();
+}
+
+private DetectionReport buildDetectionReport(
+    TargetInfo scannedTarget, NetworkService vulnerableNetworkService) {
+  return DetectionReport.newBuilder()
+      .setTargetInfo(scannedTarget)
+      .setNetworkService(vulnerableNetworkService)
+      .setDetectionTimestamp(Timestamps.fromMillis(Instant.now(utcClock).toEpochMilli()))
+      .setDetectionStatus(DetectionStatus.VULNERABILITY_VERIFIED)
+      .setVulnerability(
+          Vulnerability.newBuilder()
+              .setMainId(
+                  VulnerabilityId.newBuilder()
+                      .setPublisher("GOOGLE")
+                      .setValue("UNFINISHED_WORD_PRESS_INSTALLATION"))
+              .setSeverity(Severity.CRITICAL)
+              .setTitle("Unfinished WordPress Installation")
+              .setDescription(
+                  "An unfinished WordPress installation exposes the /wp-admin/install.php page,"
+                      + " which allows attacker to set the admin password and possibly"
+                      + " compromise the system."))
+      .build();
+}
+```
+
+## 3. Preparing the `PluginBootstrapModule`
+
+Each Tsunami plugin must have a companion `PluginBootstrapModule` that provides
+the required Guice bindings and registers the plugin to the core engine.
+
+Creating a `PluginBootstrampModule` is rather simple if you only need to
+register the plugin. You only need to call `registerPlugin` within the
+`configurePlugin` method (e.g.
+[`ExampleVulnDetectorBootstrapModule`](https://github.com/google/tsunami-security-scanner-plugins/tree/master/examples/example_vuln_detector/src/main/java/com/google/tsunami/plugins/example/ExampleVulnDetectorBootstrapModule.java)).
+
+A more complete example is the
+[GenericWeakCredentialDetectorBootstrapModule](https://github.com/google/tsunami-security-scanner-plugins/blob/master/google/detectors/credentials/generic_weak_credential_detector/src/main/java/com/google/tsunami/plugins/detectors/credentials/genericweakcredentialdetector/GenericWeakCredentialDetectorBootstrapModule.java)