Inquiry Regarding Preferred Approach for Using Dependency Track on Multi-Versioned Applications

[ansonallard]

Hello,

I came across Dependency Track when I sought to replace a proprietary Software Component Analysis tool. However, I'm not sure if I am using the tool properly for my given use case, which I will outline below.

I maintain a CI system that creates a software binary for every commit developers push to our remote. Before a software binary can be created, our business requires us to perform license validation and vulnerability assessments to comply with corporate policies. I generate a CycloneDX SBOM for each commit that I then send to Dependency Track for analysis against the policies I've written in Dependency Track. Based on my experience using Dependency Track and since I require an analysis per commit, I cannot create a project per software repository, as I cannot get a policy violation report for a given SBOM upload; I can only get a policy violation report for a project. Therefore, I need to create a project/version for each commit, which leads to a large number of projects in the system (which is to be expected).

Often, during builds, SBOMs will not contain license information, which cause a policy violation per my user defined policies, or builds contain known 3rd party vulnerabilities that the organization has accepted and can be suppressed. As noted in other discussions (#2995), suppressing violations across multiple projects is not tenable manually. Fortunately, @nscuro provided a tool to address this problem, https://github.com/nscuro/dtapac, which I've integrated with Dependency Track.

I've experienced issues with the dtapac suppression workflow (depenendency track notification (event) -> dtapac -> opa -> dtapac -> dependency track), but before I reported an issue on either project, I wanted to understand if my use of Dependency Track aligns with how the tool was designed.

Is there an alternate or preferred approach for performing policy analysis on multi-versioned applications, where a policy violation report is required per application version?

Some open issues/questions I have with this workflow are:

1. When Dependency Track emits a notification, does it wait for the callback to respond with a 2XX status code before releasing the SBOM processing? If not, this introduces a race condition when using the dtapac system, where a SBOM analysis can complete before dtapac suppresses all applicable violations. In my CI workflow, I poll on the SBOM token endpoint until the analysis is complete, then fetch the violation report to make decisions.
2. Does Dependency Track have retry logic wrapping the callback functions?
3. When processing a SBOM, Dependency Track sends a notification when identifying new policy violations. However, when one reanalyzes the project or submits a new SBOM, it appears that Dependency Track does not re-send a notification for existing policy violations. Is this the intended behavior? The existing behavior had led to situations where projects have un-suppressed violations since the dtapac system is down or rejected the original notification.
4. Is the Dependency Track API multi-replica safe? My current running configuration has up to 4 CPU cores and 16 GB of memory, but I am seeing timeout errors from dtapac when calling dependency track under load, which horizontal scaling could help mitigate. (It appears the answer is no, but is an open issue: Refactor to support DT clusters for high availability (HA) and high performance #903)
5. Dependency Track's analysis of our projects is slow. A project that has ~500 components takes, on average, 250-300 seconds to complete, and upwards of 600 seconds (which is our maximum threshold for our CI system) when under load. It appears this issue has already been raised here: Dependency-Track takes over 30 minutes to process BOM and Jenkins plugin times out  #2857. Our project processing times do not appear to be dependent on the number of projects/components in the system, as a fresh install of Dependency Track took the same amount of time as well.

[ansonallard]

Update: Regarding point number 5 above - it seems the bulk of the SBOM processing time is tied up in policy analysis. At the moment I have 12 policies, with all but one having 1 policy condition.

[nscuro]

I'm going to answer in two parts, apologies for the wall of text.

---

1. When Dependency Track emits a notification, does it wait for the callback to respond with a 2XX status code before releasing the SBOM processing?

Notifications are entirely asynchronous, and processing does not wait for notifications to be acknowledged. Doing that would cripple the system quite fast, given:

• large BOMs can trigger multiple hundreds or thousands of notifications
• Notifications can have multiple recipients (i.e. Slack, email, MS Teams)
• Each notification dispatch adds latency, depending on how close the target system is etc.

If not, this introduces a race condition when using the dtapac system, where a SBOM analysis can complete before dtapac suppresses all applicable violations.

That is true, unfortunately. Ideally, there'd be a special "hook" in DT that would invoke OPA directly, even before a notification is sent.

2. Does Dependency Track have retry logic wrapping the callback functions?

There is currently no retry logic for notifications. Delivery semantics are "at most once", if you will.

3. [...] However, when one reanalyzes the project or submits a new SBOM, it appears that Dependency Track does not re-send a notification for existing policy violations. Is this the intended behavior?

That is indeed intended. Notifications are supposed to be sent when a policy violation or vulnerability has been newly identified. If they would be re-sent for every analysis, it would become an unmanageable amount of noise.

4. Is the Dependency Track API multi-replica safe?

In its current state, it is not.

My current running configuration has up to 4 CPU cores and 16 GB of memory, but I am seeing timeout errors from dtapac when calling dependency track under load, which horizontal scaling could help mitigate.

I'd be very interested to see where the bottleneck is exactly. As in, is it the CPU being maxed-out, do you experience frequent GC pauses, or is it rather a database-related problem (high latency in communication, slow queries, too many queries, etc.). Are you monitoring the system, and would you be willing to share some insights?

---

I wanted to address your questions from where Dependency-Track v4.x is today. With that out of the way, let me do some advertising for something that's in the making.

We've been aware of the limitations you raised, and more. So we started building Hyades to address them. Small warning: The documentation is not where it should be yet. The original motivation for Hyades is documented here: https://github.com/DependencyTrack/hyades/blob/main/WTF.md

There are too many goodies in Hyades already to list them here. So I'll limit myself to what addresses your concerns:

1. We rebuilt the policy engine based on the Common Expression Language (CEL). It gives you more fine-grained control over the conditions you write, making it less likely that violations are raised that you don't even care about. For example, exclusions can be part of the policy condition itself. You can read more about it here: https://dependencytrack.github.io/hyades/latest/usage/policy-compliance/expressions/
2. What dtapac offers has been integrated directly into Dependency-Track. Instead of relying on an external OPA system, this feature re-uses CEL support from policies, and thus does not involve any network calls. Decisions are applied before notifications are sent. The initial design is documented here: Support global vulnerability analysis policies hyades#930
3. Notifications are sent to a durable message broker first, from where they are optionally picked up by a publisher. The publisher has retries built-in, which results in "at least once" semantics. Not only does publishing to a broker mean that notifications can be replayed, but you can also hook your own services directly into it, and avoid the pain of dealing with Webhooks entirely.
4. We achieved horizontal scalability of the API server, some details here: Achieve horizontal scalability of API server hyades#375
5. We put an increased focus on performance, especially for everything in the "hot paths". This includes BOM ingestion, and all the various processing steps (vulnerability analysis, policy evaluation etc.). You explicitly mention policy evaluation being slow. The new CEL-based policy engine is highly optimized and orders of magnitudes faster, despite being more flexible.

Hyades is not quite ready for GA yet. But it will eventually become Dependency-Track 5.x, and is already used in production by at least one large company.

[ansonallard]

Thank you for your extremely thorough response! We will be excited to integrate with Hyades when it becomes generally available.

Regarding the bottleneck, item 5, we do have some profiling and monitoring in play, so I will dig in to see what information I can share to help further diagnose the issue.

[ansonallard]

@nscuro I completed a SBOM anaylsis on our production system when no other clients were using the system (no load) and captured the following timestamps from the process:

[
  {
    "message": "2024-01-26 21:04:14,571 INFO [ProjectMetricsUpdateTask] Executing metrics update for project 7f671729-7f07-432e-91e5-baefadb6d6fa",
    "criTime": "2024-01-26T21:04:14.571234724Z",
    "timestamp": "1706303054571234724"
  },
  {
    "message": "2024-01-26 21:04:14,570 INFO [PolicyEngine] Policy analysis complete",
    "criTime": "2024-01-26T21:04:14.571105734Z",
    "timestamp": "1706303054571105734"
  },
  {
    "message": "2024-01-26 21:02:20,616 INFO [PolicyEngine] Evaluating 414 component(s) against applicable policies",
    "criTime": "2024-01-26T21:02:20.616175515Z",
    "timestamp": "1706302940616175515"
  },
  {
    "message": "2024-01-26 21:02:20,613 INFO [OssIndexAnalysisTask] Sonatype OSS Index analysis complete",
    "criTime": "2024-01-26T21:02:20.613627566Z",
    "timestamp": "1706302940613627566"
  },
  {
    "message": "2024-01-26 21:02:20,613 INFO [OssIndexAnalysisTask] Analyzing 2 component(s)",
    "criTime": "2024-01-26T21:02:20.613594215Z",
    "timestamp": "1706302940613594215"
  },
  {
    "message": "2024-01-26 21:02:03,625 INFO [OssIndexAnalysisTask] Starting Sonatype OSS Index analysis task",
    "criTime": "2024-01-26T21:02:03.626120851Z",
    "timestamp": "1706302923626120851"
  },
  {
    "message": "2024-01-26 21:02:03,623 WARN [OssIndexAnalysisTask] An API username or token has not been specified for use with OSS Index. Using anonymous access",
    "criTime": "2024-01-26T21:02:03.62385417Z",
    "timestamp": "1706302923623854170"
  },
  {
    "message": "2024-01-26 21:02:03,616 INFO [InternalAnalysisTask] Internal analysis complete",
    "criTime": "2024-01-26T21:02:03.617107351Z",
    "timestamp": "1706302923617107351"
  },
  {
    "message": "2024-01-26 21:02:01,423 INFO [RepositoryMetaAnalyzerTask] Completed component repository metadata analysis against 414 components",
    "criTime": "2024-01-26T21:02:01.423837096Z",
    "timestamp": "1706302921423837096"
  },
  {
    "message": "2024-01-26 21:01:59,677 INFO [InternalAnalysisTask] Analyzing 414 component(s)",
    "criTime": "2024-01-26T21:01:59.677852904Z",
    "timestamp": "1706302919677852904"
  },
  {
    "message": "2024-01-26 21:01:59,677 INFO [InternalAnalysisTask] Starting internal analysis task",
    "criTime": "2024-01-26T21:01:59.677802394Z",
    "timestamp": "1706302919677802394"
  },
  {
    "message": "2024-01-26 21:01:58,677 INFO [RepositoryMetaAnalyzerTask] Performing component repository metadata analysis against 414 components",
    "criTime": "2024-01-26T21:01:58.67783176Z",
    "timestamp": "1706302918677831760"
  },
  {
    "message": "2024-01-26 21:01:58,677 INFO [BomUploadProcessingTask] Processed 414 components and 0 services uploaded to project 7f671729-7f07-432e-91e5-baefadb6d6fa",
    "criTime": "2024-01-26T21:01:58.67775665Z",
    "timestamp": "1706302918677756650"
  },
  {
    "message": "2024-01-26 21:01:58,652 INFO [BomUploadProcessingTask] Processing CycloneDX dependency graph for project: 7f671729-7f07-432e-91e5-baefadb6d6fa",
    "criTime": "2024-01-26T21:01:58.652898821Z",
    "timestamp": "1706302918652898821"
  },
  {
    "message": "2024-01-26 21:01:58,652 INFO [BomUploadProcessingTask] Identified 414 new components",
    "criTime": "2024-01-26T21:01:58.652868751Z",
    "timestamp": "1706302918652868751"
  },
  {
    "message": "2024-01-26 21:01:49,084 INFO [BomUploadProcessingTask] Processing CycloneDX BOM uploaded to project: 7f671729-7f07-432e-91e5-baefadb6d6fa",
    "criTime": "2024-01-26T21:01:49.084419816Z",
    "timestamp": "1706302909084419816"
  },
  {
    "message": "2024-01-26 21:00:49,408 INFO [ProjectResource] Project anson-test : 2024-01-26-1 created by anson",
    "criTime": "2024-01-26T21:00:49.40891306Z",
    "timestamp": "1706302849408913060"
  }
]

The first n tasks took approximately 30 seconds to complete. The policy analysis took approximately 2 minutes to complete.

Here is some additional information about our system:

Our Dependency Track API server runs in a container in k8s on node with the following hardware specs: up to 8 CPUs and 30 GB memory requestable. The SBOM I tested against introduced 414 components, checked against 12 policies.

Policies include:

• Checking against vulnerability ids
• Checking component age
• Checking multiple license groups
• Check license
• Severity Level

We have 490 licenses, 19 license groups, and 236837 vulnerabilities.

During the processing, cpu and memory usage increased slightly, as noted by the following data points, which is to be expected:

Unfortunately, I do not have instrumentation set up that shows me response times for database queries, but I could see that the database was processing a fair number of requests during the SBOM analysis period, which is to be expected:

Please let me know if there are specific areas that would be helpful to dig into to identify the policy analysis slowness.