Capture licenses for all packages

[kzantow]

Syft should be able to include license information for packages it finds. Sometimes this information is present in the metadata on disk, other times it is only available by some remote source. This is an uber-issue about capturing licenses for all packages. Each ecosystem will likely have a different mechanism of capturing license information.

Ecosystems:

• Apk
• Dpkg
• ALPM
• binary (License not pickedup for binaries like java (openjdk), node (nodejs) #2765)
• Conan (conan.lock)
• Conan (conanfile.txt)
• Dart (pubspec.lock)
• .NET (deps.json) Get licenses for NuGet packages #1227
• .NET (from binary)
• Elixir (mix.lock)
• Erlang (rebar.lock)
• Github actions workflows (workflows using actions)
• Golang (go.mod)
• Golang (binary)
• Haskell (stack.yaml)
• Haskell (stack.yaml.lock)
• Haskell (cabal.project.freeze)
• Java (nested jars)
• Java (pom.xml)
• Java (gradle.lockfile)
• Javascript (package.json) Syft cannot get license from package.json in nested node_modules dir #2330
• Javascript (package-lock.json)
• Javascript (yarn.lock)
• Javascript (pnpm-lock.yaml)
• Kernel modules
• Nix (store)
• PHP (installed.json)
• PHP (composer.lock)
• Portage (contents file)
• Python (poetry.lock)
• Python (setup.py)
• Python (requirements.txt)
• Python (Pipfile.lock)
• Python (egg/wheel metadata)
• R (description file)
• RPM (db)
• RPM (rpm file)
• Ruby (gemfile.lock)
• Ruby (specifications gemspec)
• Rust (cargo.lock)
• Rust (binary)
• SBOM
• Swift (package.resolved)
• Swift (Podfile.lock)

Some related issues:

• Invoke known tools to gather build-time dependency information #1562
• Classify licenses based on file contents #656
• Can not have license ID #1964

[mykaul]

@kzantow - why is Go marked as checked? How do we get the license of Go modules?

[tgerla]

@mykaul check out the golang section of the Syft configuration file: https://github.com/anchore/syft/wiki/configuration -- there are two settings, search-local-mod-cache-licenses and search-remote-licenses that can be enabled to retrieve license data.

[mykaul]

@mykaul check out the golang section of the Syft configuration file: https://github.com/anchore/syft/wiki/configuration -- there are two settings, search-local-mod-cache-licenses and search-remote-licenses that can be enabled to retrieve license data.

Thanks! I think what tricked me is that by default (syft-text?) you do not see the license, so I did not even bother to look further. Very helpful, thanks again.

[Annamikhlin]

Any idea why I get different output report when scanning the same package with remote search licenses: true for Go modules on different machines?

On one machine - licenses info is presented:

{
      "id": "5a2f10fe8c37697d",
      "name": "github.com/alecthomas/units",
      "version": "v0.0.0-20211218093645-b94a6e3cc137",
      "type": "go-module",
      "foundBy": "go-module-binary-cataloger",
      "locations": [
        {
          "path": "/opt/scylladb/node_exporter/node_exporter",
          "layerID": "sha256:2f9c5474dfcad40cab09d578dfc5c919a38e0125c99501a8d7abb8540ef14188",
          "accessPath": "/opt/scylladb/node_exporter/node_exporter",
          "annotations": {
            "evidence": "primary"
          }
        }
      ],
      "licenses": [
        {
          "value": "MIT",
          "spdxExpression": "MIT",
          "type": "concluded",
          "urls": [],
          "locations": [
            {
              "path": "github.com/alecthomas/[email protected]/COPYING",
              "accessPath": "github.com/alecthomas/[email protected]/COPYING"
            }
          ]
        }
      ],
      "language": "go",
      "cpes": [
        {
          "cpe": "cpe:2.3:a:alecthomas:units:v0.0.0-20211218093645-b94a6e3cc137:*:*:*:*:*:*:*",
          "source": "syft-generated"
        }
      ],
      "purl": "pkg:golang/github.com/alecthomas/[email protected]",
      "metadataType": "go-module-buildinfo-entry",
      "metadata": {
        "goCompiledVersion": "go1.21.4",
        "architecture": "amd64",
        "h1Digest": "h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=",
        "mainModule": "github.com/prometheus/node_exporter"
      }
    },

On another machine the license info is missing:

    {
      "id": "67d84fc35f370e95",
      "name": "github.com/alecthomas/units",
      "version": "v0.0.0-20211218093645-b94a6e3cc137",
      "type": "go-module",
      "foundBy": "go-module-binary-cataloger",
      "locations": [
        {
          "path": "/opt/scylladb/node_exporter/node_exporter",
          "layerID": "sha256:2f9c5474dfcad40cab09d578dfc5c919a38e0125c99501a8d7abb8540ef14188",
          "accessPath": "/opt/scylladb/node_exporter/node_exporter",
          "annotations": {
            "evidence": "primary"
          }
        }
      ],
      "licenses": [],
      "language": "go",
      "cpes": [
        {
          "cpe": "cpe:2.3:a:alecthomas:units:v0.0.0-20211218093645-b94a6e3cc137:*:*:*:*:*:*:*",
          "source": "syft-generated"
        }
      ],
      "purl": "pkg:golang/github.com/alecthomas/[email protected]",
      "metadataType": "go-module-buildinfo-entry",
      "metadata": {
        "goCompiledVersion": "go1.21.4",
        "architecture": "amd64",
        "h1Digest": "h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=",
        "mainModule": "github.com/prometheus/node_exporter"
      }
    },

both machines are installed with Fedora release 37

[mykaul]

Very the configuration file is identical and accessible in both machines. Perhaps run syft with debug will show it.

[Annamikhlin]

Any idea why I get different output report when scanning the same package with remote search licenses: true for Go modules on different machines?

On one machine - licenses info is presented:

{
      "id": "5a2f10fe8c37697d",
      "name": "github.com/alecthomas/units",
      "version": "v0.0.0-20211218093645-b94a6e3cc137",
      "type": "go-module",
      "foundBy": "go-module-binary-cataloger",
      "locations": [
        {
          "path": "/opt/scylladb/node_exporter/node_exporter",
          "layerID": "sha256:2f9c5474dfcad40cab09d578dfc5c919a38e0125c99501a8d7abb8540ef14188",
          "accessPath": "/opt/scylladb/node_exporter/node_exporter",
          "annotations": {
            "evidence": "primary"
          }
        }
      ],
      "licenses": [
        {
          "value": "MIT",
          "spdxExpression": "MIT",
          "type": "concluded",
          "urls": [],
          "locations": [
            {
              "path": "github.com/alecthomas/[email protected]/COPYING",
              "accessPath": "github.com/alecthomas/[email protected]/COPYING"
            }
          ]
        }
      ],
      "language": "go",
      "cpes": [
        {
          "cpe": "cpe:2.3:a:alecthomas:units:v0.0.0-20211218093645-b94a6e3cc137:*:*:*:*:*:*:*",
          "source": "syft-generated"
        }
      ],
      "purl": "pkg:golang/github.com/alecthomas/[email protected]",
      "metadataType": "go-module-buildinfo-entry",
      "metadata": {
        "goCompiledVersion": "go1.21.4",
        "architecture": "amd64",
        "h1Digest": "h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=",
        "mainModule": "github.com/prometheus/node_exporter"
      }
    },

On another machine the license info is missing:

    {
      "id": "67d84fc35f370e95",
      "name": "github.com/alecthomas/units",
      "version": "v0.0.0-20211218093645-b94a6e3cc137",
      "type": "go-module",
      "foundBy": "go-module-binary-cataloger",
      "locations": [
        {
          "path": "/opt/scylladb/node_exporter/node_exporter",
          "layerID": "sha256:2f9c5474dfcad40cab09d578dfc5c919a38e0125c99501a8d7abb8540ef14188",
          "accessPath": "/opt/scylladb/node_exporter/node_exporter",
          "annotations": {
            "evidence": "primary"
          }
        }
      ],
      "licenses": [],
      "language": "go",
      "cpes": [
        {
          "cpe": "cpe:2.3:a:alecthomas:units:v0.0.0-20211218093645-b94a6e3cc137:*:*:*:*:*:*:*",
          "source": "syft-generated"
        }
      ],
      "purl": "pkg:golang/github.com/alecthomas/[email protected]",
      "metadataType": "go-module-buildinfo-entry",
      "metadata": {
        "goCompiledVersion": "go1.21.4",
        "architecture": "amd64",
        "h1Digest": "h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=",
        "mainModule": "github.com/prometheus/node_exporter"
      }
    },

both machines are installed with Fedora release 37

seems it is related to the same issue #2798
The creation of $HOME/go/pkg/mod directory - solved the issue
waiting for #2852 for official fix