Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new Path filter function to index paths based on the filter #151

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open

Add new Path filter function to index paths based on the filter #151

wants to merge 25 commits into from

Conversation

gnmahanth
Copy link

@gnmahanth gnmahanth mentioned this pull request Jan 11, 2023
Closed
@gnmahanth
fix lint issue
08e5b9c 
Signed-off-by: gnmahanth <[email protected]>
wagoodman
wagoodman reviewed Jan 13, 2023
View reviewed changes
Copy link
Contributor

@wagoodman wagoodman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the improvement to the visitor 👍 I have a couple of comments:

  1. My preference on the changes at the top-level API is to lean into the existing functional options (type Option) approach and allow for a filter to be optional. In that sense, passing a default function that always returns true is alright, but a slightly more preferable approach would be to pass nil by default and not call the filter if it doesn't exist down in the visitor.
  2. A test is needed to show the effect of the path filter on the visitor.

Shout out if you want to chat more about this!

bradleyjones and others added 21 commits June 15, 2023 17:06
@bradleyjones @gnmahanth
update go to 1.19 (anchore#155)
178bc5c 
* feat: update golang to 1.19

Go 1.18 will become EOL with the upcoming 1.20 release

Signed-off-by: Bradley Jones <[email protected]>

* chore: update golangci lint

Move to a newer version of golangci lint compatibile with go 1.19 and
remove now deprecated linters.

Signed-off-by: Bradley Jones <[email protected]>

* chore: update deprecated ioutil functions

Signed-off-by: Bradley Jones <[email protected]>

* chore: add crypto/internal/boring to bouncer ignore list

Signed-off-by: Bradley Jones <[email protected]>

* fix: nolintlint formatting updated

Signed-off-by: Bradley Jones <[email protected]>

---------

Signed-off-by: Bradley Jones <[email protected]>
@wagoodman @gnmahanth
Add additional catalog indexes for performance (anchore#154)
61a4559 
* add additional catalog indexes for performance

Signed-off-by: Alex Goodman <[email protected]>

* [wip] link resolution

Signed-off-by: Alex Goodman <[email protected]>

* add leaf link resolution on tree responses (defer ancestor link resolution)

Signed-off-by: Alex Goodman <[email protected]>

* add filetree search context

Signed-off-by: Alex Goodman <[email protected]>

* add tests for new search context object

Signed-off-by: Alex Goodman <[email protected]>

* remove unused tar header fields from file.Metadata struct

Signed-off-by: Alex Goodman <[email protected]>

* use singular file type definitions

Signed-off-by: Alex Goodman <[email protected]>

* add logging for filetree searches

Signed-off-by: Alex Goodman <[email protected]>

* add limited support for glob classes and alternatives

Signed-off-by: Alex Goodman <[email protected]>

* add failing test to show that index shortcircuits correct behavior

Signed-off-by: Alex Goodman <[email protected]>

* add link resolution via filetree search context

Signed-off-by: Alex Goodman <[email protected]>

* allow index symlink resolution to function through cycles

Signed-off-by: Alex Goodman <[email protected]>

* add tests for filetree.Index

Signed-off-by: Alex Goodman <[email protected]>

* add search by parent basename and fix requirements filtering

Signed-off-by: Alex Goodman <[email protected]>

* sort search results

Signed-off-by: Alex Goodman <[email protected]>

* change file.Type to int + fix layer 0 squashed search context

Signed-off-by: Alex Goodman <[email protected]>

* more cleanup

Signed-off-by: Alex Goodman <[email protected]>

* switch to generic set implementation

Signed-off-by: Alex Goodman <[email protected]>

* update linter

Signed-off-by: Alex Goodman <[email protected]>

* replace generic set implemetation with plain set (unstable in go1.19)

Signed-off-by: Alex Goodman <[email protected]>

* introduce filtree builter and foster usage of reader interfaces

Signed-off-by: Alex Goodman <[email protected]>

* rename content helper functions

Signed-off-by: Alex Goodman <[email protected]>

* update docs with background

Signed-off-by: Alex Goodman <[email protected]>

* fix get_xid for cross compilation

Signed-off-by: Alex Goodman <[email protected]>

* upgrade CI validations workflow

Signed-off-by: Alex Goodman <[email protected]>

* fix snapshot builds

Signed-off-by: Alex Goodman <[email protected]>

* add tests for file.Index.GetByFileType

Signed-off-by: Alex Goodman <[email protected]>

* rename file.Type and file.Resolution

Signed-off-by: Alex Goodman <[email protected]>

* ensure that glob results match search facade

Signed-off-by: Alex Goodman <[email protected]>

* replace stringset implementation + move resolution tests

Signed-off-by: Alex Goodman <[email protected]>

* add note about podman dependency for testing

Signed-off-by: Alex Goodman <[email protected]>

* address PR comments

Signed-off-by: Alex Goodman <[email protected]>

* remove extra whitespace

Signed-off-by: Alex Goodman <[email protected]>

* constrain OS build support

Signed-off-by: Alex Goodman <[email protected]>

* update/remove TODO comments

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Alex Goodman <[email protected]>
@dependabot @gnmahanth
Bump github.com/containerd/containerd from 1.6.12 to 1.6.18 (anchore#156
38cc5e5 
)
@westonsteimel @gnmahanth
fix: bump golang.org/x/net to v0.7.0 (anchore#157)
bd07bb5 
Signed-off-by: Weston Steimel <[email protected]>
@wagoodman @spiffcs @gnmahanth
Fix link cycle detection (anchore#158)
66b650a 
* test: add failing test for cycle case

Signed-off-by: Christopher Phillips <[email protected]>

* test: test updates sym links

Signed-off-by: Christopher Phillips <[email protected]>

* change the filetree recursive pathset to represent open calls

Signed-off-by: Alex Goodman <[email protected]>

* add another cycle test

Signed-off-by: Alex Goodman <[email protected]>

* change filetree attempting path set to counters

Signed-off-by: Alex Goodman <[email protected]>

* remove comment

Signed-off-by: Alex Goodman <[email protected]>

* fix linting

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Christopher Phillips <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: Christopher Phillips <[email protected]>
@spiffcs @wagoodman @gnmahanth
Link Detection Stack Depth FailSafe (anchore#159)
c032eae 
* test: add failing test for cycle case

Signed-off-by: Christopher Phillips <[email protected]>

* test: test updates sym links

Signed-off-by: Christopher Phillips <[email protected]>

* change the filetree recursive pathset to represent open calls

Signed-off-by: Alex Goodman <[email protected]>

* add another cycle test

Signed-off-by: Alex Goodman <[email protected]>

* change filetree attempting path set to counters

Signed-off-by: Alex Goodman <[email protected]>

* remove comment

Signed-off-by: Alex Goodman <[email protected]>

* fix linting

Signed-off-by: Alex Goodman <[email protected]>

* feat: decrement stack depth

Signed-off-by: Christopher Phillips <[email protected]>

* test: remove old wip test name

Signed-off-by: Christopher Phillips <[email protected]>

* chore: style updates

Signed-off-by: Christopher Phillips <[email protected]>

* feat: move maxLinkDepth decrement to inside ancestor loop

Signed-off-by: Christopher Phillips <[email protected]>

* feat: move maxLinkDepth decrement into resolveNodeLinks loop

Signed-off-by: Christopher Phillips <[email protected]>

* feat: move detection to top and write minimal test case

Signed-off-by: Christopher Phillips <[email protected]>

* test: update linkResolution test to use internal value

Signed-off-by: Christopher Phillips <[email protected]>

---------

Signed-off-by: Christopher Phillips <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: Alex Goodman <[email protected]>
@kzantow @gnmahanth
fix: thread safety for progress (anchore#162)
dbb5dc0
@KushalBeniwal @gnmahanth
chore: fix typo (anchore#163)
9425f2a 
Signed-off-by: Kushal Beniwal <[email protected]>
@vaikas @gnmahanth
Add ability to explicitly specify an Authenticator as well as Keychai…
7aad86f 
…n. (anchore#164)

* Add Authenticator as an explicit option.

Signed-off-by: Ville Aikas <[email protected]>

* wire in keychain also.

Signed-off-by: Ville Aikas <[email protected]>

---------

Signed-off-by: Ville Aikas <[email protected]>
@wagoodman @gnmahanth
fix directory leak leading to out of disk (anchore#161)
19345b8 
* fix tmpDirGenerator chain of responsibility associated with anchore#132

Signed-off-by: Joseph Barnett <[email protected]>
Signed-off-by: [email protected] <[email protected]>

* reduce log message to debug
Signed-off-by: Joseph Barnett <[email protected]>

Signed-off-by: [email protected] <[email protected]>

* restore global cleanup function

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Joseph Barnett <[email protected]>
Signed-off-by: [email protected] <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: Alex Goodman <[email protected]>
@wagoodman @gnmahanth
Set the default platform for select sources based on host arch (ancho…
b7456e6 
…re#152)

* set the default platform for select sources based on host arch

Signed-off-by: Alex Goodman <[email protected]>

* decompose into smaller function and add tests for setting default platform

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Alex Goodman <[email protected]>
@dependabot @gnmahanth
Bump github.com/docker/docker (anchore#167)
d9a9a51 
Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.12+incompatible to 20.10.24+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v20.10.12...v20.10.24)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@wobito @wagoodman @gnmahanth
Preserve time and expose link strategy (anchore#166)
6c1941d 
* bind file times to metadata

Signed-off-by: Adrian Wobito <[email protected]>

* expose link strategy to walkConditions

Signed-off-by: Adrian Wobito <[email protected]>

* update: test times

Signed-off-by: Adrian Wobito <[email protected]>

* update: link options api

Signed-off-by: Adrian Wobito <[email protected]>

* stub mod time for dynamic tar test fixtures

Signed-off-by: Alex Goodman <[email protected]>

* set mtime on tar fixture explicitly

Signed-off-by: Alex Goodman <[email protected]>

* always interpret tar header timestamps as UTC

Signed-off-by: Alex Goodman <[email protected]>

* interpret all file metadata timestamps as UTC

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Adrian Wobito <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: Alex Goodman <[email protected]>
@Amar-Babu @gnmahanth
Bump github.com/containerd/containerd from 1.6.18 to 1.7.0 (anchore#168)
162a14e 
Signed-off-by: Amar Babu <[email protected]>
@wagoodman @gnmahanth
Add format make target (anchore#170)
abe6881 
* fix linting

Signed-off-by: Alex Goodman <[email protected]>

* add format make target

Signed-off-by: Alex Goodman <[email protected]>

* add developing reference to format target

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Alex Goodman <[email protected]>
@tri-adam @wagoodman @gnmahanth
refactor: embed fs.FileInfo within file.Metadata (anchore#172)
3ab9510 
* refactor: embed fs.FileInfo within file.Metadata

Embed the original `fs.FileInfo` directly within the `file.Metadata
struct`, replacing the previous `Size`, `IsDir`, `Mode`, and `ModTime`
fields.

Signed-off-by: Adam Hughes <[email protected]>

* add ManualInfo helper struct

Signed-off-by: Alex Goodman <[email protected]>

---------

Signed-off-by: Adam Hughes <[email protected]>
Signed-off-by: Alex Goodman <[email protected]>
Co-authored-by: Alex Goodman <[email protected]>
@willmurphyscode @gnmahanth
Specify platform in integ test images (anchore#181)
73cb649 
Otherwise, "make integration" will fail on M1 Macs. Note that this is a
workaround which assumes that, if "make integration" is run on an arm64
host, that host is able to run containers from amd64 images. This change
enables development on stereoscope on M1 Macs, but may not enable it on
other arm64 machines.

Signed-off-by: Will Murphy <[email protected]>
@dependabot @gnmahanth
Bump github.com/docker/distribution (anchore#178)
ad7eb21 
Bumps [github.com/docker/distribution](https://github.com/docker/distribution) from 2.8.1+incompatible to 2.8.2+incompatible.
- [Release notes](https://github.com/docker/distribution/releases)
- [Commits](distribution/distribution@v2.8.1...v2.8.2)

---
updated-dependencies:
- dependency-name: github.com/docker/distribution
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@willmurphyscode @gnmahanth
Change platform selection logic (anchore#189)
7f87e8b 
This reverts commit d7551b7.

Defaulting to platform for all providers resulted in syft having unnecessary
when pulling an image that had a particular digest, if that digest didn't match
the architecture of the host running the pull. Revert commit
d7551b7, which introduced that error, but then
add platform defaulting logic back for the OCI Registry Provider, since
defaulting there has been specifically requested. Add integ tests to cover the
new behavior. Also, update integ tests to use the manifest ID for assertsion,
since the RepoDigests array can be empty.

Signed-off-by: Will Murphy <[email protected]>
@gnmahanth
Merge branch 'anchore:main' into main
dde2147
@gnmahanth
Merge branch 'main' into path-filter
b9ab08f
@wagoodman
Copy link
Contributor

@gnmahanth would you like help incorporating the above comments?

@wagoodman wagoodman self-assigned this Jul 25, 2023
@wagoodman wagoodman removed their assignment Feb 26, 2024
@wagoodman wagoodman self-assigned this May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wagoodman wagoodman wagoodman left review comments

At least 1 approving review is required to merge this pull request.

Assignees

@wagoodman wagoodman

Labels
None yet
Projects
OSS
Status: Stalled
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

syft memory usage is high