Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tests for ceremony edge cases & improve type annotations #607

Merged

Conversation

MVrachev
Copy link
Member

@MVrachev MVrachev commented Jun 4, 2024

Description

Add:

  • test we cannot add online key as one of root keys
  • requirement that threshold > 2 & add test for that
  • test that expiration days must be a positive number

Additionally, improve type annotations in a few places
and move ceremony tests input inside conftest to be reused.

Related to #533

Additional requirements

  • Tests have been added for the bug fix or new feature
  • Docs have been added for the bug fix or new feature

Code of Conduct

By submitting this PR, you agree to follow our Code of Conduct.

  • I agree to follow this project's Code of Conduct

Copy link

codecov bot commented Jun 4, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.93%. Comparing base (47d0a69) to head (66a1bbc).
Report is 90 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #607      +/-   ##
==========================================
+ Coverage   98.90%   98.93%   +0.03%     
==========================================
  Files          20       26       +6     
  Lines        1367     1877     +510     
==========================================
+ Hits         1352     1857     +505     
- Misses         15       20       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@MVrachev
Copy link
Member Author

MVrachev commented Jun 4, 2024

Not sure how this not covered line: https://app.codecov.io/gh/repository-service-tuf/repository-service-tuf-cli/pull/607/blob/repository_service_tuf/helpers/tuf.py#L272 is part of my pr, but will check if I can do something about it.

@MVrachev
Copy link
Member Author

MVrachev commented Jun 4, 2024

I decided to just ignore it as it's part of the old md update which will be soon refactored.

@MVrachev MVrachev force-pushed the tests-ceremony-edge-cases branch from 7d4c7eb to 5f5549c Compare June 6, 2024 11:18
@MVrachev
Copy link
Member Author

MVrachev commented Jun 6, 2024

FT fix is required before merging this pr.
Have a look at: repository-service-tuf/repository-service-tuf#742

@MVrachev MVrachev mentioned this pull request Jun 6, 2024
6 tasks
Copy link
Collaborator

@lukpueh lukpueh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. I have one inline comment, which should be addressed, and two high level comments, which aren't blockers:

Generally, I think it's not feasible to test all code branches of the cli tool. Maybe it would be enough to just test this feature on the related helper?

If you do test the ceremony command, I recommend to store the complete list of inputs separately for each invocation, even if there is some redundancy. Re-using some parts (e.g input_step1, input_step3) and patching others (input_step2) makes the test a bit harder to read.

@@ -42,3 +145,6 @@ def test_ceremony(self, patch_getpass, patch_utcnow):

assert [s["keyid"] for s in sigs_r] == [s["keyid"] for s in sigs_e]
assert result.data == expected
# Asser that at least root_threshold number of public keys are added.
root_role = result.data["metadata"]["root"]["signed"]["roles"]["root"]
assert len(root_role["keyids"]) <= root_role["threshold"]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assert would also pass, if there were less than threshold keys, which is probably not what you want. Also, given that you control that threshold and number of keys are equal, why not assert for that?

@MVrachev
Copy link
Member Author

MVrachev commented Jun 7, 2024

This looks good. I have one inline comment, which should be addressed, and two high level comments, which aren't blockers:

Generally, I think it's not feasible to test all code branches of the cli tool. Maybe it would be enough to just test this feature on the related helper?

Well, that's a discussion worth having.
I am okay with having helper-specific tests instead of full-flow tests for those cases specific to the helpers.
For example, for threshold, we could test the helper itself only.
Still, given that the test is there and working I won't change it now.

If you do test the ceremony command, I recommend to store the complete list of inputs separately for each invocation, even if there is some redundancy. Re-using some parts (e.g input_step1, input_step3) and patching others (input_step2) makes the test a bit harder to read.

I disagree. I prefer reading what is changed as to have all of the inputs there and wonder are there any changes in any of the steps.
When you change only for example input2, then you know that the change is ONLY related to the root keys configuration.

MVrachev added 8 commits June 7, 2024 15:07
Signed-off-by: Martin Vrachev <[email protected]>
Add clarifying comment, remove a question and answer tuple that was
out of place as we are not given a choice if to add or remove a key
and make an assertion more precise as we control the threshold and
how many keys are loaded.

Signed-off-by: Martin Vrachev <[email protected]>
@MVrachev MVrachev force-pushed the tests-ceremony-edge-cases branch from ebd7603 to 66a1bbc Compare June 7, 2024 12:08
@MVrachev
Copy link
Member Author

MVrachev commented Jun 7, 2024

Rebased on top of main.
The only new/changed commit is the last one addressing @lukpueh comment.

@MVrachev
Copy link
Member Author

MVrachev commented Jun 7, 2024

Again, we need to have repository-service-tuf/repository-service-tuf#742 merged, before ft tests pass here.

@kairoaraujo kairoaraujo merged commit db0c94b into repository-service-tuf:main Jun 8, 2024
18 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants