Please clarify the versioning scheme and types of changes we should expect in a given major/minor/patch version

[briantist]

I've checked the readme and the docs and I could not find anything detailing whether the project uses semver, or some other versioning scheme, nor what types of changes we should expect to see.

Trying to use ansible-lint in CI pinned to "major" version ranges results in my CI being broken very frequently due to changes in the project, especially changes to rules (new rules, changes to existing ones, renaming, etc.), and these changes appear to be introduced in both minor and patch versions.

For example, v6.12.2 renames some rules and as far as I can tell offers no grace period where the old name still works.

While I'm sure there's an interpretation of new rules being a non-breaking change (a new backward-compatible feature), in practice due to the way ansible-lint is used, new rules will cause failures that didn't exist. I'm not sure how to address that, but I'm also not looking to address it in this discussion.

What I'm really asking for now is that whatever the project's policy on versioning and the types of changes that might be included, please detail those explicitly in the README and/or docsite, so that we can make an informed decision on pinning or whether to even use ansible-lint.

Thanks!

[ssbarnea]

Officially we aim to follow SemVer. Still, mistakes can happen and the only way to ensure reproducible behavior is if you pin ansible-lint version, especially as there is no way for us to unrelease a release once is out. If you use pre-commit to run the linter, that is an implied feature, alongside its ability to auto-upgrade monthly or quarterly.

I should also note that the linter entered a period of heavy/fast development about 6 months ago and will continue to be under heavy development for the next ~6 months. Once that is done we should see far less releases and far less surprises from one release to another, especially as one of our current focus is to improve the test coverage.

I would want to say that we aim for zero bugs, but that's a dream. A more realistic goal is to ensure that each release is more stable and better tested than the previous one.

Failures caused by new rules

That is the way to address them, just add them to wither warn_list or skip-list with a comment to look at them later. This allows you to update the linter and benefit from someo of the bug-fixes and continue working on your project without having to bother about some fresh rules that might not be yet mature enough. It might also not be the right time for you to spend time addressing some of the new rules. As long you put a comment before your rule disabling option like # temporary disabled there is nothing wrong with this approach.

[briantist]

Hi @ssbarnea thanks for your reply.

Mistakes can happen and I have no expectation of never having any bugs or mistaken releases.

Having a published policy allows us who use the tool to quickly report mistakes and bugs that don't follow the policy, instead of being unsure about the intention.

I'm not opening this to argue on what your policy should be, only asking that it be published somewhere easy to find.

---

Regarding your other suggestions:

I understand that pinning to a specific version is the only way to have consistent behavior. It comes with downsides like having to manually update the version even for what should be non-breaking patch releases, or missing out on important updates, or running an additional pipeline against newer versions that doesn't fail (can be tricky in some CI systems).

For me, this is all about CI; pre-commit is a not something I can (or want to) enforce on all the contributors and maintainers of my internal collections.

I should also note that the linter entered a period of heavy/fast development about 6 months ago and will continue to be under heavy development for the next ~6 months. Once that is done we should see far less releases and far less surprises from one release to another, especially as one of our current focus is to improve the test coverage.

Yes I've noticed that, but that's only more reason to explicitly state a versioning and release policy.
Also doesn't this doesn't really change the situation with regard to semver imo, if this is a "move fast and break things" period, then I expect to see a large number of major version releases in a short amount of time, which is just fine.

The project is not zerover, and if we're pinning to major versions we should not be seeing previously working things breaking every other week.

just add them to wither warn_list or skip-list with a comment to look at them later

This doesn't work for rules that never existed before. There's no way to say "ignore future new rules" (I don't necessarily think there should be either) so the process is always reactive.

If I were coming at this from the perspective of being a maintainer for a single collection, then sure, maybe I'd do this. When I have dozens of collections, this process is exhausting.

I am not asking you to solve issues related to my specific use case, just giving you some perspective on what the experience is like for a use case you may not have run into before.

When publishing a policy, please explicitly state that new, potentially failing rules could be introduced in <major|minor|patch> versions. Again, whatever the policy is please make it clear.

I also get that waiting to introduce new rules to major versions would either be too slow, reducing benefit of the linter, or new major versions would be released so fast that it could cause a similar headache as pinning, so I'm not suggesting doing that either.

I might have a suggestion for this that I'll post as a new idea.

---

Thanks for your time!