lessening the impact of new rules on existing content

[briantist]

This is tangentially related to #3178 but is a separate idea.

Problem

• New rules may be added to ansible-lint in minor (maybe patch?) versions
• A collection currently passing ansible-lint may fail due only to brand new rules, during non-major upgrades

Solutions that are not ideal

• We don't want to push all new rules to major versions because it significantly reduces the benefit those rules could offer
• Pinning versions to avoid new rules means we might miss important bugfixes and improvements to the linter engine itself

Proposal

• Leverage "profiles" by adding another dimension to them
• An extra configuration parameter that controls which set of rules for a given profile is used, based on when the rules were introduced
• Example parameter name: profile_version
• Example parameter values: major, minor, patch
• The idea is that if you choose, say, profile_version: major, then regardless of other engine changes, the list of rules used will be the one that existed when this major version was first released (X.0.0), even if new rules have been added to this profile since then.
• If you chose profile_version: minor, you would get all rules in this profile that existed in X.Y.0 (but if a rule were added in a patch version you would not get that one).
• The default value of profile_version can be patch to match existing behavior and encourage new rule adoption.

Alternative variations

• The parameter could instead take values like stable / edge or something like that

---

I'm sure there are some issues with the idea as it stands, but I hope this in an inspiration at least.

[ssbarnea]

@briantist A big thanks for opening this and including several ideas. While I personally do not love the idea of adding more configuration options to the linter, I also realise that we should find a solution for the problem.

The classic answer for this dilemma would be a stable branch and a development one, but I am realistic that for our particular case this would not work: we do not have the manpower to maintain two branches and the project still needs to move fast.

We could ask users that want the stable behavior to disable "experimental" tag in their config as we had a rule of always adding new rules as experimental.

The tricky bit is that sometimes we introduce sub-rules (different tag like foo[bar]) and these do inherit the experimental property from the rule itself.

On the other hand, I think it might be quite easy to have two profile configurations. I will try to look if I can add versioning to profiles and have a an implicit 'latest'.

The real benefit on having this is that you could use the latest code of the linter and still target a specific profile version that you accepted as being "safe" to use.

If I would implement that I would use a CalVer for profile versions like v23.3 being for March 2023 and I would suspect that we would only make a new profile version twice a year, or at least not more often that every quarter. The linter could also warn the user if a newer profile was released since they configured the linter to target a specific one.

[briantist]

Thanks @ssbarnea ! Those sound like very reasonable changes to me, and I think they would support a variety of use cases.

The real benefit on having this is that you could use the latest code of the linter and still target a specific profile version that you accepted as being "safe" to use.

Definitely agree! Wanting to take advantage of linter engine fixes and improvements, with a more "stable" set of rules is exactly what I'm looking for.

[briantist]

Hi @ssbarnea I'm wondering if there's been any progress on this, or any new ideas or proposals. The impact of new rules is still pretty high. Thanks!

[cidrblock]

This is a brilliant idea @briantist. TY for taking the time to raise this, I think this is something we should prioritize.

[briantist]

Thanks @cidrblock ! I'd guess the suggestions from @ssbarnea are more practical implementation-wise (#3183 (comment)) but yeah, I'm happy for anything that helps the underlying concern.

[dshafer-godaddy]

I will admit that my work team started pushing changes to our internal Ansible repo, and our PRs started failing because of the new role prefix variable requirement. It broke our workflow.