Test lower compat bounds?

[oschulz]

The current DocStringExtensions = "0.9.3" makes DifferentiationInterface incompatible with some packages that are still stuck on DocStringExtensions v0.8. Would it be possible to relax the version bound?

[gdalle]

Oh yes definitely. I wonder why CompatHelper didn't pick it up. Care to open a PR?

[oschulz]

I wonder why CompatHelper didn't pick it up

Because I wanted to extend it downwards to DocStringExtensions = "0.8, 0.9", not upwards. :-) But I can't find the package anymore that caused the trouble, maybe my dependencies are all v0.9-compatible now. I'll close this for now ...

[oschulz]

In any case, since DifferentiationInterface wil (hopefully) be wide used, maybe we could use https://github.com/julia-actions/julia-downgrade-compat to keep the dependency version bounds wide, but in a safe manner?

[adrhill]

Sounds like a great idea, thanks!

[gdalle]

While a good idea in principle, downgrade compat testing is more tricky than it looks because I perform Pkg operations inside the DI test suite. The DI test suite is split into one CI workflow per AD backend, and each of those workflows activates a custom environment like DI/test/Back/Diffractor/Project.toml. This is designed to avoid instantiating an environment with all backends at once, which is very tricky due to conflicts (especially on older versions of said backends).
The trouble with this method is that compat bounds of the DI package are not enforced in the environment activated on top of the test environment, which is why I had to explicitly bound Diffractor for instance. So I'm not sure how the GitHub action you refer to will handle that.

[adrhill]

each of those workflows activates a custom environment like DI/test/Back/Diffractor/Project.toml.

This could possibly be handled through the projects argument:
https://github.com/julia-actions/julia-downgrade-compat?tab=readme-ov-file#usage

# Comma-separated list of Julia projects to modify. The Project.toml files in all of
# these directories will be modified.
# Example: ., test, docs
# Default: .
projects: ''

[gdalle]

I don't think so because the individual Project.toml files don't store the compatibility bounds of AD backends for DI: that's stored in the main Project.toml file. So we would need a way to enforce these bounds in the stacked environment.

[gdalle]

Maybe we could add the packages of test/.../Project.toml to the current environment instead of activating a new one? Not sure if there is a command for that

[adrhill]

Could you point me to the location where the /test/Back environments get activated?

[gdalle]

DifferentiationInterface.jl/DifferentiationInterface/test/runtests.jl

Lines 21 to 32 in a0a96c7

	function subtest(category, folder)
	Pkg.activate(joinpath(@__DIR__, category, folder))
	Pkg.instantiate()
	@testset "$file" for file in filter(
	endswith(".jl"), readdir(joinpath(@__DIR__, category, folder))
	)
	@info "Testing $category/$folder/$file"
	include(joinpath(@__DIR__, category, folder, file))
	end
	Pkg.activate(TEST_ENV)
	return nothing
	end

[gdalle]

And a separate mechanism is at work for adding

• the right version of DIT to the DI test env (

  DifferentiationInterface.jl/DifferentiationInterface/test/runtests.jl

  Lines 5 to 10 in a0a96c7

  	DIT_PATH = joinpath(@__DIR__, "..", "..", "DifferentiationInterfaceTest")
  	if isdir(DIT_PATH)
  	Pkg.develop(; path=DIT_PATH)
  	else
  	Pkg.add("DifferentiationInterfaceTest")
  	end

  )
• the right version of DI to the DIT test env (

  DifferentiationInterface.jl/DifferentiationInterfaceTest/test/runtests.jl

  Lines 8 to 11 in a0a96c7

  	DI_PATH = joinpath(@__DIR__, "..", "..", "DifferentiationInterface")
  	if isdir(DI_PATH)
  	Pkg.develop(; path=DI_PATH)
  	end

  )

because otherwise Pkg will try to grab it from the general registry

[adrhill]

To the best of my understanding, the way julia-downgrade-compat works is by simply rewriting the Project.toml files at the specified projects paths.

If we first activate DI/Project.toml before activating Pkg.activate(joinpath(@__DIR__, category, folder)), wouldn't the downgraded compat entries be enforced in the /test/BACK environments as well?

[gdalle]

No because compat bounds of environments deeper in the LOAD_PATH stack are not necessarily enforced:

• https://docs.julialang.org/en/v1/manual/code-loading/#Environment-stacks
• https://discourse.julialang.org/t/compat-bound-violations-with-stacked-environements/116492

Side note: activating one then the other is not enough, we need to push the former into the LOAD_PATH before activating the latter

[gdalle]

I thought it was cleaner to store this stuff in a Project.toml but maybe I can just write the list of packages explicitly. Let me open a quick PR.

[gdalle]

See #381, if tests pass on this one you can rebase #380 on it afterwards @adrhill

[adrhill]

#380 reveals some annoying properties of the julia-downgrade-compat action.
It looks like it rewrites the Project.toml to only keep the lowest compat entry, which often can't be properly resolved.

For example, the tests on #380 fail because SparseMatrixColorings has its DocStringExtensions compat entry set to v0.9:

ERROR: LoadError: Unsatisfiable requirements detected for package SparseMatrixColorings [0a514795]:
 SparseMatrixColorings [0a514795] log:
 ├─possible versions are: 0.1.0-0.3.5 or uninstalled
 ├─restricted to versions 0.3.0 by DifferentiationInterface [a0c0ee7d], leaving only versions 0.3.0
 │ └─DifferentiationInterface [a0c0ee7d] log:
 │   ├─possible versions are: 0.5.9 or uninstalled
 │   └─DifferentiationInterface [a0c0ee7d] is fixed to version 0.5.9
 └─restricted by compatibility requirements with DocStringExtensions [ffbed154] to versions: 0.1.0 or uninstalled — no versions left
   └─DocStringExtensions [ffbed154] log:
     ├─possible versions are: 0.4.6-0.9.3 or uninstalled
     └─restricted to versions 0.8.0 by DifferentiationInterface [a0c0ee7d], leaving only versions 0.8.0
       └─DifferentiationInterface [a0c0ee7d] log: see above

So these tests might be a headache in the future.

[gdalle]

Oh yeah I remember that discussion, essentially it tries to resolve all lower bounds at the same time.

https://discourse.julialang.org/t/psa-add-downgrade-ci-to-better-check-version-compatibility/110063/15?u=gdalle

[gdalle]

@adrhill can you modify #380 by

• undoing Relax DocStringExtensions compat entries #379 so that Pkg installs v0.9 of DocStringExtensions
• rebasing on Keep tests in same env #381

[adrhill]

undoing #379 so that Pkg installs v0.9 of DocStringExtensions

I bet that doing this will just reveal another dependency conflict.

I suggest we keep #379 (which @oschulz needs) and don't use julia-downgrade-compat until there is a way to automatically resolve an environment with lower compat bounds. We can't be the ones the manually resolve all lower bounds. Quoting from discourse:

trying to use [julia-downgrade-compat] on a big package with many deps just felt like a total waste of time. It needs to do something smarter than just pinning everything to the minimum versions.

[gdalle]

I bet that doing this will just reveal another dependency conflict.

Or maybe not, that's my hope.

If that's the only problem, gdalle/SparseMatrixColorings.jl#31 should be enough to fix it for good

[adrhill]

I bet that doing this will just reveal another dependency conflict.

Or maybe not, that's my hope.

If not, then that's pure luck.

We'll encounter the same problem of having to manually resolve lower compat bounds every time we need to modify lower bounds. And that's going to happen every time we require a new feature from a new release of any dependency.

[adrhill]

While we should test lower compat bounds, julia-downgrade-compat doesn't seem to be the answer for DI. We tried using it in #380, but it currently requires manual resolving of dependencies, which should ideally be handled by Pkg rather than a human. This problem was discussed on Discourse as well.

A proposal to add down to Pkg can be found here: JuliaLang/Pkg.jl#1062
Until this is added, this issue will probably have to be left open.