Testing can leave a stale Manifest.toml, leading to test failures

[c42f]

After #55, I get the following somewhat inscrutable test failure on my machine, on Julia 1.10.

@filchristou - I think it's best to resolve this before releasing a new RemoteREPL version. But once it's sorted, I think we can just release RemoteREPL-1.0 and get into the major version numbers. It's an incompatible release in some sense, since we've bumped the PROTOCOL_VERSION.

To reproduce, make sure you've got an ssh server running on your machine and can log into localhost so the tunnel will work.

julia> using Pkg

julia> Pkg.test("RemoteREPL", test_args=["use_ssh=true"])
     Testing RemoteREPL
      Status `/tmp/jl_5CWYeP/Project.toml`
  [1bd9f7bb] RemoteREPL v0.2.17 `~/.julia/dev/RemoteREPL`
  [b873ce64] ReplMaker v0.2.7
  [9bd350c2] OpenSSH_jll v9.3.2+0
  [56ddb016] Logging
  [3fa0cd96] REPL
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test
  [cf7118a7] UUIDs
      Status `/tmp/jl_5CWYeP/Manifest.toml`
  [692b3bcd] JLLWrappers v1.5.0
  [21216c6a] Preferences v1.4.3
  [1bd9f7bb] RemoteREPL v0.2.17 `~/.julia/dev/RemoteREPL`
  [b873ce64] ReplMaker v0.2.7
  [9bd350c2] OpenSSH_jll v9.3.2+0
  [458c3c95] OpenSSL_jll v3.0.14+0
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [b77e0a4c] InteractiveUtils
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [83775a58] Zlib_jll v1.2.13+1
Precompiling project...
  1 dependency successfully precompiled in 2 seconds. 7 already precompiled.
     Testing Running tests...
Test Summary:             | Pass  Total  Time
Protocol header handshake |    5      5  0.0s
Test Summary:     | Pass  Total  Time
Repl Magic syntax |    8      8  0.0s
Test Summary: | Pass  Total  Time
Prompt text   |    6      6  0.1s
[ Info: Running tests with SSH tunnel
[ Info: Using session id 9ac0ff8e-71e4-449d-af99-df812df0917a
┌ Warning: ssh() is deprecated, use the non-do-block form
│   caller = ip:0x0
└ @ Core :-1
┌ Debug: Connecting SSH tunnel to remote address 127.111.111.111 via ssh tunnel to 44375
│   ssh_cmd = `/home/c42f/.julia/artifacts/2b5ac70d0e216e6edbc4e5418cb8605ac4547bbc/bin/ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -L 127.0.0.1:44589:localhost:44375 127.111.111.111`
└ @ RemoteREPL ~/.julia/dev/RemoteREPL/src/tunnels.jl:43
┌ Warning: Process I/O error
│   exception =
│    ArgumentError: ensureroom failed, IOBuffer is not writeable
│    Stacktrace:
│     [1] ensureroom_slowpath(io::IOBuffer, nshort::UInt64)
│       @ Base ./iobuffer.jl:311
│     [2] ensureroom
│       @ ./iobuffer.jl:333 [inlined]
│     [3] unsafe_write(to::IOBuffer, p::Ptr{UInt8}, nb::UInt64)
│       @ Base ./iobuffer.jl:458
│     [4] unsafe_write
│       @ ./io.jl:698 [inlined]
│     [5] write
│       @ ./io.jl:721 [inlined]
│     [6] write(to::IOBuffer, from::Base.PipeEndpoint)
│       @ Base ./io.jl:768
│     [7] (::Base.var"#788#789"{IOBuffer, Bool, Base.PipeEndpoint, IOBuffer, Base.PipeEndpoint})()
│       @ Base ./process.jl:300
└ @ Base process.jl:302
[ Info: Using session id 1ea638f6-40b7-4871-bffb-12020166eb55
┌ Debug: Connecting SSH tunnel to remote address 127.111.111.111 via ssh tunnel to 44375
│   ssh_cmd = `/home/c42f/.julia/artifacts/2b5ac70d0e216e6edbc4e5418cb8605ac4547bbc/bin/ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -L 127.0.0.1:42859:localhost:44375 127.111.111.111`
└ @ RemoteREPL ~/.julia/dev/RemoteREPL/src/tunnels.jl:43
┌ Warning: Process I/O error
│   exception =
│    ArgumentError: ensureroom failed, IOBuffer is not writeable
│    Stacktrace:
│     [1] ensureroom_slowpath(io::IOBuffer, nshort::UInt64)
│       @ Base ./iobuffer.jl:311
│     [2] ensureroom
│       @ ./iobuffer.jl:333 [inlined]
│     [3] unsafe_write(to::IOBuffer, p::Ptr{UInt8}, nb::UInt64)
│       @ Base ./iobuffer.jl:458
│     [4] unsafe_write
│       @ ./io.jl:698 [inlined]
│     [5] write
│       @ ./io.jl:721 [inlined]
│     [6] write(to::IOBuffer, from::Base.PipeEndpoint)
│       @ Base ./io.jl:768
│     [7] (::Base.var"#788#789"{IOBuffer, Bool, Base.PipeEndpoint, IOBuffer, Base.PipeEndpoint})()
│       @ Base ./process.jl:300
└ @ Base process.jl:302
[ Info: Using session id 9934ef90-36b3-4c86-b4a8-5358478ea4f1
┌ Debug: Connecting SSH tunnel to remote address 127.111.111.111 via ssh tunnel to 44375
│   ssh_cmd = `/home/c42f/.julia/artifacts/2b5ac70d0e216e6edbc4e5418cb8605ac4547bbc/bin/ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -L 127.0.0.1:39361:localhost:44375 127.111.111.111`
└ @ RemoteREPL ~/.julia/dev/RemoteREPL/src/tunnels.jl:43
┌ Warning: Process I/O error
│   exception =
│    ArgumentError: ensureroom failed, IOBuffer is not writeable
│    Stacktrace:
│     [1] ensureroom_slowpath(io::IOBuffer, nshort::UInt64)
│       @ Base ./iobuffer.jl:311
│     [2] ensureroom
│       @ ./iobuffer.jl:333 [inlined]
│     [3] unsafe_write(to::IOBuffer, p::Ptr{UInt8}, nb::UInt64)
│       @ Base ./iobuffer.jl:458
│     [4] unsafe_write
│       @ ./io.jl:698 [inlined]
│     [5] write
│       @ ./io.jl:721 [inlined]
│     [6] write(to::IOBuffer, from::Base.PipeEndpoint)
│       @ Base ./io.jl:768
│     [7] (::Base.var"#788#789"{IOBuffer, Bool, Base.PipeEndpoint, IOBuffer, Base.PipeEndpoint})()
│       @ Base ./process.jl:300
└ @ Base process.jl:302
[ Info: Using session id df073aa2-fef1-43bc-8cda-c1fd28aaed31
┌ Debug: Connecting SSH tunnel to remote address 127.111.111.111 via ssh tunnel to 44375
│   ssh_cmd = `/home/c42f/.julia/artifacts/2b5ac70d0e216e6edbc4e5418cb8605ac4547bbc/bin/ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o ServerAliveInterval=60 -N -L 127.0.0.1:39001:localhost:44375 127.111.111.111`
└ @ RemoteREPL ~/.julia/dev/RemoteREPL/src/tunnels.jl:43
┌ Warning: Process I/O error
│   exception =
│    ArgumentError: ensureroom failed, IOBuffer is not writeable
│    Stacktrace:
│     [1] ensureroom_slowpath(io::IOBuffer, nshort::UInt64)
│       @ Base ./iobuffer.jl:311
│     [2] ensureroom
│       @ ./iobuffer.jl:333 [inlined]
│     [3] unsafe_write(to::IOBuffer, p::Ptr{UInt8}, nb::UInt64)
│       @ Base ./iobuffer.jl:458
│     [4] unsafe_write
│       @ ./io.jl:698 [inlined]
│     [5] write
│       @ ./io.jl:721 [inlined]
│     [6] write(to::IOBuffer, from::Base.PipeEndpoint)
│       @ Base ./io.jl:768
│     [7] (::Base.var"#788#789"{IOBuffer, Bool, Base.PipeEndpoint, IOBuffer, Base.PipeEndpoint})()
│       @ Base ./process.jl:300
└ @ Base process.jl:302
Client server tests: Error During Test at /home/c42f/.julia/dev/RemoteREPL/test/runtests.jl:121
  Got exception outside of a @test
  RemoteREPL protocol magic number mismatch: "" != "RemoteREPL"
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:35
    [2] verify_header(io::TCPSocket, ser_version::Int64)
      @ RemoteREPL ~/.julia/dev/RemoteREPL/src/client.jl:82
    [3] verify_header
      @ ~/.julia/dev/RemoteREPL/src/client.jl:77 [inlined]
    [4] setup_connection!(conn::RemoteREPL.Connection)
      @ RemoteREPL ~/.julia/dev/RemoteREPL/src/client.jl:143
    [5] RemoteREPL.Connection(; host::IPv4, port::UInt16, tunnel::Symbol, ssh_opts::Cmd, region::Nothing, namespace::Nothing, in_module::Symbol, session_id::Nothing)
      @ RemoteREPL ~/.julia/dev/RemoteREPL/src/client.jl:127
    [6] Connection
      @ ~/.julia/dev/RemoteREPL/src/client.jl:116 [inlined]
    [7] wait_conn(host::IPv4, port::UInt16, use_ssh::Bool; max_tries::Int64, session_id::Nothing)
      @ Main ~/.julia/dev/RemoteREPL/test/runtests.jl:70
    [8] wait_conn(host::IPv4, port::UInt16, use_ssh::Bool)
      @ Main ~/.julia/dev/RemoteREPL/test/runtests.jl:67
    [9] macro expansion
      @ ~/.julia/dev/RemoteREPL/test/runtests.jl:122 [inlined]
   [10] macro expansion
      @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [11] top-level scope
      @ ~/.julia/dev/RemoteREPL/test/runtests.jl:122
   [12] include(fname::String)
      @ Base.MainInclude ./client.jl:489
   [13] top-level scope
      @ none:6
   [14] eval
      @ ./boot.jl:385 [inlined]
   [15] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:291
   [16] _start()
      @ Base ./client.jl:552
Test Summary:       | Error  Total   Time
Client server tests |     1      1  13.7s
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
in expression starting at /home/c42f/.julia/dev/RemoteREPL/test/runtests.jl:119
ERROR: Package RemoteREPL errored during testing
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Types.jl:70
 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{…}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
   @ Pkg.Operations ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2019
 [3] test
   @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1900 [inlined]
 [4] test(ctx::Pkg.Types.Context, pkgs::Vector{…}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Vector{…}, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{…})
   @ Pkg.API ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:444
 [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{test_args::Vector{String}})
   @ Pkg.API ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
 [6] test
   @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined]
 [7] #test#74
   @ ~/.julia/juliaup/julia-1.10.4+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:146 [inlined]
 [8] top-level scope
   @ REPL[2]:1
Some type information was truncated. Use `show(err)` to see complete types.

[filchristou]

Hey, I cannot reproduce.

Works for me on the main branch:

I think what shows up for me are these notorious soft errors that we choose to ignore for now.

Could you try out whether the RemoteRPL ssh in a live julia kernel works for you ? I also didn't have any problems with that using the main branch.

Your problem looks like the server and the client have different RemoteREPL versions.
Can it be something like that you have RemoteREPL in the global environment and

run(`$(Base.julia_cmd()) --project -e "using RemoteREPL, Sockets, UUIDs ; serve_repl($test_port)"`, wait=false)

gets RemoteREPL from the registry release, while the test works with the local clone ?

[c42f]

Thanks for trying to reproduce!

It turns out this was a stale manifest in my development environment. Perhaps the julia --project in the tests is incorrect. I'm not sure how we should do that - we do need the dev copy of RemoteREPL to be loaded; we really want to duplicate the test environment in the spawned processes. Maybe TestEnv.jl is the right thing but it doesn't seem exactly correct?

[c42f]

Ok, I've done some fixes in #71 which should make the tests a lot cleaner. Including trying to run the server subprocesses in the correct test environment. Hopefully that'll fix the root cause here.

[filchristou]

I think the julia --project is necessary. Otherwise the general repository version will be downloaded.
TestEnv.jl is a good way forward but I was getting these weird errors with the CI, so that's why I reverted.
I will have a look at #71, maybe that fixes them.
What I commonly do I create a test environment like in ./test/Project.toml and I ] dev .. the core package. That works well but breaks easy ] testing. Also since julia v1.11 we will not need to check-in the ./test/Manifest toml because of JuliaLang/Pkg.jl#3783.