4.0.x generic binary build uses an incorrect version

[chenrui333]

Describe the bug

👋 looks like lots of plugins are bundled with rc in the path

...
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_client_sup.beam
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_fifo_client.beam
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_fifo_dlx_client.beam
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_observer_cli.beam
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_observer_cli_classic_queues.beam
plugins/rabbit-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_observer_cli_quorum_queues.beam
plugins/rabbitmq_auth_backend_oauth2-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.AddSigningKeyCommand.beam
plugins/rabbitmq_auth_backend_oauth2-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.AddUaaKeyCommand.beam
plugins/rabbitmq_consistent_hash_exchange-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Diagnostics.Commands.ConsistentHashExchangeRingStateCommand.beam
plugins/rabbitmq_federation-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.FederationStatusCommand.beam
plugins/rabbitmq_federation-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.RestartFederationLinkCommand.beam
plugins/rabbitmq_management-4.0.0+rc.2.6.g7c7b6a5/priv/www/cli/
plugins/rabbitmq_management-4.0.0+rc.2.6.g7c7b6a5/priv/www/js/oidc-oauth/oidc-client-ts.3.0.1.min.js
plugins/rabbitmq_management-4.0.0+rc.2.6.g7c7b6a5/priv/www/js/oidc-oauth/oidc-client-ts.3.0.1.min.js.map
plugins/rabbitmq_management-4.0.0+rc.2.6.g7c7b6a5/priv/www/js/oidc-oauth/oidc-client-ts.js
plugins/rabbitmq_management_agent-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ResetStatsDbCommand.beam
plugins/rabbitmq_mqtt-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListMqttConnectionsCommand.beam
plugins/rabbitmq_peer_discovery_etcd-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbitmq_peer_discovery_etcd_v3_client.beam
plugins/rabbitmq_shovel-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.DeleteShovelCommand.beam
plugins/rabbitmq_shovel-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.RestartShovelCommand.beam
plugins/rabbitmq_shovel-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ShovelStatusCommand.beam
plugins/rabbitmq_stomp-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStompConnectionsCommand.beam
plugins/rabbitmq_stomp-4.0.0+rc.2.6.g7c7b6a5/ebin/rabbit_stomp_client_sup.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.AddSuperStreamCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.DeleteSuperStreamCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamConnectionsCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamConsumerGroupsCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamConsumersCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamGroupConsumersCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamPublishersCommand.beam
plugins/rabbitmq_stream-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListStreamTrackingCommand.beam
plugins/rabbitmq_web_mqtt-4.0.0+rc.2.6.g7c7b6a5/ebin/Elixir.RabbitMQ.CLI.Ctl.Commands.ListWebMqttConnectionsCommand.beam

maybe regenerate release tarball with a clean path?

Reproduction steps

n/a

Expected behavior

n/a

Additional context

relates to Homebrew/homebrew-core#191130

[michaelklishin]

I can confirm this for the generic UNIX packages but not Debian, and nothing in the build logs stands out until the very last stage of the generic UNIX package build.

The parts of th pipeline that produce GA releases are independent from the parts that produce alphas, betas and RCs, respectively.

[michaelklishin]

Here is one example where this is publicly visible even though the pipelines are not: the new tag says 4.0.0 rabbitmq/rabbitmq-packaging@bb535b5.

[michaelklishin]

The source archive also uses 4.0.0 as the version:

[michaelklishin]

The source package has a list of revisions which also seems correct:

RabbitMQ Server 4.0.0
rabbitmq_server_release 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
accept 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
amqp10_client 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
amqp10_common 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
amqp_client 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
aten 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
base64url 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
cowboy 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
cowlib 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
credentials_obfuscation 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
csv 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
cuttlefish 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
eetcd 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
elvis_mk deff47a4bc32a095c2b8c34f6d82cde77d559b94 3.2.5
enough 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
gen_batch_server 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
getopt 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
gun 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
horus 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
jose 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
json 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
khepri 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
khepri_mnesia_migration 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
oauth2_client 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
observer_cli 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
osiris 3376967204014c6cab7d5248056324c53b5bac98 v1.8.3
prometheus 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
quantile_estimator 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
ra 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbit 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbit_common 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_amqp1_0 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_auth_backend_cache 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_auth_backend_http 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_auth_backend_ldap 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_auth_backend_oauth2 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_auth_mechanism_ssl 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_aws 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_cli 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_codegen 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_consistent_hash_exchange 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_event_exchange 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_federation 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_federation_management 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_federation_prometheus 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_jms_topic_exchange 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_management 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_management_agent 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_mqtt 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_peer_discovery_aws 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_peer_discovery_common 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_peer_discovery_consul 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_peer_discovery_etcd 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_peer_discovery_k8s 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_prelaunch 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_prometheus 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_random_exchange 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_recent_history_exchange 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_sharding 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_shovel 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_shovel_management 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_shovel_prometheus 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_stomp 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_stream 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_stream_common 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_stream_management 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_top 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_tracing 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_trust_store 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_web_dispatch 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_web_mqtt 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_web_mqtt_examples 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_web_stomp 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
rabbitmq_web_stomp_examples 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
ranch 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
recon 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
redbug 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
seshat 0e2a1c2f7c6b208372ecacdbdad592f39cb81f32 v0.6.1
stdout_formatter 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
syslog 67704aec9e7ee2ef9c69d20152b09825d7e6a236 4.0.0
sysmon_handler 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
systemd 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 
thoas 7c7b6a529ac94072a1dde6db9f5bbb2789420bbe 

Many core server parts correctly point to 7c7b6a5.

[michaelklishin]

I have some preliminary findings and a new version, 4.0.1.

In the generic binary build, the problem is still present but it looks less confusing.

The version is set to 4.0.0+2 instead of 4.0.1. For 4.0.0 it was set to 4.0.0-rc.2 so the problem looks like an off-by-one in a list of tags on the v4.0.x branch.

After inspecting our pipelines and build logs for some 90 minutes, our (relatively) recent Makefile refactoring. So it can take some time to track down.

Tanzu RabbitMQ (includings it generic binary build), OSS RabbitMQ Debian, RPM, Windows installer packages are not affected, which is why I further suspect a certain portion of the build system.

In any case, this is not a functional issue but it can affect builds of artefacts such as the community Docker image or the Homebrew bottle.

We will continue investigating tomorrow (Sep 19th) and later this week.

[lhoguin]

I have found where the issue comes from. It's something I did, but not related to the Make changes, rather one of the very few Concourse pipeline changes I did a few months back. Reverting the offending commit should fix the issue.

[lermontex]

I installed the latest version using Docker. It looks like the wrong version was installed or the information displayed is incorrect.

  rabbitmq:
    image: rabbitmq:4-management-alpine
  ...

[michaelklishin]

@lermontex this is exactly what this issue says. The community Docker image (in fact, most Docker images) are produced using the generic binary build.

[michaelklishin]

Another day of investigations suggests that what @lhoguin has found does not seem to be the root cause, so I am back to square one.

What I could confirm is that

gmake -C rabbitmq/packaging package-generic-unix PACKAGES_DIR=/path/to/PACKAGES SOURCE_DIST_FILE=/tmp/build/80754af9/source-archive/rabbitmq-server-4.0.2-alpha.1.tar.xz VERSION=4.0.2-alpha.1 TARBALL_SUFFIX=generic-unix

ignores the explicitly provided version and relies on tags where targets for other package types do not.

[lukebakken]

@michaelklishin I don't have a good way of trying this out or I would do it myself, but could you give this a try?

gmake -C rabbitmq/packaging package-generic-unix PACKAGES_DIR=/path/to/PACKAGES SOURCE_DIST_FILE=/tmp/build/80754af9/source-archive/rabbitmq-server-4.0.2-alpha.1.tar.xz VERSION=4.0.2-alpha.1 PROJECT_VERSION=4.0.2-alpha.1 TARBALL_SUFFIX=generic-unix

[michaelklishin]

@lukebakken see our internal chat. The problem on the Concourse side does exist but not in the form I've expected. The newly pushed tag is not available to the job that builds this package type.

I'm afraid PROJECT_VERSION is not used in the relevant Makefile.

[lukebakken]

Hmm, this didn't work as I expected, anyway. The files in plugins/ still are versioned via git describe --dirty ...

lbakken@PROKOFIEV ~/development/rabbitmq/rabbitmq-server (main =)
$ make VERSION=4.0.0-luke.1 PROJECT_VERSION=4.0.0-luke.1 dist

[michaelklishin]

We have narrowed it down quite a bit in the pipeline but the root cause is still not known. One fundamental solution would be to make gmake package-generic-unix not depend on tags when given an explicit version, just like other package type targets don't depend on tags and are not affected by this issue.

Now let's see what the rest of the team says :)

[michaelklishin]

We should have 4.0.2 published with a fix soon, I will do one extra round of testing before unpausing the publishing jobs.

[michaelklishin]

4.0.2 is out with the correct version in in the generic UNIX package.

[lermontex]

@michaelklishin it seems it didn't work

Generic binary packages used an incorrect version (4.0.0+2 instead of 4.0.1) at build time

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.2

The rabbitmq:4-management-alpine image shows version 4.0.0+2 instead of 4.0.2

[michaelklishin]

@lermontex our team does not maintain that image. Have you read what the README of that image says about its update timeline? You should.

[michaelklishin]

This peculiarity of the generic UNIX package's build job was addressed. If you have something else to report on the topic for 4.0.2, please file a new issue with evidence and a link to the package you are using.

Derived installation methods (such as the Docker image mentioned above, the Homebrew formula/bottle, Bitnami images) will be updated by their respective teams as they see fit or according to their publishing schedules. This can take several days, which is very common for all releases.

[michaelklishin]

docker-library/official-images#17604