Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LIMS-1354: Migrate from ActiveMQ to RabbitMQ #826

Conversation

JPHall-DLS
Copy link
Collaborator

JIRA ticket: LIMS-1354

Summary:
SynchWeb sends messages to ActiveMQ to trigger MX reprocessing and AlphaFold model prediction jobs. A single ActiveMQ server proved unreliable so a RabbitMQ cluster was established in early 2021. An ActiveMQ to RabbitMQ bridge service was implemented as a temporary solution while applications migrated to RabbitMQ. The failure of the bridge service in mid-2024 highlighted the need to update SynchWeb to send messages direct to RabbitMQ.
SynchWeb uses the Stomp (Simple Text Oriented Message Protocol) PHP client to send messages to ActiveMQ. This ticket updates SynchWeb to use an AMQP (Advanced Message Queuing Protocol) client to send messages to RabbitMQ. The php-amqplib client is the de facto standard and detailed in the RabbitMQ PHP tutorial.

Changes:

  • Update composer.json to replace stomp-php dependency with php-amqplib.
  • Add global variables for RabbitMQ configuration that reflect RabbitMQ nomenclature.
  • Update Queue class and Queue::send method to use AMQPStreamConnection and AMQPMessage. Remove redundant ActiveMQ header information e.g. synchweb.host and synchweb.user.
  • Update Process class to use new global variable names.
  • Update Page class to use new global variable names, Queue class signature, and Queue::send method signature.

To test:

  • Reprocess MX data via the gearwheel icon on the Data Collection page. In the database, SynchWeb will create a new record in the ProcessingJob table. SynchWeb will send a message to RabbitMQ with this ProcessingJobId. Zocalo will read the message from RabbitMQ and create a corresponding record in the AutoProcProgram table. A new processing job will soon be shown under Auto Processing on the Data Collection page.
  • AlphaFold model prediction jobs are triggered when proteins are added or updated with a sequence. Model prediction does not currently work as AlphaFold itself was compressed in the great GPFS02 purge of 2024.

@JPHall-DLS JPHall-DLS marked this pull request as ready for review September 5, 2024 19:28
api/src/Queue.php Outdated Show resolved Hide resolved
@ndg63276 ndg63276 changed the base branch from master to pre-release/2024-R4.4 October 1, 2024 09:52
@ndg63276 ndg63276 merged commit ca7badb into pre-release/2024-R4.4 Oct 1, 2024
2 checks passed
ndg63276 added a commit that referenced this pull request Oct 22, 2024
* LIMS-1003: Fix searching of container inspections (#825)

Co-authored-by: Mark Williams <[email protected]>

* LIMS-380: Fix position of autoPROC images (#831)

* LIMS-380: Fix position of autoPROC images

* LIMS-380: Just use iframe size of 98%

* Set width/height to percentage value (#838)

---------

Co-authored-by: Mark Williams <[email protected]>
Co-authored-by: Guilherme Francisco <[email protected]>

* LIMS-1430: Fix bugs on prepare for data collection page (#829)

* LIMS-1430: Allow queuing even if no visible samples

* LIMS-1430: Show number of samples to queue

* LIMS-1430: Watch all subsamples for changes

* LIMS-1430: Remove TODO list

* LIMS-1430: Display number is of data collections, not samples

* LIMS-1430: Dont allow queueing if hidden data collections are invalid

* Update client/src/js/modules/imaging/views/queuecontainer.js

Co-authored-by: Guilherme Francisco <[email protected]>

---------

Co-authored-by: Mark Williams <[email protected]>
Co-authored-by: Guilherme Francisco <[email protected]>

* LIMS-1390: Fix searching of data collection groups (#833)

Co-authored-by: Mark Williams <[email protected]>

* LIMS-1286: Show ERA status on calendar and visits list (#834)

Co-authored-by: Mark Williams <[email protected]>

* LIMS-1354: Migrate from ActiveMQ to RabbitMQ (#826)

* LIMS-1354: Migrate from ActiveMQ to RabbitMQ

* Unwrap try...catch

* Add routing_key and vhost

* Add polyfill for BCMath as php-bcmath extension is not installed but required by php-amqplib for AMQP

---------

Co-authored-by: Mark Williams <[email protected]>
Co-authored-by: Guilherme Francisco <[email protected]>
Co-authored-by: James Hall <[email protected]>
@ndg63276 ndg63276 linked an issue Oct 28, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AMPQ vs STOMP for reprocessing
2 participants