diff --git a/guides/scaling.md b/guides/scaling.md index ab49c637..fc7051ea 100644 --- a/guides/scaling.md +++ b/guides/scaling.md @@ -4,13 +4,14 @@ Oban uses PubSub notifications for communication between nodes, like job inserts, pausing queues, resuming queues, and metrics for Web. The default notifier is `Oban.Notifiers.Postgres`, which -sends all messages through the database. Postgres' notifications adds up at scale because each one -requires a separate query. - -If you're clustered, switch to an alternative notifier like `Oban.Notifiers.PG`. That keeps -notifications out of the db, reduces total queries, and allows larger messages. As long as you -have a functional Distributed Erlang cluster, then it’s a single line change to your Oban -config. +sends all messages through the database. This provides transactional consistency, but Postgres' +notifications adds up at scale because each one requires a separate query. + +If you're clustered, consider switching to an alternative notifier like `Oban.Notifiers.PG`. That +keeps notifications out of the db, reduces total queries, and allows larger messages, with the +tradeoff that notifications from within a database transaction may be sent even if the transaction +is rolled back. As long as you have a functional Distributed Erlang cluster, switching is a +single line change to your Oban config. ```diff config :my_app, Oban, diff --git a/lib/oban/notifier.ex b/lib/oban/notifier.ex index 2828a01a..30311111 100644 --- a/lib/oban/notifier.ex +++ b/lib/oban/notifier.ex @@ -13,7 +13,7 @@ defmodule Oban.Notifier do an implementation of the `Oban.Notifier` behaviour. * `Oban.Notifiers.Postgres` — A Postgres notifier that uses `LISTEN/NOTIFY` to broadcast - messages. This is the default. + messages. * `Oban.Notifiers.PG` — A process groups notifier that relies on Distributed Erlang to broadcast messages. diff --git a/lib/oban/notifiers/pg.ex b/lib/oban/notifiers/pg.ex index 0cee3edd..8734647b 100644 --- a/lib/oban/notifiers/pg.ex +++ b/lib/oban/notifiers/pg.ex @@ -1,6 +1,8 @@ defmodule Oban.Notifiers.PG do @moduledoc """ A [PG (Process Groups)][pg] based notifier implementation that runs with Distributed Erlang. + This notifier scales much better than `Oban.Notifiers.Postgres` but lacks its transactional + guarantees. > #### Distributed Erlang {: .info} > diff --git a/lib/oban/notifiers/postgres.ex b/lib/oban/notifiers/postgres.ex index 6c98899c..68a32922 100644 --- a/lib/oban/notifiers/postgres.ex +++ b/lib/oban/notifiers/postgres.ex @@ -22,6 +22,11 @@ if Code.ensure_loaded?(Postgrex) do The notifications system is built on PostgreSQL's `LISTEN/NOTIFY` functionality. Notifications are only delivered **after a transaction completes** and are de-duplicated before publishing. + This means that notifications sent during a transaction will not be sent if the transaction is + rolled back, providing consistency; this is the only notifer which provides that guarantee. + However, it is not as scalable as other notifiers because because each notification requires a + separate query and notifications can't exceed 8kb. + Typically, applications run Ecto in sandbox mode while testing, but sandbox mode wraps each test in a separate transaction that's rolled back after the test completes. That means the transaction is never committed, which prevents delivering any notifications.