rabbitmq event pusher payload problems after 2037 bytes

[ThomasChiroux]

MongooseIM version: 4.1.0
Installed from: source
Erlang/OTP version: 23.1.5

I've configured mongoose to send amqp message to my rabbitmq server when a user send new message (for internal notificaton solution purposes).

• when the message payload is <= 2037 bytes, all is fine
• when the message payload is > 2037 bytes, the payload gets "shifted", so then the json decoding of the message is impossible.

Here is below a working example and non working example.
They are extracted in rabbitmq directly in the queue, using admin command:

rabbitmqadmin get queue=zamita.Q.prod.tasks.mongoose.chat ackmode=ack_requeue_false

working example, message at exactly 2037 bytes:

+--------------------------------------+---------------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+
|             routing_key              |         exchange          | message_count |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       payload                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | payload_bytes | payload_encoding | properties | redelivered |
+--------------------------------------+---------------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+
| [email protected]_msg_sent | zamita.E.topic.prod.tasks | 0             | {"to_user_id":"c02a3dd5142911e98a400242ac170002@mongoose","message":"<p>Qu'est-ce que le Lorem Ipsum? Le Lorem Ipsum est simplement du faux texte employe dans la composition et la mise en page avant impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les annees 1500, quand un imprimeur anonyme assembla ensemble des morceaux de texte pour realiser un livre specimen de polices de texte. Il n'a pas fait que survivre cinq siecles, mais s'est aussi adapte a la bureautique informatique, sans que son contenu n'en soit modifie. Il a ete popularise dans les annees 1960 grace a la vente de feuilles Letraset contenant des passages du Lorem Ipsum, et, plus recemment, par son inclusion dans des applications de mise en page de texte, comme Aldus PageMaker. Pourquoi l'utiliser? On sait depuis longtemps que travailler avec du texte lisible et contenant du sens est source de distractions, et empeche de se concentrer sur la mise en page elle-meme. L'avantage du Lorem Ipsum sur un texte generique comme 'Du texte. Du texte. Du texte.' est qu'il possede une distribution de lettres plus ou moins normale, et en tout cas comparable avec celle du français standard. De nombreuses suites logicielles de mise en page ou editeurs de sites Web ont fait du Lorem Ipsum leur faux texte par defaut, et une recherche pour 'Lorem Ipsum' vous conduira vers de nombreux sites qui n'en sont encore qu'a leur phase de construction. Plusieurs versions sont apparues avec le temps, parfois par accident, souvent intentionnellement (histoire d'y rajouter de petits clins d'oeil, voire des phrases embarassantes). Qu'est-ce que le Lorem Ipsum? Le Lorem Ipsum est simplement du faux texte employe dans la composition et la mise en page avant impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les annees 1500, quand un imprimeur anonyme assembla ensemble des morceaux de texte pour sdfsdfsdfsdf sdfsdfsd dddeeefffggghhhiiijjjkkkll</p>","from_user_id":"5ba214010b061@mongoose/a512a117f1bc60821613-42622-491802"} | 2037          | string           |            | False       |
+--------------------------------------+---------------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+

Non working example with message at 2038 bytes (I only added a 'z' caracter at then end of the same message)

+--------------------------------------+---------------------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+
|             routing_key              |         exchange          | message_count |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        payload                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | payload_bytes | payload_encoding | properties | redelivered |
+--------------------------------------+---------------------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+
| [email protected]_msg_sent | zamita.E.topic.prod.tasks | 0             | "5ba214010b061@mongoose/a512a117f1bc60821613-42622-491802"}{"to_user_id":"c02a3dd5142911e98a400242ac170002@mongoose","message":"<p>Qu'est-ce que le Lorem Ipsum? Le Lorem Ipsum est simplement du faux texte employe dans la composition et la mise en page avant impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les annees 1500, quand un imprimeur anonyme assembla ensemble des morceaux de texte pour realiser un livre specimen de polices de texte. Il n'a pas fait que survivre cinq siecles, mais s'est aussi adapte a la bureautique informatique, sans que son contenu n'en soit modifie. Il a ete popularise dans les annees 1960 grace a la vente de feuilles Letraset contenant des passages du Lorem Ipsum, et, plus recemment, par son inclusion dans des applications de mise en page de texte, comme Aldus PageMaker. Pourquoi l'utiliser? On sait depuis longtemps que travailler avec du texte lisible et contenant du sens est source de distractions, et empeche de se concentrer sur la mise en page elle-meme. L'avantage du Lorem Ipsum sur un texte generique comme 'Du texte. Du texte. Du texte.' est qu'il possede une distribution de lettres plus ou moins normale, et en tout cas comparable avec celle du français standard. De nombreuses suites logicielles de mise en page ou editeurs de sites Web ont fait du Lorem Ipsum leur faux texte par defaut, et une recherche pour 'Lorem Ipsum' vous conduira vers de nombreux sites qui n'en sont encore qu'a leur phase de construction. Plusieurs versions sont apparues avec le temps, parfois par accident, souvent intentionnellement (histoire d'y rajouter de petits clins d'oeil, voire des phrases embarassantes). Qu'est-ce que le Lorem Ipsum? Le Lorem Ipsum est simplement du faux texte employe dans la composition et la mise en page avant impression. Le Lorem Ipsum est le faux texte standard de l'imprimerie depuis les annees 1500, quand un imprimeur anonyme assembla ensemble des morceaux de texte pour sdfsdfsdfsdf sdfsdfsd dddeeefffggghhhiiijjjkkkllz</p>","from_user_id": | 2038          | string           |            | False       |
+--------------------------------------+---------------------------+---------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------+------------------+------------+-------------+

you can see that the resulting payload is "shifted":

a part that should be at the end of the payload:

"5ba214010b061@mongoose/a512a117f1bc60821613-42622-491802"}

is placed at the beginning of the payload. The resulting payload is not json-decodage anymore.

in addition: (In my case i do not need payloads (I ignore them, but I still need to decode json to get to and from parameters.
not regarding this bug, it could be a good bandwith saving configuration option to be able to remove the payload for such amqp message like "backend.rabbit.chat_msg_exchange.payload = false" (with true by default, same for groupchat))

Thanks,

Regards,
Thomas.

PS
the mongoose config:

[general]
all_metrics_are_global = true
hosts = ["mongoose"]
language = "en"
loglevel = "all"
max_fsm_queue = 1000
registration_timeout = 600
replaced_wait_timeout = 5000
hide_service_name = true
sm_backend = "mnesia"

# [[listen.c2s]]
# access = "c2s"
# max_stanza_size = 262144
# port = 5222
# shaper = "c2s_shaper"

[[listen.http]]
port = 5480
transport.max_connections = 1024
transport.num_acceptors = 10

[[listen.http.handlers.mod_websockets]]
host = "_"
path = "/ws"
timeout = 120_000
ping_rate = 30_000

[listen.http.handlers.mod_websockets.service]
access = "all"
password = "secret"
shaper_rule = "fast"

[[listen.http]]
port = 5488
transport.max_connections = 1024
transport.num_acceptors = 10

[[listen.http.handlers.lasse_handler]]
host = "_"
module = "mongoose_client_api_sse"
path = "/api/sse"

[[listen.http.handlers.mongoose_api_admin]]
host = "_"
path = "/api"
# username = "admin"
# password = "admin"


# [[listen.http.handlers.mongoose_client_api_messages]]
# host = "_"
# path = "/api/messages/[:with]"

# [[listen.http.handlers.mongoose_client_api_contacts]]
# host = "_"
# path = "/api/contacts/[:jid]"

# [[listen.http.handlers.mongoose_client_api_rooms]]
# host = "_"
# path = "/api/rooms/[:id]"

# [[listen.http.handlers.mongoose_client_api_rooms_config]]
# host = "_"
# path = "/api/rooms/[:id]/config"

# [[listen.http.handlers.mongoose_client_api_rooms_users]]
# host = "_"
# path = "/api/rooms/:id/users/[:user]"

# [[listen.http.handlers.mongoose_client_api_rooms_messages]]
# host = "_"
# path = "/api/rooms/[:id]/messages"

# [[listen.http.handlers.cowboy_swagger_redirect_handler]]
# host = "_"
# path = "/api-docs"

# [[listen.http.handlers.cowboy_swagger_json_handler]]
# host = "_"
# path = "/api-docs/swagger.json"

# [[listen.http.handlers.cowboy_static]]
# app = "cowboy_swagger"
# content_path = "swagger"
# host = "_"
# path = "/api-docs/[...]"
# type = "priv_dir"

[auth]
methods = ["jwt"]

[auth.jwt]
secret.value = "SECRET"
algorithm = "HS256"
username_key = "user"

[outgoing_pools.rdbms.default]
scope = "global"
workers = 10

[outgoing_pools.rdbms.default.connection]
driver = "mysql"
host = "db_mongoose"
database = "mongoose"
username = "mongoose"
password = "mongoose"

[outgoing_pools.rabbit.event_pusher]
scope = "global"
workers = 5

[outgoing_pools.rabbit.event_pusher.connection]
amqp_host = "rabbitmq"
amqp_port = 5672
amqp_username = "test"
amqp_password = "test"
confirms_enabled = false
max_worker_queue_len = 1000

[services.service_admin_extra]
submods = [
  "node",
  "accounts",
  "sessions",
  "vcard",
  "gdpr",
  "upload",
  "roster",
  "last",
  "private",
  "stanza",
  "stats",
]

# [services.service_mongoose_system_metrics]
# initial_report = 300_000
# periodic_report = 10_800_000

[modules.mod_adhoc]

[modules.mod_caps]

[modules.mod_carboncopy]

[modules.mod_commands]

# [modules.mod_muc_commands]

# [modules.mod_muc_light_commands]


# client state indication
#[modules.mod_csi]
#  buffer_max = 40

[modules.mod_disco]
users_can_see_hidden_services = false

[modules.mod_event_pusher]
backend.rabbit.presence_exchange.name = "zamita.E.topic.prod.tasks"
backend.rabbit.presence_exchange.type = "topic"
backend.rabbit.chat_msg_exchange.name = "zamita.E.topic.prod.tasks"
backend.rabbit.chat_msg_exchange.type = "topic"
backend.rabbit.chat_msg_exchange.sent_topic = "chat_msg_sent"
backend.rabbit.chat_msg_exchange.recv_topic = "chat_msg_recv"
backend.rabbit.groupchat_msg_exchange.name = "zamita.E.topic.prod.tasks"
backend.rabbit.groupchat_msg_exchange.type = "topic"
backend.rabbit.groupchat_msg_exchange.sent_topic = "groupchat_msg_sent"
backend.rabbit.groupchat_msg_exchange.recv_topic = "groupchat_msg_recv"

[modules.mod_last]
backend = "rdbms"

[modules.mod_mam_meta]
backend = "rdbms"
async_writer = true # this option enables async writer for RDBMS backend
message_retraction = false # for now
rdbms_message_format = "simple" # or "internal"
# user_prefs_store = "rdbms" 
full_text_search = false
max_result_limit = 500 # default: 50

[modules.mod_mam_meta.pm] # enable one2one message archiving
# [modules.mod_mam_meta.muc] # defining this section enables MUC support


# [modules.mod_muc_light]
#   host = "muclight.example.com"
#   equal_occupants = true
#   legacy_mode = true
#   rooms_per_user = 10
#   blocking = false
#   all_can_configure = true
#   all_can_invite = true
#   max_occupants = 50
#   rooms_per_page = 5
#   rooms_in_rosters = true

#   [[modules.mod_muc_light.config_schema]] 
#     field = "roomname"
#     value = "The Room"

#   [[modules.mod_muc_light.config_schema]] 
#     field = "display-lines"
#     value = 30
#     internal_key = "display_lines"
#     type = "integer"

# [modules.mod_muc_light_commands]

[modules.mod_ping]
send_pings = true
ping_interval = 59 # 30
timeout_action = "none"
ping_req_timeout = 32

[modules.mod_privacy]
backend = "rdbms"

[modules.mod_private]
backend = "rdbms"

# [modules.mod_pubsub]
#   access_createnode = "pubsub_createnode"
#   ignore_pep_from_offline = false
#   backend = "rdbms"
#   last_item_cache = "mnesia"
#   max_items_node = 1000
#   plugins = ["flat", "pep"]

#   [[modules.mod_pubsub.pep_mapping]]
#     namespace = "urn:xmpp:microblog:0"
#     node = "mb"

# [modules.mod_register]
# access = "register"
# ip_access = [
#   { address = "127.0.0.0/8", policy = "allow" },
#   { address = "0.0.0.0/0", policy = "deny" },
# ]
# welcome_message = { body = "", subject = "" }

[modules.mod_roster]
backend = "rdbms"
versioning = true
store_current_id = false

[modules.mod_stream_management]
buffer = true
buffer_max = 50
ack = true
ack_freq = 1
resume_timeout = 240 # 600
stale_h.enabled = true
stale_h.repeat_after = 1800
stale_h.geriatric = 3600

[modules.mod_vcard]
backend = "rdbms"

[modules.mod_version]
os_info = false

[shaper.normal]
max_rate = 50000

[shaper.fast]
max_rate = 50000

[shaper.mam_shaper]
max_rate = 1

[shaper.mam_global_shaper]
max_rate = 1000

[acl]
local = [{ user_regexp = "" }]

[access]
max_user_sessions = [{ acl = "all", value = 10 }]

max_user_offline_messages = [{ acl = "admin", value = 5000 }, { acl = "all", value = 100 }]

local = [{ acl = "local", value = "allow" }]

c2s = [{ acl = "blocked", value = "deny" }, { acl = "all", value = "allow" }]

c2s_shaper = [{ acl = "admin", value = "none" }, { acl = "all", value = "normal" }]

s2s_shaper = [{ acl = "all", value = "fast" }]

muc_admin = [{ acl = "admin", value = "allow" }]

muc_create = [{ acl = "local", value = "allow" }]

muc = [{ acl = "all", value = "allow" }]

register = [{ acl = "all", value = "allow" }]

mam_set_prefs = [{ acl = "all", value = "default" }]

mam_get_prefs = [{ acl = "all", value = "default" }]

mam_lookup_messages = [{ acl = "all", value = "default" }]

mam_set_prefs_shaper = [{ acl = "all", value = "mam_shaper" }]

mam_get_prefs_shaper = [{ acl = "all", value = "mam_shaper" }]

mam_lookup_messages_shaper = [{ acl = "all", value = "mam_shaper" }]

mam_set_prefs_global_shaper = [{ acl = "all", value = "mam_global_shaper" }]

mam_get_prefs_global_shaper = [{ acl = "all", value = "mam_global_shaper" }]

mam_lookup_messages_global_shaper = [{ acl = "all", value = "mam_global_shaper" }]

[[host_config]]
host = "mongoose"

[ThomasChiroux]

complement analysis:

Below are assumptions, i've not tested the hypothesis and i'm not - at all - an Erlang expert ;-)

This may be linked to the way jiffy encodes json, here:

MongooseIM/src/event_pusher/mod_event_pusher_rabbit.erl

Line 230 in 6171e56

jiffy:encode(Msg).

It seems that if the message reach a certain size, jiffy returns a list instead of bytes:
davisp/jiffy#183

My asymption is that this list is later sent / aggregated in the wrong order..