Code

Parallel synchronization#78

Code

Merged

jeromegn merged 98 commits intomainsuperfly/corrosion:mainfrom sync-mem-tweakssuperfly/corrosion:sync-mem-tweaksCopy head branch name to clipboardOct 19, 2023

+2,736-963

Commits

Commits on Oct 6, 2023

• smaller channel for changes, using too much memory right now

  jeromegn

  committed
  c5ab6dc

Commits on Oct 8, 2023

• track QUIC connection metrics, in aggregate

  jeromegn

  committed
  47b5c30

• easier to query these metrics if they have the same name but different labels

  jeromegn

  committed
  a61664e

• higher limit of max unidirectional QUIC streams

  jeromegn

  committed
  4320746

• limit number of concurrent syncs to 3

  jeromegn

  committed
  f5d7220

• pool unidirectional streams so we don't create nearly as many

  jeromegn

  committed
  34ee8e8

• don't pool unidirectional send streams

  jeromegn

  committed
  62329e1

• finish unidirectional streams when sending data

  jeromegn

  committed
  c247b3c

Commits on Oct 9, 2023

• emit tokio runtime metrics

  jeromegn

  committed
  53fd61b

• track cwdn per addr

  jeromegn

  committed
  6a56f11

• track congestion events and blackhols per addr

  jeromegn

  committed
  efddae3

Commits on Oct 10, 2023

• retry opening a unidirectional stream once if it failed when trying to send a unidirectional payload

  jeromegn

  committed
  a5747c6

• split booked versions in their state (cleared, current, partial)

  jeromegn

  committed
  5f49d56

• first part of syncing more efficiently between nodes

  jeromegn

  committed
  06bf45f

• fixed sync mistake

  jeromegn

  committed
  cabceda

• move comment

  jeromegn

  committed
  76501e2

Commits on Oct 12, 2023

• wip parallel sync

  jeromegn

  committed
  d081764

• hopefully massive improvements on sync speed, especially from scratch

  jeromegn

  committed
  f0bb3ee

• patch upgrade quinn-proto to use less memory

  jeromegn

  committed
  ed895a6

• observe member syncing

  jeromegn

  committed
  c173a7c

Commits on Oct 13, 2023

• something is weird here, syncing should be a lot faster

  jeromegn

  committed
  59300e5

• hopefully fix tests, add cluster inspection commands

  jeromegn

  committed
  37a7a54

• measure how many sent sync requests

  jeromegn

  committed
  52762b1

• send more often

  jeromegn

  committed
  3438e90

• process completed empties in much smaller batches

  jeromegn

  committed
  359939e

• log a bit more

  jeromegn

  committed
  49ae55d

• send much smaller chunks

  jeromegn

  committed
  e282a72

• possibly faster cleared empties processing

  jeromegn

  committed
  9a714bb

• need to acquire an async read lock here

  jeromegn

  committed
  960afd0

• don't timeout on receive

  jeromegn

  committed
  efbf224

• send sync requests in bigger chunks

  jeromegn

  committed
  88396ae

• better metrics for changes received and lookout for the send stream stopping

  jeromegn

  committed
  1fd747a

• try more servers

  jeromegn

  committed
  27a4695

• try to aggregate on the server-side too

  jeromegn

  committed
  4f7f2f6

• process smaller chunks

  jeromegn

  committed
  4292492

• reduce number of locks being acquired

  jeromegn

  committed
  da662cd

• process each chunk of sync requests concurrently

  jeromegn

  committed
  6c5db98

• bring back chunking for performance reasons

  jeromegn

  committed
  52554ed

• different concurrency attempt

  jeromegn

  committed
  c2c95ca

• properly break out of loop

  jeromegn

  committed
  2a013cf

Commits on Oct 14, 2023

• don't break until we've read everything

  jeromegn

  committed
  b8a368f

• send success to the response ends

  jeromegn

  committed
  9a2504e

• log which actor_id returned an error

  jeromegn

  committed
  f35501a

• allow a few more changes in the channels

  jeromegn

  committed
  cf84a44

• send smaller chunks

  jeromegn

  committed
  8ad87e7

• try to block less when processing empties

  jeromegn

  committed
  cefbf8a

• better qualify lock name for booked writer when processing multiple versions, don't process empties constantly

  jeromegn

  committed
  23cfda8

• let's try applying more changes now that empties are out of the way

  jeromegn

  committed
  1d8f2db

• don't count empties when determining when to process multiple changes

  jeromegn

  committed
  a379ae5

• log how many changes were inserted and in how much time

  jeromegn

  committed
  6921c0b

Commits on Oct 15, 2023

• when clearing detected-cleared db versions, do it lazily as part of the normal process, bigger batches for inserts, measure more times

  jeromegn

  committed
  cb411b4

• manually declare actor a down if its timestamp is lower than previously recorded

  jeromegn

  committed
  dd2b81c

• use duration instead of value of u64 or timestamp

  jeromegn

  committed
  a77deb5

• log less

  jeromegn

  committed
  2bc13c1

Commits on Oct 16, 2023

• attempt to add open-telemetry tracing

  jeromegn

  committed
  59477ec

• instrument a ton more

  jeromegn

  committed
  9e264fd

• unclog connection locks

  jeromegn

  committed
  b42f4b9

• reduce lock times

  jeromegn

  committed
  e7de255

• don't include insane span list in JSON

  jeromegn

  committed
  2e246e4

• don't have to clone a full connection, just get its stats

  jeromegn

  committed
  266b708

• reduce level of read/write bookie locks so we don't spam otel

  jeromegn

  committed
  a03f841

• still trying to fix otel

  jeromegn

  committed
  9e2a783

• set send_datagram to debug log level

  jeromegn

  committed
  050412b

• maybe this helps

  jeromegn

  committed
  98d248f

• make a parent span somehow to better track process_multiple_changes

  jeromegn

  committed
  29f3f71

Commits on Oct 17, 2023

• remove instrumentation to see if it helps

  jeromegn

  committed
  0102bb7

• using a span across awaits can lead to weird stuff

  jeromegn

  committed
  c56b57d

• don't know why but this needs to be an async rwlock

  jeromegn

  committed
  d51ac81

• more instrumentation fiddling

  jeromegn

  committed
  50bfbcb

• more instrumentation, including result errors and such

  jeromegn

  committed
  bc86463

• remove span across awaits

  jeromegn

  committed
  a2d8f0e

• propagate trace context when starting sync

  jeromegn

  committed
  2a06e9a

• more instrumentation for sync process, both ways

  jeromegn

  committed
  05e6cd5

• less instrumentation for stuff that happens too often

  jeromegn

  committed
  bf330ee

• try to get rid of more traces

  jeromegn

  committed
  054e241

• log needs len

  jeromegn

  committed
  01875ac

• don't block when sending into empties channel

  jeromegn

  committed
  15cbe87

• seqs start at 0

  jeromegn

  committed
  79091b8

Commits on Oct 18, 2023

• add a timeout to quic connect

  jeromegn

  committed
  c19d2f2

• warn if sending too large datagram, log its size

  jeromegn

  committed
  7da737a

• also log max datagram size

  jeromegn

  committed
  966aa23

• set a smaller max_packet_size for foca, appears we are limited by path mtu

  jeromegn

  committed
  66fd461

• don't require locking the bookie to get at the registry

  jeromegn

  committed
  2de4bf4

• fix a deadlock

  jeromegn

  committed
  fb2248e

• allow far more unidirectional streams

  jeromegn

  committed
  a4b38c9

• try to simplify reading from unidirectional stream

  jeromegn

  committed
  e8f631b

• use multiple endpoints to parallelize sending data

  jeromegn

  committed
  35d0e19

• use seahash instead of manually modulo-ing to figure out endpoint index

  jeromegn

  committed
  3abf6fc

• restore corro_peer_stream_bytes_recv_total metric

  jeromegn

  committed
  08d5493

• reduce number of max concurrent uni streams

  jeromegn

  committed
  db1bca1

• record gossip recv lag

  jeromegn

  committed
  94adf28

Commits on Oct 19, 2023

• don't send empty trace states

  jeromegn

  committed
  444c017

• more prometheus histogram buckets for more precision

  jeromegn

  committed
  143ad89

• fix panic, somehow it received a 0-value head version in the sync generated from another actor, added a log to investigate next time it happens

  jeromegn

  committed
  f94b0fa

• update CHANGELOG

  jeromegn

  committed
  04586cc

• upgrade mdbook-admonish

  jeromegn

  committed
  c667075

• probably fix rustflags on CI

  jeromegn

  committed
  ba4b29e

• try sccache instead fo rust-cache

  jeromegn

  committed
  4ca321f