feat(store-sync): add support for live sync from indexer #3226

alvrs · 2024-09-23T21:07:03Z

No description provided.

changeset-bot · 2024-09-23T21:07:07Z

🦋 Changeset detected

Latest commit: a12cb15

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 26 packages

Name	Type
@latticexyz/store-sync	Patch
@latticexyz/dev-tools	Patch
@latticexyz/explorer	Patch
@latticexyz/store-indexer	Patch
ts-benchmarks	Patch
@latticexyz/abi-ts	Patch
@latticexyz/block-logs-stream	Patch
@latticexyz/cli	Patch
@latticexyz/common	Patch
@latticexyz/config	Patch
create-mud	Patch
@latticexyz/faucet	Patch
@latticexyz/gas-report	Patch
@latticexyz/protocol-parser	Patch
@latticexyz/react	Patch
@latticexyz/recs	Patch
@latticexyz/schema-type	Patch
solhint-config-mud	Patch
solhint-plugin-mud	Patch
@latticexyz/stash	Patch
@latticexyz/store	Patch
@latticexyz/utils	Patch
@latticexyz/world-module-metadata	Patch
@latticexyz/world-modules	Patch
@latticexyz/world	Patch
mock-game-contracts	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

packages/store-sync/src/createStoreSync.ts

vercel · 2024-09-30T20:15:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
explorer	❌ Failed (Inspect)			Sep 30, 2024 8:56pm

packages/store-sync/src/createStoreSync.ts

holic · 2024-10-02T09:28:32Z

packages/store-sync/src/createStoreSync.ts

+      debug("error streaming logs from indexer:", e);
+      debug("falling back to streaming logs from RPC");
+      return storedRpcLogs$;
+    }),


have we tested the fallback behavior here? any way to make sure e2e tests cover this case like they do for the previous indexers?

The fallback behavior already kicks in if no indexer is provided or the existing indexer doesn't support this API, so in that sense it's covered by the existing e2e tests

holic · 2024-10-02T10:48:34Z

packages/store-sync/src/fromEventSource.ts

+  return new Observable<MessageEvent>((subscriber) => {
+    const eventSource = new EventSource(url);
+    eventSource.onmessage = (ev): void => subscriber.next(ev);
+    eventSource.onerror = (): void => subscriber.error(new Error("Event source closed: " + url));


I am pretty sure EventSource attempts to reconnect during an error (I think indefinitely) but unclear if it emits onerror during this reconnect flow. We might want to give it a bad URL and see if this emits a bunch of onerror and maybe allow a few reconnection attempts before giving up and closing the stream with subscriber.error?

Just tried that, it does indeed try to reconnect, but it seems like once it reconnects it skips the blocks that happened while the connection was down and just continues with the current block. I think we can solve this on the indexer side by using the block number as event id and also add code to handle reconnections with last_event_id, but would leave that as an optimization for later.

yeah exactly, we need to update the indexer SSE stream to include an id and use the Last-Event-ID header for reconnections

ID would likely need to be block number + log index

holic · 2024-10-02T11:06:37Z

packages/store-sync/src/isStorageAdapterBlock.ts

should this be in store-sync/src/sql? because its specific to sql api?

and could we have a corresponding toStorageAdapterBlock that contains the blockNumber conversion so we don't have it in a few places?

it's not specific to the sql api, it's the /logs api

could rename to isIndexerStorageAdapterBlock or isIndexerLogsResponse or something 🤷‍♂️

either way 🤷 just something to differentiate from the StorageAdapterBlock type (which already uses bigint not string for block number)

renamed to isLogsApiResponse

holic · 2024-10-02T11:41:17Z

packages/store-sync/src/fromEventSource.ts

+  return new Observable<MessageEvent>((subscriber) => {
+    const eventSource = new EventSource(url);
+    eventSource.onmessage = (ev): void => subscriber.next(ev);
+    eventSource.onerror = (): void => subscriber.error(new Error("Event source failed" + new URL(url).origin));


Suggested change

eventSource.onerror = (): void => subscriber.error(new Error("Event source failed" + new URL(url).origin));

// we immediately close instead of allowing `EventSource` to retry because the logs API doesn't support `Last-Event-ID` yet

eventSource.onerror = (): void => subscriber.error(new Error("Event source failed" + new URL(url).origin));

Since fromEventSource is technically not specific to the logs API, we should probably add an option to fromEventSource to enable/disable retries, so consumers can decide based on the API capabilities. Would leave for a followup

holic · 2024-10-02T12:48:46Z

packages/store-sync/src/createStoreSync.ts

@@ -61,7 +64,7 @@ export async function createStoreSync({
  maxBlockRange,
  initialState,
  initialBlockLogs,
-  indexerUrl,
+  indexerUrl: indexerUrlInput,


small nit: I tend to name these "initial"

Suggested change

indexerUrl: indexerUrlInput,

indexerUrl: initialIndexerUrl,

holic · 2024-10-02T12:52:28Z

packages/store-sync/src/createStoreSync.ts

+          if (!isLogsApiResponse(data)) {
+            throw new Error("Received unexpected from indexer:" + messageEvent.data);
+          }
+          return { ...data, blockNumber: BigInt(data.blockNumber) };


not blocking but might be nice to move these lines into a toStorageAdapterBlock since we have this logic in a couple spots

arktype can help a lot with this pattern of parsing+validating+strong types but can save that for a later improvement

mud/packages/world/ts/node/loadSystemsManifest.ts

Line 9 in 111bb1b

const parseManifest = type("string").pipe.try((s) => JSON.parse(s), SystemsManifest);

mud/packages/world/ts/node/buildSystemsManifest.ts

Lines 13 to 30 in 111bb1b

export const SystemsManifest = type({

systems: [

{

// labels

namespaceLabel: "string",

label: "string",

// resource ID

namespace: "string",

name: "string",

systemId: ["string", ":", (s): s is Hex => isHex(s, { strict: false })],

// abi

abi: "string[]",

worldAbi: "string[]",

},

"[]",

],

createdAt: "number",

});

alvrs added 6 commits September 23, 2024 13:31

export logToRecord

3ff11bf

wip: indexer live sync

2306fb7

add storage adapter

4c1167f

extract fetchAndFilter logs

f1808fd

eventSource -> fromEventSource

581afbc

remove dozer workaround

6898ae1

vercel bot deployed to Preview September 23, 2024 21:07 View deployment

alvrs added 2 commits September 23, 2024 23:09

merge

94ebfc5

Create great-dragons-sit.md

8880207

alvrs marked this pull request as ready for review September 23, 2024 21:11

alvrs requested a review from holic as a code owner September 23, 2024 21:11

vercel bot deployed to Preview September 23, 2024 21:14 View deployment

holic reviewed Sep 24, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Outdated Show resolved Hide resolved

holic reviewed Sep 24, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Show resolved Hide resolved

holic reviewed Sep 24, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Outdated Show resolved Hide resolved

holic reviewed Sep 24, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Outdated Show resolved Hide resolved

wait for storage adapter before fetching next block range

f0472cd

vercel bot had a problem deploying to Preview – explorer September 30, 2024 20:15 Failure

vercel bot deployed to Preview – mud-docs September 30, 2024 20:17 View deployment

self-review

11be850

vercel bot deployed to Preview – mud-docs September 30, 2024 20:22 View deployment

vercel bot had a problem deploying to Preview – explorer September 30, 2024 20:22 Failure

update url construction

11e9799

vercel bot had a problem deploying to Preview – explorer September 30, 2024 20:56 Failure

vercel bot deployed to Preview – mud-docs September 30, 2024 20:58 View deployment

holic reviewed Oct 2, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Show resolved Hide resolved

holic reviewed Oct 2, 2024

View reviewed changes

packages/store-sync/src/createStoreSync.ts Outdated Show resolved Hide resolved

holic reviewed Oct 2, 2024

View reviewed changes

vercel bot deployed to Preview – mud-docs October 2, 2024 10:44 View deployment

better error on closed event source

cd2ea8c

holic reviewed Oct 2, 2024

View reviewed changes

alvrs added 2 commits October 2, 2024 11:49

change error message

c7abd73

change error message

b7f5744

vercel bot had a problem deploying to Preview – explorer October 2, 2024 10:50 Failure

vercel bot deployed to Preview – mud-docs October 2, 2024 10:52 View deployment

verify indexer return data

04d7355

vercel bot had a problem deploying to Preview – explorer October 2, 2024 10:58 Failure

vercel bot deployed to Preview – mud-docs October 2, 2024 11:00 View deployment

holic reviewed Oct 2, 2024

View reviewed changes

change error message

90e3fa3

vercel bot deployed to Preview – mud-docs October 2, 2024 11:27 View deployment

vercel bot had a problem deploying to Preview – explorer October 2, 2024 11:27 Failure

rename isStorageAdapterBlock -> isLogsApiResponse

bb795fe

vercel bot had a problem deploying to Preview – explorer October 2, 2024 11:28 Failure

vercel bot deployed to Preview – mud-docs October 2, 2024 11:30 View deployment

Merge branch 'main' into alvrs/indexer-live-sync

d9f509f

vercel bot had a problem deploying to Preview – explorer October 2, 2024 11:39 Failure

holic reviewed Oct 2, 2024

View reviewed changes

vercel bot deployed to Preview – mud-docs October 2, 2024 11:42 View deployment

alvrs mentioned this pull request Oct 2, 2024

Add option to reconnect in fromEventSource #3246

Open

holic previously approved these changes Oct 2, 2024

View reviewed changes

alvrs added 2 commits October 3, 2024 12:16

indexerUrlInput -> initialIndexerUrl

c835909

add util to turn logs api response into storage adapter block

a12cb15

alvrs dismissed holic’s stale review via a12cb15 October 3, 2024 11:17

vercel bot had a problem deploying to Preview – explorer October 3, 2024 11:17 Failure

vercel bot deployed to Preview – mud-docs October 3, 2024 11:19 View deployment

alvrs merged commit 61930ee into main Oct 3, 2024
12 of 13 checks passed

alvrs deleted the alvrs/indexer-live-sync branch October 3, 2024 11:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(store-sync): add support for live sync from indexer #3226

feat(store-sync): add support for live sync from indexer #3226

alvrs commented Sep 23, 2024

changeset-bot bot commented Sep 23, 2024 •

edited

Loading

vercel bot commented Sep 30, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading

alvrs Oct 2, 2024

holic Oct 2, 2024

alvrs Oct 2, 2024

holic Oct 2, 2024

holic Oct 2, 2024 •

edited

Loading

holic Oct 2, 2024

holic Oct 2, 2024 •

edited

Loading

alvrs Oct 2, 2024

alvrs Oct 2, 2024

holic Oct 2, 2024 •

edited

Loading

alvrs Oct 2, 2024

holic Oct 2, 2024

alvrs Oct 2, 2024

alvrs Oct 2, 2024

holic Oct 2, 2024

holic Oct 2, 2024 •

edited

Loading

	eventSource.onerror = (): void => subscriber.error(new Error("Event source failed" + new URL(url).origin));
	// we immediately close instead of allowing `EventSource` to retry because the logs API doesn't support `Last-Event-ID` yet
	eventSource.onerror = (): void => subscriber.error(new Error("Event source failed" + new URL(url).origin));

	export const SystemsManifest = type({
	systems: [
	{
	// labels
	namespaceLabel: "string",
	label: "string",
	// resource ID
	namespace: "string",
	name: "string",
	systemId: ["string", ":", (s): s is Hex => isHex(s, { strict: false })],
	// abi
	abi: "string[]",
	worldAbi: "string[]",
	},
	"[]",
	],
	createdAt: "number",
	});

feat(store-sync): add support for live sync from indexer #3226

feat(store-sync): add support for live sync from indexer #3226

Conversation

alvrs commented Sep 23, 2024

changeset-bot bot commented Sep 23, 2024 • edited Loading

🦋 Changeset detected

vercel bot commented Sep 30, 2024 • edited Loading

holic Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

changeset-bot bot commented Sep 23, 2024 •

edited

Loading

vercel bot commented Sep 30, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading

holic Oct 2, 2024 •

edited

Loading