Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Subgraph stuck forever after (falsly?) detecting a reorg #5586

Open
1 of 3 tasks
paulperegud opened this issue Aug 6, 2024 · 1 comment
Open
1 of 3 tasks

[Bug] Subgraph stuck forever after (falsly?) detecting a reorg #5586

paulperegud opened this issue Aug 6, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@paulperegud
Copy link

Bug report

This bug report is related to #5101, but involves a at least one important difference - no reorg!

Multiple copies of the same subgraph (against different deployments of the same set of contracts) are deployed to a node running locally. Anvil is providing RPC and it is not instructed to perform (nor it performs to the best of my knowledge) any reorgs. At some point, graph-node detects the reorg. Whole setup is running inside a docker-compose.

This bug is rather repeatable. More subgraphs is deployed, the higher is the chance of seeing it. More load on the machine leads to more frequent occurances.

Log below was redacted to be of reasonable size (original had 2.5mln of lines). I've removed lines mentioning all but two subgraphs, sections where logs were repeated in a loop, messages with following substrings:

  • Committed write batch
  • Done processing trigger
  • Contract call reverted
  • Call to getCurrentEpoch

I've observed this bug with both v0.34.1 and v0.35.1.

Relevant log output

Aug 05 12:03:56.060 INFO Graph Node version: 0.34.1 (2024-02-08)
Aug 05 12:03:56.060 WARN GRAPH_POI_ACCESS_TOKEN not set; might leak POIs to the public via GraphQL
Aug 05 12:03:56.060 INFO Generating configuration from command line arguments
Aug 05 12:03:56.060 WARN No fork base URL specified, subgraph forking is disabled
Aug 05 12:03:56.060 INFO Starting up
Aug 05 12:03:56.060 INFO Trying IPFS node at: http://ipfs:5001/
Aug 05 12:03:56.072 INFO Successfully connected to IPFS node at: http://ipfs:5001/
Aug 05 12:03:56.075 INFO Creating transport, capabilities: archive, traces, url: http://anvil:8545, provider: localhost-rpc-0
Aug 05 12:03:56.080 WARN Expensive queries file not set to a valid file: /etc/graph-node/expensive-queries.txt
Aug 05 12:03:56.093 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://graph-node:HIDDEN_PASSWORD@graph-postgres:5432/graph-node?sslmode=prefer, pool: main, shard: primary
Aug 05 12:03:56.094 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
Aug 05 12:03:56.098 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
Aug 05 12:03:56.105 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
Aug 05 12:03:56.454 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
Aug 05 12:03:56.455 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
Aug 05 12:03:56.459 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
Aug 05 12:03:56.479 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: localhost-rpc-0
Aug 05 12:03:56.480 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 1337, provider: localhost-rpc-0
Aug 05 12:03:56.497 INFO Creating LoadManager in disabled mode, component: LoadManager
Aug 05 12:03:56.497 INFO Started block ingestor, network_name: localhost
Aug 05 12:03:56.497 INFO Starting block ingestor for network, network_name: localhost
Aug 05 12:03:56.497 INFO Starting job runner with 5 jobs, component: JobRunner
Aug 05 12:03:56.497 INFO Starting JSON-RPC admin server at: http://localhost:8020, component: JsonRpcServer
Aug 05 12:03:56.497 INFO Starting GraphQL HTTP server at: http://localhost:8000, component: GraphQLServer
Aug 05 12:03:56.497 INFO Started all assigned subgraphs, node_id: default, count: 0, component: SubgraphRegistrar
Aug 05 12:03:56.498 INFO Starting index node server at: http://localhost:8030, component: IndexNodeServer
Aug 05 12:03:56.498 INFO Starting metrics server at: http://localhost:8040, component: MetricsServer
Aug 05 12:03:56.498 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
Aug 05 12:03:56.498 INFO Downloading latest blocks from Ethereum, this may take a few minutes..., provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:04:30.589 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 19, current_block_head: 0, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:04:40.412 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_ntQyCN") }, component: JsonRpcServer
Aug 05 12:04:51.703 INFO Syncing 9 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 9, blocks_behind: 9, latest_block_head: 28, current_block_head: 19, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:04:54.727 INFO Syncing 10 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 10, blocks_behind: 10, latest_block_head: 38, current_block_head: 28, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:04:55.760 INFO Syncing 9 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 9, blocks_behind: 9, latest_block_head: 47, current_block_head: 38, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:05:05.512 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_qWI4L4") }, component: JsonRpcServer
Aug 05 12:05:16.830 INFO Syncing 7 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 7, blocks_behind: 7, latest_block_head: 54, current_block_head: 47, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:05:20.867 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 73, current_block_head: 54, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:05:31.934 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_m84HKa") }, component: JsonRpcServer
Aug 05 12:05:52.006 INFO Syncing 4 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 4, blocks_behind: 4, latest_block_head: 77, current_block_head: 73, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:05:53.029 INFO Syncing 15 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 15, blocks_behind: 15, latest_block_head: 92, current_block_head: 77, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:03.970 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_x7hO46") }, component: JsonRpcServer
Aug 05 12:06:16.131 INFO Syncing 7 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 7, blocks_behind: 7, latest_block_head: 99, current_block_head: 92, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:17.162 INFO Syncing 2 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 2, blocks_behind: 2, latest_block_head: 101, current_block_head: 99, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:18.177 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 102, current_block_head: 101, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:22.191 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 121, current_block_head: 102, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:34.273 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_0vdwLp") }, component: JsonRpcServer
Aug 05 12:06:45.292 INFO Syncing 5 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 5, blocks_behind: 5, latest_block_head: 126, current_block_head: 121, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:48.310 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 145, current_block_head: 126, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:06:59.163 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_rJsjIb") }, component: JsonRpcServer
Aug 05 12:07:11.408 INFO Syncing 14 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 14, blocks_behind: 14, latest_block_head: 159, current_block_head: 145, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:07:12.467 INFO Syncing 5 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 5, blocks_behind: 5, latest_block_head: 164, current_block_head: 159, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:07:22.062 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_nC7XTw") }, component: JsonRpcServer
Aug 05 12:07:32.524 INFO Syncing 7 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 7, blocks_behind: 7, latest_block_head: 171, current_block_head: 164, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:07:35.549 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 190, current_block_head: 171, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:07:46.876 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_UHlAQ7") }, component: JsonRpcServer
Aug 05 12:07:57.658 INFO Syncing 5 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 5, blocks_behind: 5, latest_block_head: 195, current_block_head: 190, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:00.677 INFO Syncing 10 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 10, blocks_behind: 10, latest_block_head: 205, current_block_head: 195, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:01.708 INFO Syncing 9 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 9, blocks_behind: 9, latest_block_head: 214, current_block_head: 205, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:12.172 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_CH3Fsu") }, component: JsonRpcServer
Aug 05 12:08:21.772 INFO Syncing 3 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 3, blocks_behind: 3, latest_block_head: 217, current_block_head: 214, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:22.786 INFO Syncing 6 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 6, blocks_behind: 6, latest_block_head: 223, current_block_head: 217, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:25.807 INFO Syncing 19 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 19, blocks_behind: 19, latest_block_head: 242, current_block_head: 223, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:37.031 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_tJyvJN") }, component: JsonRpcServer
Aug 05 12:08:46.137 INFO Received subgraph_deploy request, params: SubgraphDeployParams { name: SubgraphName("octant_test_tJyvJN"), ipfs_hash: DeploymentHash("QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt"), node_id: None, debug_fork: None, history_blocks: None }, component: JsonRpcServer
Aug 05 12:08:46.140 INFO Set subgraph start block, block: None, sgd: 0, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphRegistrar
Aug 05 12:08:46.140 INFO Graft base, block: None, base: None, sgd: 0, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphRegistrar
Aug 05 12:08:46.721 INFO Resolve subgraph files using IPFS, n_templates: 0, n_data_sources: 4, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:46.721 INFO Successfully resolved subgraph files using IPFS, features: ø, n_templates: 0, n_data_sources: 4, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:46.723 INFO Starting subgraph writer, queue_size: 5, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:46.731 INFO Data source count at start: 4, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:46.784 INFO Scanning blocks [0, 0], range_size: 1, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:08:46.787 INFO Applying 1 entity operation(s), block_hash: 0x4657810184697a7dc6d140e376f12c576aec6086962c671e661de57ee0cb0fb9, block_number: 0, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:47.163 INFO Applying 2 entity operation(s), block_hash: 0x51e616e7bcf4a68be42e173c45fca193d946bcb5e131443cf855b6362b18d6e5, block_number: 228, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:47.183 INFO Applying 3 entity operation(s), block_hash: 0x1f94f6f512fea2bff03bbe1b71bb441ce78d9179243f2fab320cf3ca8a3db0b3, block_number: 240, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:47.900 INFO Syncing 6 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 6, blocks_behind: 6, latest_block_head: 248, current_block_head: 242, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:48.451 INFO Applying 4 entity operation(s), block_hash: 0x782768aabf1016a47cfead315ab893dfca9999e818826e0ac26b480719c9c7e5, block_number: 248, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:48.915 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 249, current_block_head: 248, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:49.434 INFO Applying 2 entity operation(s), block_hash: 0x0c214fca6cc708e09e20c5b2ee0c775ed131b232876113aacb79f095970cc649, block_number: 249, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:08:51.923 INFO Syncing 11 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 11, blocks_behind: 11, latest_block_head: 260, current_block_head: 249, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:08:52.967 INFO Syncing 6 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 6, blocks_behind: 6, latest_block_head: 266, current_block_head: 260, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:09:02.272 INFO Received subgraph_create request, params: SubgraphCreateParams { name: SubgraphName("octant_test_m1OoKI") }, component: JsonRpcServer
Aug 05 12:09:13.031 INFO Syncing 7 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 7, blocks_behind: 7, latest_block_head: 273, current_block_head: 266, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:09:13.606 INFO Applying 2 entity operation(s), block_hash: 0xc1c59edc6729ea3eb8e302690480bfb996e1c0fd75a3f351bfcfcb2b409655f0, block_number: 273, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:14.048 INFO Syncing 1 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 1, blocks_behind: 1, latest_block_head: 274, current_block_head: 273, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:09:14.565 INFO Applying 2 entity operation(s), block_hash: 0x058339e9953ec14f8070e51091ac5704d8f5ceec7708e6b7f28b031cb82fb4fd, block_number: 274, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:17.056 INFO Syncing 16 blocks from Ethereum, code: BlockIngestionLagging, blocks_needed: 16, blocks_behind: 16, latest_block_head: 290, current_block_head: 274, provider: localhost-rpc-0, component: EthereumPollingBlockIngestor
Aug 05 12:09:17.606 INFO Reverting block to get back to main chain, revert_to_ptr: #273 (c1c59edc6729ea3eb8e302690480bfb996e1c0fd75a3f351bfcfcb2b409655f0), subgraph_ptr: #274 (058339e9953ec14f8070e51091ac5704d8f5ceec7708e6b7f28b031cb82fb4fd), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:17.629 INFO Reverting block to get back to main chain, revert_to_ptr: #272 (a206f5eddb5d29551e2d2c2136d7e1d17b092f7faac93d1fb08f366592171fc0), subgraph_ptr: #273 (c1c59edc6729ea3eb8e302690480bfb996e1c0fd75a3f351bfcfcb2b409655f0), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.621 INFO Reverting block to get back to main chain, revert_to_ptr: #217 (77e5a5b7376f8a6218e181632774379282346ee26abf2b8db942e58a1b776585), subgraph_ptr: #218 (ba522f62057a9947f4a8b06bfa1a0e1cdc784c71c333f1843973226ab17e143f), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.637 INFO Reverting block to get back to main chain, revert_to_ptr: #216 (f9a810abc3da477559dced2db76502c5d716e65ebc238f65c2d31ca89cb30546), subgraph_ptr: #217 (77e5a5b7376f8a6218e181632774379282346ee26abf2b8db942e58a1b776585), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.653 INFO Reverting block to get back to main chain, revert_to_ptr: #215 (b4eb938b80a6660610920f43349dcbfb00a7d049a2fb963ddd2fba2915d05039), subgraph_ptr: #216 (f9a810abc3da477559dced2db76502c5d716e65ebc238f65c2d31ca89cb30546), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.663 INFO Reverting block to get back to main chain, revert_to_ptr: #214 (7113a9e1e1fb65015a1629932fa817c4ef3dd732a9dec81d405e4088ba938860), subgraph_ptr: #215 (b4eb938b80a6660610920f43349dcbfb00a7d049a2fb963ddd2fba2915d05039), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.689 INFO Reverting block to get back to main chain, revert_to_ptr: #213 (2e7980beecae1ebed5e83c2b175bf6f58274bd4bfc47b9ca9371c773c7797bf8), subgraph_ptr: #214 (7113a9e1e1fb65015a1629932fa817c4ef3dd732a9dec81d405e4088ba938860), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:18.706 INFO Reverting block to get back to main chain, revert_to_ptr: #212 (07c05ab5d8549240122d75116c00a518b870452c43caa761fb1cf41d4fa8fc31), subgraph_ptr: #213 (2e7980beecae1ebed5e83c2b175bf6f58274bd4bfc47b9ca9371c773c7797bf8), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.847 INFO Reverting block to get back to main chain, revert_to_ptr: #46 (61e9db94e2081e7b541ac197b7f3873d7f1c042efc4111f9ea3e0449f7d3b786), subgraph_ptr: #47 (75dd5e9809a02288ca5fd70c6dbd5cfb76cbb3c66f363474cdb938daa42d5f54), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.863 INFO Reverting block to get back to main chain, revert_to_ptr: #45 (094759e1352336b8f418b4cf8c88a6bedd11bccf03bcc1d92e3fafcef2c43199), subgraph_ptr: #46 (61e9db94e2081e7b541ac197b7f3873d7f1c042efc4111f9ea3e0449f7d3b786), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.884 INFO Reverting block to get back to main chain, revert_to_ptr: #44 (021a4490b3d46abe48f96f3a2c0abbdb773f34ec8965e9befcb075cf0f60096f), subgraph_ptr: #45 (094759e1352336b8f418b4cf8c88a6bedd11bccf03bcc1d92e3fafcef2c43199), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.905 INFO Reverting block to get back to main chain, revert_to_ptr: #43 (3ef6b2eee589f036ff02489e2ecb5b4b35af8e7ce171cad83d985c1b5f083aa1), subgraph_ptr: #44 (021a4490b3d46abe48f96f3a2c0abbdb773f34ec8965e9befcb075cf0f60096f), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.920 INFO Reverting block to get back to main chain, revert_to_ptr: #42 (b608701a2f8aa6b9f8adbe6e914de4b10c2cd42bac7cdcb0c4838804131e50af), subgraph_ptr: #43 (3ef6b2eee589f036ff02489e2ecb5b4b35af8e7ce171cad83d985c1b5f083aa1), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.941 INFO Reverting block to get back to main chain, revert_to_ptr: #41 (eef114c50ab517b197c1c904fdfb4dd52400e1eb8961ba4cbe4a531017b25c00), subgraph_ptr: #42 (b608701a2f8aa6b9f8adbe6e914de4b10c2cd42bac7cdcb0c4838804131e50af), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.960 INFO Reverting block to get back to main chain, revert_to_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), subgraph_ptr: #41 (eef114c50ab517b197c1c904fdfb4dd52400e1eb8961ba4cbe4a531017b25c00), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.981 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:21.982 INFO Scanning blocks [40, 40], range_size: 10, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.019 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.021 INFO Scanning blocks [40, 40], range_size: 100, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.052 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.062 INFO Scanning blocks [40, 40], range_size: 1000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.091 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.101 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.134 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.140 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.168 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.170 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:09:22.204 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:09:22.211 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:23.915 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:23.943 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:23.953 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:23.987 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:23.996 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.032 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.034 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:24.064 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:24.069 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.281 INFO Scanning blocks [40, 40], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.320 INFO Reverting block to get back to main chain, revert_to_ptr: #39 (8a866e1b66b36df6092c336b150b0e75124b460293a78a02af13d60199f80cf8), subgraph_ptr: #40 (efb0d96faf3fb39334e625c9ab92175e560c24beffc36ffbe95b94e635e29608), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:24.360 INFO Scanning blocks [40, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.411 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.449 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.498 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.553 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:24.600 INFO Reverting block to get back to main chain, revert_to_ptr: #47 (75dd5e9809a02288ca5fd70c6dbd5cfb76cbb3c66f363474cdb938daa42d5f54), subgraph_ptr: #48 (176c8b512c90231e1c50f5e11fab56ec53a015421243622ed5bf049732e47334), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:24.602 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:25.352 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:25.403 INFO Scanning blocks [48, 48], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:25.403 INFO Reverting block to get back to main chain, revert_to_ptr: #47 (75dd5e9809a02288ca5fd70c6dbd5cfb76cbb3c66f363474cdb938daa42d5f54), subgraph_ptr: #48 (176c8b512c90231e1c50f5e11fab56ec53a015421243622ed5bf049732e47334), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:25.453 INFO Reverting block to get back to main chain, revert_to_ptr: #47 (75dd5e9809a02288ca5fd70c6dbd5cfb76cbb3c66f363474cdb938daa42d5f54), subgraph_ptr: #48 (176c8b512c90231e1c50f5e11fab56ec53a015421243622ed5bf049732e47334), sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: SubgraphInstanceManager
Aug 05 12:11:25.462 INFO Scanning blocks [48, 52], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:25.505 INFO Scanning blocks [52, 52], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream
Aug 05 12:11:25.542 INFO Scanning blocks [52, 52], range_size: 2000, sgd: 10, subgraph_id: QmaYtA9X9zi689TiGLxB2cyUZK2mWtkAwwL6D8VmhPDvdt, component: BlockStream

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux
@paulperegud paulperegud added the bug Something isn't working label Aug 6, 2024
@paulperegud paulperegud mentioned this issue Aug 7, 2024
Merged
@paulperegud
Copy link
Author

A bit more information. We are using graph-node in Octant. We use self-hosted instance of graph-node on prod. One of automated testing environments deploys set of contracts and a subgraph per test. This is where we observe this behavior most often.

paulperegud added a commit to golemfoundation/octant that referenced this issue Aug 7, 2024
@paulperegud
apitest: skip e2e test if subgraph is stuck not following chain head
b2387e7 
The cause of the error is likely this:
graphprotocol/graph-node#5586

Please run the test locally to make sure it passes.

Long term we should find a way to have a stable graph-node.

Possible avenues:
-> fixing the problem in issue above (#5586)
-> finding some other way to work around it
-> reducing brittleness of graph-node by removing subraphs on test teardown
-> ...
paulperegud added a commit to golemfoundation/octant that referenced this issue Aug 7, 2024
@paulperegud
apitest: skip e2e test if subgraph is stuck not following chain head
2559125 
The cause of the error is likely this:
graphprotocol/graph-node#5586

Please run the test locally to make sure it passes.

Long term we should find a way to have a stable graph-node.

Possible avenues:
-> fixing the problem in issue above (#5586)
-> finding some other way to work around it
-> reducing brittleness of graph-node by removing subraphs on test teardown
-> ...
paulperegud added a commit to golemfoundation/octant that referenced this issue Aug 7, 2024
@paulperegud
apitest: skip e2e test if subgraph is stuck not following chain head
6281ac4 
The cause of the error is likely this:
graphprotocol/graph-node#5586

Please run the test locally to make sure it passes.

Long term we should find a way to have a stable graph-node.

Possible avenues:
-> fixing the problem in issue above (#5586)
-> finding some other way to work around it
-> reducing brittleness of graph-node by removing subraphs on test teardown
-> ...
paulperegud added a commit to golemfoundation/octant that referenced this issue Aug 7, 2024
@paulperegud
apitest: skip e2e test if subgraph is stuck not following chain head
fce2818 
The cause of the error is likely this:
graphprotocol/graph-node#5586

Please run the test locally to make sure it passes.

Long term we should find a way to have a stable graph-node.

Possible avenues:
-> fixing the problem in issue above (#5586)
-> finding some other way to work around it
-> reducing brittleness of graph-node by removing subraphs on test teardown
-> ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@paulperegud