Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: update indexer in local environment #2268

Merged
merged 1 commit into from
Jan 25, 2024
Merged

chore: update indexer in local environment #2268

merged 1 commit into from
Jan 25, 2024

Conversation

paul121
Copy link
Contributor

@paul121 paul121 commented Jan 24, 2024

Description

See regen-network/indexer#53

Updates to the latest indexer commit and refactors local docker indexer image to use the latest graphile migrations.

Follows the existing patterns in the indexer Docker indexer.Dockerfile, indexer_start.sh and indexer_init.sh: https://github.com/regen-network/indexer/tree/main/docker

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • provided a link to the relevant issue or specification
  • provided instructions on how to test
  • reviewed "Files changed" and left comments if necessary
  • confirmed all CI checks have passed

How to test

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed all author checklist items have been addressed
  • reviewed code correctness and readability
  • verified React components follow DRY principles
  • reviewed documentation is accurate
  • reviewed tests
  • manually tested (if applicable)

@netlify Netlify
Copy link

netlify bot commented Jan 24, 2024 •
edited
Loading

✅ Deploy Preview for regen-website ready!

Name Link
🔨 Latest commit a85ddc1
🔍 Latest deploy log https://app.netlify.com/sites/regen-website/deploys/65b05c754691650008c306a2
😎 Deploy Preview https://deploy-preview-2268--regen-website.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@paul121
Copy link
Contributor Author

paul121 commented Jan 24, 2024

Some simple testing demonstrating that the indexer starts up, uses graphile migrations and is able to index blocks after genesis. I'm still getting up to speed on things and not sure all the ways regen-web interacts with the indexer, specifically if regen-web needs to be updated with this change as well.

Before 
$ docker compose logs indexer
indexer_container  | First start, running init script...
indexer_container  | + psql -c '\i V1__init.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE TABLE
indexer_container  | CREATE TABLE
indexer_container  | CREATE TABLE
indexer_container  | CREATE TABLE
indexer_container  | CREATE INDEX
indexer_container  | CREATE TABLE
indexer_container  | CREATE TABLE
indexer_container  | + psql -c '\i V1_1__fix_msg_attr.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | ALTER TABLE
indexer_container  | CREATE EXTENSION
indexer_container  | UPDATE 0
indexer_container  | ALTER TABLE
indexer_container  | ALTER TABLE
indexer_container  | + psql -c '\i V1_2__add_retirements_table.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE TABLE
indexer_container  | + psql -c '\i V1_3__add_msg_event_attr_type_idx.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE INDEX
indexer_container  | + psql -c '\i V1_4__retirements_owner_idx.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE INDEX
indexer_container  | + psql -c '\i V1_5__add_proposals_table.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE TABLE
indexer_container  | + psql -c '\i V1_6__add_tx_hash.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | ALTER TABLE
indexer_container  | + psql -c '\i V1_7__index_group_policy_address.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE INDEX
indexer_container  | + psql -c '\i V1_8__index_proposal_id.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE INDEX
indexer_container  | + psql -c '\i V1_9__all_ecocredit_txes.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | psql:V1_9__all_ecocredit_txes.sql:1: NOTICE:  index "tx_data_tx_response_code_idx" does not exist, skipping
indexer_container  | DROP INDEX
indexer_container  | CREATE INDEX
indexer_container  | CREATE INDEX
indexer_container  | psql:V1_9__all_ecocredit_txes.sql:9: NOTICE:  function all_ecocredit_txes() does not exist, skipping
indexer_container  | DROP FUNCTION
indexer_container  | CREATE FUNCTION
indexer_container  | + psql -c '\i V1_10__class_issuers.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE TABLE
indexer_container  | + psql -c '\i V1_11__class_issuers_indexes.sql' postgres://postgres:password@localhost:5432/indexer
indexer_container  | CREATE INDEX
indexer_container  | CREATE INDEX
indexer_container  | CREATE INDEX
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | [INFO/PollingProcess-1] child process calling self.run()
indexer_container  | [INFO/PollingProcess-2] child process calling self.run()
indexer_container  | [INFO/PollingProcess-3] child process calling self.run()
indexer_container  | [INFO/MainProcess] process shutting down
indexer_container  | [INFO/MainProcess] calling join() for process PollingProcess-4
indexer_container  | [INFO/PollingProcess-4] child process calling self.run()
indexer_container  | 2024-01-23 23:39:26,943 - index_blocks - INFO - indexing regen-local block 1
indexer_container  | 2024-01-23 23:39:26,949 - index_blocks - INFO - height=1 len(txs)=0
indexer_container  | 2024-01-23 23:39:31,972 - index_blocks - INFO - indexing regen-local block 2
indexer_container  | 2024-01-23 23:39:31,974 - index_blocks - INFO - height=2 len(txs)=0
indexer_container  | 2024-01-23 23:39:36,999 - index_blocks - INFO - indexing regen-local block 3
indexer_container  | 2024-01-23 23:39:37,004 - index_blocks - INFO - height=3 len(txs)=0
indexer_container  | 2024-01-23 23:39:42,036 - index_blocks - INFO - indexing regen-local block 4
indexer_container  | 2024-01-23 23:39:42,040 - index_blocks - INFO - height=4 len(txs)=0
indexer_container  | 2024-01-23 23:39:47,075 - index_blocks - INFO - indexing regen-local block 5
indexer_container  | 2024-01-23 23:39:47,081 - index_blocks - INFO - height=5 len(txs)=0
indexer_container  | 2024-01-23 23:39:52,115 - index_blocks - INFO - indexing regen-local block 6
After 
$ docker compose logs indexer
indexer_container  | First start, running init script...
indexer_container  | yarn run v1.22.19
indexer_container  | $ graphile-migrate reset --erase
indexer_container  | graphile-migrate: dropped database 'indexer'
indexer_container  | graphile-migrate: recreated database 'indexer'
indexer_container  | graphile-migrate: Running migration '000001.sql'
indexer_container  | graphile-migrate: Running migration '000002.sql'
indexer_container  | graphile-migrate: 2 committed migrations executed
indexer_container  | Done in 0.97s.
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | INSERT 0 1
indexer_container  | [INFO/PollingProcess-1] child process calling self.run()
indexer_container  | [INFO/PollingProcess-2] child process calling self.run()
indexer_container  | [INFO/PollingProcess-3] child process calling self.run()
indexer_container  | [INFO/PollingProcess-4] child process calling self.run()
indexer_container  | [INFO/MainProcess] process shutting down
indexer_container  | [INFO/MainProcess] calling join() for process PollingProcess-4
indexer_container  | [INFO/PollingProcess-5] child process calling self.run()
indexer_container  | 2024-01-24 00:25:06,498 - index_blocks - INFO - indexing regen-local block 1
indexer_container  | 2024-01-24 00:25:06,500 - index_blocks - INFO - height=1 len(txs)=0
indexer_container  | 2024-01-24 00:25:11,512 - index_blocks - INFO - indexing regen-local block 2
indexer_container  | 2024-01-24 00:25:11,516 - index_blocks - INFO - height=2 len(txs)=0
indexer_container  | 2024-01-24 00:25:16,533 - index_blocks - INFO - indexing regen-local block 3
indexer_container  | 2024-01-24 00:25:16,537 - index_blocks - INFO - height=3 len(txs)=0
indexer_container  | 2024-01-24 00:25:21,553 - index_blocks - INFO - indexing regen-local block 4
indexer_container  | 2024-01-24 00:25:21,557 - index_blocks - INFO - height=4 len(txs)=0
indexer_container  | 2024-01-24 00:25:26,573 - index_blocks - INFO - indexing regen-local block 5
indexer_container  | 2024-01-24 00:25:26,576 - index_blocks - INFO - height=5 len(txs)=0
indexer_container  | 2024-01-24 00:25:31,602 - index_blocks - INFO - indexing regen-local block 6
indexer_container  | 2024-01-24 00:25:31,605 - index_blocks - INFO - height=6 len(txs)=0
indexer_container  | 2024-01-24 00:25:36,630 - index_blocks - INFO - indexing regen-local block 7
indexer_container  | 2024-01-24 00:25:36,631 - index_blocks - INFO - height=7 len(txs)=0

paul121
paul121 commented Jan 24, 2024
View reviewed changes
docker/indexer.Dockerfile
@@ -1,11 +1,11 @@
FROM python:3.9-slim
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can revert back to python:3.9. In a previous PR I made a change to use python:3.9-slim: #2125

Today I ran into the same issue I initially reported. But doing another docker pull python:3.9 seems to have pulled a newer image that has fixed this issue. I believe this issue explains more why this was happening for me: docker-library/python#837

@paul121 paul121 mentioned this pull request Jan 24, 2024
Closed
wgwz
wgwz reviewed Jan 24, 2024
View reviewed changes
Copy link
Contributor

@wgwz wgwz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK, this LGTM at a glance!

@blushi blushi requested review from blushi and a team and removed request for blushi January 24, 2024 16:27
blushi
blushi approved these changes Jan 25, 2024
View reviewed changes
Copy link
Member

@blushi blushi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK

@blushi blushi merged commit 04fd3b7 into dev Jan 25, 2024
14 checks passed
@blushi blushi deleted the chore-update-indexer-graphile branch January 25, 2024 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wgwz wgwz wgwz left review comments

@blushi blushi blushi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@paul121 @blushi @wgwz