fix: export events tsv directly to postgres instance #2048
Conversation
|
Vercel deployment URL: https://stacks-blockchain-3tc7rhdm1-hirosystems.vercel.app 🚀 |
Codecov ReportAttention: Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Can we make the local export an opt-in feature? I made a modification (2f9e591) and just tested against a kubectl port-forward to a mainnet dev instance. It exported a ~50GB tsv to my machine, and I validated that every line is valid json.
|
that looks great @zone117x , would you mind applying that patch to this branch? |
|
My guess is that the CI failure is because host docker filesystem usage causing the |
|
@CharlieC3 do you have a suggestion on what I can do to fix this failure? https://github.com/hirosystems/stacks-blockchain-api/actions/runs/10392592660/job/28778191413#step:7:655 I think @zone117x is right that the user difference is making the test fail here. Perhaps it won't be possible to test this particular thing on gh actions? |
|
@rafaelcr The postgres image runs the database as the postgres user, so you're both right it is a user mismatch. You may have to |
|
That did it @CharlieC3 thanks! |
* fix: export events tsv directly to postgres instance * fix: remove unused function * chore: option to export events to either local file or client * chore: try new docker path * chore: divide remote and local paths * fix: try relative path for mkdir * ci: try chmod * ci: run mkdir first * ci: try with sudo * fix: file paths --------- Co-authored-by: Matthew Little <[email protected]>
* fix: export events tsv directly to postgres instance * fix: remove unused function * chore: option to export events to either local file or client * chore: try new docker path * chore: divide remote and local paths * fix: try relative path for mkdir * ci: try chmod * ci: run mkdir first * ci: try with sudo * fix: file paths --------- Co-authored-by: Matthew Little <[email protected]>
## [8.0.0-beta.6](v8.0.0-beta.5...v8.0.0-beta.6) (2024-08-27) ### ⚠ BREAKING CHANGES * refactor from express to fastify (#2045) ### Features * cursor-based pagination on blocks endpoint ([#2060](#2060)) ([bfdcce1](bfdcce1)) * export events tsv directly to postgres instance ([#2048](#2048)) ([f401a0f](f401a0f)) * export events tsv directly to postgres instance ([#2048](#2048)) ([#2058](#2058)) ([a1f5b12](a1f5b12)) * refactor from express to fastify ([#2045](#2045)) ([bd65fcf](bd65fcf)), closes [#2042](#2042) ### Bug Fixes * index on `principal_stx_txs` table for faster `/v1/address/{addr}/transactions` lookups ([#2059](#2059)) ([ab64ab7](ab64ab7))
|
🎉 This PR is included in version 8.0.0-beta.6 🎉 The release is available on: Your semantic-release bot 📦🚀 |
## [8.0.0](v7.14.1...v8.0.0) (2024-08-28) ### ⚠ BREAKING CHANGES * refactor from express to fastify (#2045) * refactor from Express to Fastify ### Features * cursor-based pagination on blocks endpoint ([#2060](#2060)) ([bfdcce1](bfdcce1)) * export events tsv directly to postgres instance ([#2048](#2048)) ([f401a0f](f401a0f)) * refactor from Express to Fastify ([aa0e51e](aa0e51e)), closes [#2042](#2042) * refactor from express to fastify ([#2045](#2045)) ([bd65fcf](bd65fcf)), closes [#2042](#2042) ### Bug Fixes * missing event limit max overrides on a few endpoints ([4f70930](4f70930)) * pagination and query param parsing bugs ([a382d2b](a382d2b)), closes [#2042](#2042) * perform status endpoint sql inside transactions ([b23445c](b23445c)) * tx event-limit default should be 100 ([32d0670](32d0670))
|
🎉 This PR is included in version 8.0.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This PR adds a
localoption to theexport-eventsAPI CLI command. If the argument to--filestarts withlocal:, it will export events using only postgres to a file path local to its container.