Subsquid framework is a set of tools to ingest, transform and present blockchain data.
Our current focus is on substrate-based chains, with support for EVM-based chains coming next.
Subsquid framework consists of several key components.
An Archive is a set of services formed by substrate-ingest, substrate-gateway, postgres compatible database and optionally substrate-explorer, which together
- ingest blockchain data
- decode and persist it in a normalized form
- make it available for further processing and exploration.
Compared to data access using a conventional chain node RPC, an archive allows one to access data in a more granular fashion and from multiple blocks at once. The explorer service provides a GraphQL API for querying historical blocks, events and transactions with rich filtering capabilities.
substrate-processor fetches on-chain data from an archive and executes user-defined mapping code against it.
substrate-typegen(1) can generate facade TypeScript classes for substrate events, calls and storage items, allowing to work with chain data in a fully typesafe and runtime upgrades aware fashion.
Similarly, squid-evm-typegen(1) generates facade TypeScript classes for type-safe processing of EVM logs.
substrate-processor is designed to be able to load the processed data into an arbitrary database and indeed one can run it without a persistent store.
However, extra tooling for data modeling and serving the data with GraphQL is available when a processor is run against a postgres-compatible database. In such a case, the Subsquid framework suggests the following dev flow:
- Define the target database schema using a GraphQL dialect enriched with custom directives
- Generate TypeORM entity classes with typeorm-codegen
- Generate and apply database migrations with typeorm-migration
- Use powerful and extensible GraphQL server to serve produced data right away.
ETL projects built with the Subsquid framework are called Squids.
Squids have a certain structure and are supposed to be developed as regular node.js packages. See squid-template for a reference.
A typical squid implements both data mapping and an API server presenting the data.
Squids can be deployed to a Subsquid cloud service called Aquairum. In the future, the cloud service will be replaced by an open network of decentralized squid operators.
For more details on how to build, run and deploy a squid consult the Docs
The Subsquid framework incorporates a few npm packages that might be useful in a standalone mode:
- scale-codec - encoder and decoder for SCALE data serialization format
- ss58-codec - encoder and decoder for SS58 address format
- ss58 - ss58-codec paired with SS58 registry.
The best way to get started is to clone and inspect squid-template project.
Our developers are active on telegram. Feel free to join and ask any question!
Subsquid is an OpenSource project, contributions are welcomed, encouraged and will be rewarded!
Please consult CONTRIBUTING.md for hacking instructions and make sure to read our code of conduct.