The filecoin pipeline infra for w3protocol built on SST. w3filecoin is responsible for keeping the state over time from CAR files being ready to aggregate, to an aggregate landing in Filecoin Storage providers.
The repo contains the infra deployment code and the service implementation.
├── stacks - sst and aws cdk code to deploy all the things
└── api - lambda & dynamoDB implementations for the filecoin pipeline
To work on this codebase you need:
- Node.js >= v18 (prod env is node v18)
- An AWS account with the AWS CLI configured locally
- Copy
.env.tplto.env.local - Install the deps with
npm i
Deploy dev services to your aws account and start dev console
npm startSee: https://docs.sst.dev for more info on how things get deployed.
Deployment is managed by seed.run. PR's are deployed automatically to https://<pr#>.filecoin.web3.storage.
The main branch is deployed to https://staging.filecoin.web3.storage and staging builds are promoted to prod manually via the UI at https://console.seed.run
You can use sst to create a custom dev deployment on aws, with a local dev console for debugging.
To do that you need
- An AWS account with the AWS CLI configured locally
- Copy
.env.tplto.env.local
Then run npm dev to deploy dev services to your aws account and start dev console
See: https://docs.sst.dev for more info on how things get deployed.
Ensure the following variables are set in the env when deploying
The root domain to deploy the aggregator API to. e.g aggregator.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.
The root domain to deploy the deal tracker API to. e.g tracker.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.
The root domain to deploy the dealer API to. e.g dealer.web3.storage. The value should match a hosted zone configured in route53 that your aws account has access to.
DID of the ucanto server running for the Aggregator service. e.g. did:key:abc... Optional: if omitted, a did:key will be derrived from AGGREGATOR_PRIVATE_KEY
DID of the ucanto server running for the Deal Tracker service. e.g. did:key:abc... Optional: if omitted, a did:key will be derrived from DEAL_TRACKER_PRIVATE_KEY
DID of the ucanto server running a Dealer service. e.g. did:web:dealer.web3.storage.
URL of the UCAN log server. e.g. https://up.web3.storage/ucan
Set production secrets in aws SSM via sst secrets. The region must be set to the one you deploy that stage to
# set `PRIVATE_KEY` for prod
$ npx sst secrets set --region us-west-2 --stage prod PRIVATE_KEY "MgCblCY...="To set a fallback value for staging or an ephmeral PR build use sst secrets set-fallback
# set `PRIVATE_KEY` for any stage in us-east-2
$ npx sst secrets set --fallback --region us-east-2 PRIVATE_KEY "MgCZG7...="note The fallback value can only be inherited by stages deployed in the same AWS account and region.
Confirm the secret value using sst secrets list
$ npx sst secrets list --region us-east-2
PRIVATE_KEY MgCZG7...= (fallback)
$ npx sst secrets list --region us-west-2 --stage prod
PRIVATE_KEY M...=The base64pad multibase encoded ED25519 keypair used as the signing key for the aggregator-api.
Generated by @ucanto/principal EdSigner via ucan-key
Example: MgCZG7EvaA...1pX9as=
The base64pad multibase encoded ED25519 keypair used as the signing key for the dealer-api.
Generated by @ucanto/principal EdSigner via ucan-key
Example: MgCZG7EvaA...1pX9as=
The multibase encoded ED25519 keypair used as the signing key for the deal-tracker-api.
Generated by @ucanto/principal EdSigner via ucan-key
Example: MgCZG7EvaA...1pX9as=
The HTTP Basic auth token for the UCAN Invocation entrypoint, where UCAN invocations can be stored and proxied to the UCAN Stream.
Example: MgCZG7EvaA...1pX9as=