-
Notifications
You must be signed in to change notification settings - Fork 10
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #98 from covalenthq/develop
- Loading branch information
Showing
12 changed files
with
355 additions
and
138 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,7 +8,17 @@ | |
| - [Upload a file](#upload-a-file) | ||
| - [Download content (given cid)](#download-content-given-cid) | ||
| - [Find the cid given some content](#find-the-cid-given-some-content) | ||
| - [migration to UCAN and capabilities setup](#migration-to-ucan-and-capabilities-setup) | ||
| - [setting up w3cli](#setting-up-w3cli) | ||
| - [installation](#installation) | ||
| - [login and check spaces](#login-and-check-spaces) | ||
| - [generate ucan key](#generate-ucan-key) | ||
| - [create delegation to store/add and upload/add](#create-delegation-to-storeadd-and-uploadadd) | ||
| - [communicate to the operator](#communicate-to-the-operator) | ||
| - [operator invocation](#operator-invocation) | ||
| - [Running ipfs-pinner server with docker](#running-ipfs-pinner-server-with-docker) | ||
| - [Running the image](#running-the-image) | ||
| - [Building the docker image](#building-the-docker-image) | ||
| - [Docker Volume setup](#docker-volume-setup) | ||
| - [Port mapping setup](#port-mapping-setup) | ||
| - [Development](#development) | ||
|
|
@@ -24,7 +34,7 @@ | |
| ## Introduction | ||
|
|
||
| - A wrapper on top of ipfs node, utilising go-ipfs as a library. | ||
| - Extended support for custom file upload endpoints provided by pinata & web3.storage. | ||
| - Extended support for custom file upload endpoints provided by web3.storage. | ||
| - Content archive file generation and lightweight deterministic CID generation on client side (using CARs). | ||
| - It can be used as a go library (see `binary/main.go` for usage) or as a http server. | ||
|
|
||
|
|
@@ -40,18 +50,19 @@ To avoid this issue, the merkle DAG thus generated is exported into special file | |
|
|
||
| ## Running ipfs-pinner server | ||
|
|
||
| 1. Set the environment variable `WEB3_JWT` | ||
| 1. Get the agent key, did and delegation proof from Covalent | ||
|
|
||
| 2. to start a server which listens for request on 3001 port, run: | ||
| 2. build the server and run: | ||
|
|
||
| ```bash | ||
| make clean server-dbg run | ||
| make clean server-dbg | ||
| ``` | ||
|
|
||
| NOTE: If you want more control over CLI params, you can run the server binary (after `make clean server-dbg`): | ||
|
|
||
| ```bash | ||
| ./build/bin/server -jwt <WEB3_JWT> -port 3001 | ||
| ./build/bin/server -w3-agent-key <AGENT_KEY> -w3-delegation-file <DELEGATION_PROOF_FILE_PATH> | ||
| ``` | ||
|
|
||
| NOTE: If you get some error when running this, check if the diagnostic is there in [known issues](#known-issues) | ||
|
|
@@ -107,6 +118,96 @@ There's a timeout (check code for value) for the download request, if it doesn't | |
| {"cid": "bafkreicszve3ewhhrgobm366mdctki2m2qwzide5e54zh5aifnesg3ofne"}% | ||
| ``` | ||
|
|
||
|
|
||
| ## migration to UCAN and capabilities setup | ||
| - web3.storage is sunsetting its custom upload endpoint (on 9th January, 2024), and we need to migrate from using that to w3up. | ||
| - w3up uses UCAN which is a capabilities-based authorization system (learn more [here](https://web3.storage/docs/concepts/ucans-and-web3storage/)). | ||
| - In this setup, the "central account" (owned by Covalent) sets up a "space" (think namespace for data). The central account (controlled by the email) is delegated the capabilty to operate on this space. | ||
| - among other capabilties, the central account can delegate certain capabilities (like uploading to space) to other **agents**. This has to be done at our end, and scripts will be made available for it in this repo. | ||
| - once an agent is granted the capability, we share the credentials with the operators, who run ipfs-pinner with it, and can then upload or fetch. | ||
|
|
||
|
|
||
| ### setting up w3cli | ||
|
|
||
| - Create a web3.storage account in the [console](https://console.web3.storage/). | ||
| - Create a space which you want to use to upload artifacts. We want to use different spaces for different artifacts to keep a clear separation. | ||
|
|
||
| We'll use w3cli to login and create a new space and register. | ||
|
|
||
| #### installation | ||
| ```bash | ||
| ➜ npm install -g @web3-storage/w3cli | ||
|
|
||
| ➜ w3 --version | ||
| w3, 7.0.3 | ||
| ``` | ||
|
|
||
| #### login and check spaces | ||
| ```bash | ||
| ➜ w3 login [email protected] | ||
|
|
||
| ➜ w3 space ls | ||
| did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe mock_artifacts | ||
|
|
||
| ➜ w3 space use did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe | ||
| did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe | ||
| ``` | ||
|
|
||
| The did key is the identifier for this space. Now let's generate some DIDs for an operator and delegate upload capabilities to it. | ||
|
|
||
| #### generate ucan key | ||
| ```bash | ||
| ➜ npx ucan-key ed --json | ||
| { | ||
| "did": "did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj", | ||
| "key": "MgCbc48J8n+BMdzA4XxwYOaKmdu5Ov34jE71U8vV07IVIjO0BnJa05mNMcB8GSz1lib014QAhvAxorG6zACrstm6PBGA=" | ||
| } | ||
| ``` | ||
|
|
||
| #### create delegation to store/add and upload/add | ||
|
|
||
| ```bash | ||
| ➜ w3 delegation create -c 'store/add' -c 'upload/add' did:key:z6MkpzWw1fDZYMpESgVKFAT87SZAuHiCQZVBC3hmQjB18Nzj -o proof.out | ||
| ``` | ||
|
|
||
|
|
||
| Copy the output. This is the delegation string. | ||
|
|
||
| #### communicate to the operator | ||
|
|
||
| Provide the operator with the `did`, `key` string + `proof.out` file. These will be passed to operator's setup of the | ||
| ipfs-pinner, which can then make the delegations. | ||
|
|
||
|
|
||
| #### operator invocation | ||
|
|
||
| the operator can pass the `key` for `-w3-agent-key` and proof file in `-w3-delegation-file` flag. | ||
|
|
||
| ```bash | ||
| go run server/main.go -w3-agent-key <agent-key> -w3-delegation-file ./proof.out | ||
| ipfs-pinner | ||
| ipfs-pinner Version: 0.1.16 | ||
| Architecture: arm64 | ||
| Go Version: go1.20.5 | ||
| Operating System: darwin | ||
| GOPATH=/Users/sudeep/go/ | ||
| GOROOT=/usr/local/go | ||
| 2024/01/04 15:52:05 agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y | ||
| generating 2048-bit RSA keypair...done | ||
| peer identity: QmY49BMJdGneQjJAbTPrGSqaQcLjpCE1WFkRBP6XZEHd6i | ||
| 2024/01/04 15:52:09 setting up w3up for uploads.... | ||
| 2024/01/04 15:52:10 w3up agent did: did:key:z6MkoLvhaiE9NRYs3vJcynCM8CeyP8hXduWhE5Ter2U2x93y | ||
| 2024/01/04 15:52:10 w3up space did: did:key:z6MkgSK6VEu3bvrAFtYNyjsnzG7dVXzYi3yT5TasEgeaQrCe | ||
| 2024/01/04 15:52:10 w3up setup complete | ||
| 2024/01/04 15:52:10 Listening... | ||
| 2024/01/04 15:52:15 generated dag has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu | ||
| 2024/01/04 15:52:15 car file location: /var/folders/w0/bf3y1c7d6ys15tq97ffk5qhw0000gn/T/3475885728.car | ||
| 2024/01/04 15:53:06 w3 up output: {"root":{"/":"bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu"}} | ||
| 2024/01/04 15:53:28 uploaded file has root cid: bafybeigvijf76lcsjwcmkr6rmzovoiiqdog3muqs5vnplvf4jxh47shfiu | ||
| ``` | ||
|
|
||
|
|
||
|
|
||
| ## Running ipfs-pinner server with docker | ||
|
|
||
| We can also run the ipfs-pinner server via docker. | ||
|
|
@@ -117,34 +218,46 @@ for ipfs-pinner to function properly with docker, we need | |
|
|
||
| Docker run command should have: | ||
|
|
||
| - Volumes for data persistence | ||
| - Volumes for data persistence; | ||
| - Port mappings | ||
| - JWT token passed in the env | ||
| - W3up agent key passed in the env | ||
|
|
||
|
|
||
| ### Running the image | ||
|
|
||
| Copy the delegation proof file into the ipfs directory which will be mapped onto the docker image. | ||
|
|
||
| ```bash | ||
| docker buildx create --name builder --use --platform=linux/amd64,linux/arm64 && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest | ||
| mv proof.out /tmp/data/.ipfs/ | ||
| ``` | ||
|
|
||
| Now, we can run the container: | ||
| Then one can run the docker container: | ||
|
|
||
| ```bash | ||
| docker container run --detach --name ipfs-pinner-instance \ | ||
| --volume /tmp/data/.ipfs/:/root/.ipfs/ \ | ||
| -p 3001:3001 \ | ||
| --env WEB3_JWT=$WEB3_JWT \ | ||
| --env W3_AGENT_KEY=$W3_AGENT_KEY \ | ||
| --env W3_DELEGATION_FILE=/root/.ipfs/proof.out | ||
| <image-id> | ||
| ``` | ||
|
|
||
|
|
||
| ### Building the docker image | ||
| ```bash | ||
| docker buildx create --name builder --use --platform=linux/amd64,linux/arm64 && docker buildx build --platform=linux/amd64,linux/arm64 . -t gcr.io/covalent-project/ipfs-pinner:latest | ||
| ``` | ||
|
|
||
| ### Docker Volume setup | ||
|
|
||
| There's 1 docker volume that needs to be shared (and persisted) between the container and the host - this `~/.ipfs` directory needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed. | ||
| There's 1 docker volume that needs to be shared (and persisted) between the container and the host - the `~/.ipfs` directory, which needs to have its lifecycle unaffected by container lifecycle (since it contains the merklelized nodes, blockstore etc.), and so that is docker volume managed. | ||
|
|
||
| ### Port mapping setup | ||
|
|
||
| :4001 : swarm port for p2p (currently disabled) | ||
| :8080 - http gateway (used by encapsulated ipfs-node) | ||
| :5001: local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node) | ||
| :3001: The ipfs-pinner itself exposes its REST API on this port | ||
| `:4001` - swarm port for p2p (currently disabled) | ||
| `:8080` - http gateway (used by encapsulated ipfs-node) | ||
| `:5001` - local api (should be bound to 127.0.0.1 only, and must never be exposed publicly as it allows one to control the ipfs node; also used by encapsulated ipfs-node) | ||
| `:3001` - The ipfs-pinner itself exposes its REST API on this port | ||
|
|
||
| <B> Out of the above, only the swarm port and the REST api port (3001) are essential.</B> | ||
|
|
||
|
|
@@ -237,7 +350,7 @@ Users would sometimes want to maintain a different volume to fulfil large storag | |
| ipfs-pinner currently uses some known IPFS gateways to fetch content. These gateways are expected to be run and maintained for a long time, but if you need to update the gateways list due to one of the going down, or a more efficient gateway being introduced etc. you can change the list: | ||
| ```bash | ||
| ./build/bin/server -jwt <WEB3_JWT> -port 3001 -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s" | ||
| ./build/bin/server -ipfs-gateway-urls "https://w3s.link/ipfs/%s,https://dweb.link/ipfs/%s,https://ipfs.io/ipfs/%s" ##OTHER PARAMS | ||
| ``` | ||
| The `-ipfs-gateways-urls` is a comma separated list of http urls with a `%s` present in it, which is formatted to replace the IPFS content identifier (CID) in it. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.