Updated docs site

Updated docs site with more details.

site/Gemfile

@@ -35,3 +35,5 @@ gem "http_parser.rb", "~> 0.6.0", :platforms => [:jruby]
 # theme
 gem "just-the-docs"
 
+gem 'jemoji'
+

site/Gemfile.lock

@@ -260,6 +260,7 @@ DEPENDENCIES
   github-pages (~> 228)
   http_parser.rb (~> 0.6.0)
   jekyll-feed (~> 0.12)
+  jemoji
   just-the-docs
   minima (~> 2.5)
   tzinfo (>= 1, < 3)

site/_config.yml

@@ -18,19 +18,33 @@
 # You can create any custom variable you would like, and they will be accessible
 # in the templates via {{ site.myvariable }}.
 
-title: Project Yellowstone - Old Faithful
+title: ♨️ Old Faithful
 email: [email protected]
-description: >- # this means to ignore newlines until "baseurl:"  
-    Old Faithful is part of Project Yellowstone, reinventing the Solana RPC stack. 
+description: >- # this means to ignore newlines until "baseurl:"
+    Old Faithful is part of Project Yellowstone, reinventing the Solana RPC stack.
 baseurl: "" # the subpath of your site, e.g. /blog
-url: "https://www." # the base hostname & protocol for your site, e.g. http://example.com
+url: "https://old-faithful.net" # the base hostname & protocol for your site, e.g. http://example.com
 twitter_username: triton_one
 github_username: rpcpool
 
 # Build settings
 theme: just-the-docs
 plugins:
   - jekyll-feed
+  - jemoji
+
+callouts:
+  warning:
+    title: Warning
+    color: red
+  important:
+    title: Important
+    color: blue
+
+
+aux_links:
+    "Github":
+      - "https://github.com/rpcpool/yellowstone-faithful"
 
 # Exclude from processing.
 # The following items will not be processed, by default.

site/_includes/nav_footer_custom.html

@@ -0,0 +1 @@
+ 

site/about.markdown

@@ -2,17 +2,15 @@
 layout: page
 title: About
 permalink: /about/
+nav_order: 7
 ---
 
-This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at [jekyllrb.com](https://jekyllrb.com/)
+This project is currently managed by [Triton One](https://triton.one/). If you want more information contact us via [Telegram](https://t.me/+K0ONdq7fE4s0Mjdl).
 
-You can find the source code for Minima at GitHub:
-[jekyll][jekyll-organization] /
-[minima](https://github.com/jekyll/minima)
+Old Faithful is a part of Project Yellowstone. Project Yellowstone is an initiative to reinvent the Solana RPC stack.
 
-You can find the source code for Jekyll at GitHub:
-[jekyll][jekyll-organization] /
-[jekyll](https://github.com/jekyll/jekyll)
+The originator of this project was [Richard Patel](https://github.com/terorie) ([Twitter](https://twitter.com/fd_ripatel)).
 
+[@immaterial.ink](https://github.com/gagliardetto) ([Twitter](https://twitter.com/immaterial_ink)) is currently the lead dev on this project at Triton One.
 
-[jekyll-organization]: https://github.com/jekyll
+This work has been supported greatly by Protocol Labs (special shout out to [anjor](https://github.com/anjor) ([Twitter](https://twitter.com/__anjor)) for all the guidance in Filecoin land to us Solana locals).

site/archives.markdown

@@ -0,0 +1,11 @@
+---
+layout: page
+title: Archives
+permalink: /archives/
+has_children: true
+nav_order: 4
+---
+
+The core of the project is history archives in Content Addressable format ([overview](https://web3.storage/docs/how-tos/work-with-car-files/), [specs](https://ipld.io/specs/transport/car/carv1/)). These represent a verifiable, immutable view of the Solana history. The CAR files that this project generates follows a [schema](https://github.com/rpcpool/yellowstone-faithful/blob/main/ledger.ipldsch) specifically developed for Solana's historical archives.
+
+The content addressable nature means that each epoch, block, transaction and shredding is uniquely identified by a content hash. By knowing this content hash a user will be able to retreive a specific object of interest in a trustless manner, i.e. retrieve an object verifiably from a non-trusted source. Retrievals can be made via IPFS, the Filecoin network, or even by hosting the CAR files yourself on disk, a ceph cluster, S3, you name it.

site/cid.markdown

@@ -0,0 +1,25 @@
+---
+layout: page
+title: Content Identifiers
+permalink: /archives/cid/
+parent: Archives
+nav_order: 1
+---
+
+# Content Identifiers
+
+In the archives each Epoch, Block and Transaction gets a computed content identifier. These content identifiers can be used to uniquely identify any of these elements when retrieving either from a locally stored CAR file or from IPFS or Filecoin.
+
+In order to retrieve objects using Solana slot numbers, transaction signatures or epoch numbers, indexes are used to map these Solana specific identifiers onto CIDs.
+
+To learn more about CIDs, you can deep dive into the IPFS documentation [here](https://docs.ipfs.tech/concepts/content-addressing/).
+
+From [https://docs.ipfs.tech/concepts/content-addressing/#what-is-a-cid](https://docs.ipfs.tech/concepts/content-addressing/#what-is-a-cid):
+
+  > A content identifier, or CID, is a label used to point to material in IPFS. It doesn't indicate where the content is stored, but it forms a kind of address based on the content itself. CIDs are short, regardless of the size of their underlying content.
+  > CIDs are based on the content’s cryptographic hash. That means:
+  > - Any difference in the content will produce a different CID
+  > - The same content added to two different IPFS nodes using the same settings will produce the same CID.
+
+
+The CAR files and CIDs that are generated by Old Faithful are reproducible and will be the same regardless from which Rocksdb ledger archive they are generated from. This means that entities that are running archive nodes and store Rocksdb ledger archives can easily generated their own CIDs for the same blocks and transactions and compare to the ones being generated by the Old Faithful project. In theory this means that you can also choose to run a node that /just/ generates and stores the CIDs but not the durable data and then compare those CIDs to the ones provided by Faithful. Using these CIDs you can retrieve the actual data any time without trusting anyone but your own data generating nodes.

site/data-preparation.markdown

@@ -0,0 +1,42 @@
+---
+layout: page
+title: Data Preparation
+permalink: /archives/dataprep/
+parent: Archives
+nav_order: 2
+---
+
+# Data preparation
+
+The primary data preparation tooling used in this project is based in the `radiance` tool developed by Jump's Firedancer team. It is rapidely developing, and active development for this project is currently based out of this repository and branch: [Radiance Triton](https://github.com/gagliardetto/radiance-triton/).
+
+The radiance tool utilises the rocksdb snapshots that have been generated by [Warehouse](https://github.com/solana-labs/solana-bigtable) nodes. From these snapshots a CAR file per epoch is generated. This CAR file then needs to be processed by Filecoin tools such as [split-and-commp](https://github.com/anjor/go-fil-dataprep/) which generates the details needed for making a Filecoin deal.
+
+Currently, this tool is being tested from the following warehouse archives:
+  - Solana Foundation (public)
+    -  gs://mainnet-beta-ledger-us-ny5
+    -  gs://mainnet-beta-ledger-europe-fr2
+    -  gs://mainnet-beta-ledger-asia-sg1
+  - Triton One (private)
+
+If you have warehouse nodes generating rocksdb archive snapshots, please contact [email protected] (even if they can't be made publicly available). We would like to have you generate CAR files for verification purposes.
+
+## CAR files
+
+CAR file generation produces a CAR containing a DAG. This DAG is reproducible and follows the structure of Epoch -> Block -> Transaction see [schema](https://github.com/rpcpool/yellowstone-faithful/blob/main/ledger.ipldsch). The CAR file generation is deterministic, so even if you use different rocksdb source snapshots you should end up with the same CAR output. This allows comparison between different providers.
+
+The data generation flow is illustrated below:
+
+![radiance datagen flow](https://github.com/rpcpool/yellowstone-faithful/assets/5172293/65f6dd75-189b-4253-968a-e81bfe6c130f)
+
+## Generating an epoch car file
+
+Once you have downloaded rocksdb ledger archives you can run the Radiance tool to generate a car file for an epoch. Make sure you have all the slots available in rocksdb ledger archive for the epoch. You may need to download multiple ledger snapshots in order to have a full set of slots available. Once you know you have a rocksdb that covers all the slots for the epoch run the radiance tool like follows:
+
+```
+radiance car create2 107 --db=46223992/rocksdb --out=/storage/car/epoch-107.car
+```
+
+This will produce a car file called epoch-107.car containing all the blocks and transactions for that epoch.
+
+

site/filecoin.markdown

@@ -0,0 +1,53 @@
+---
+layout: page
+title: Filecoin
+permalink: /filecoin/
+nav_order: 5
+---
+
+# Filecoin
+
+As part of the Old Faithful project we are placing all of Solana's archive data on Filecoin. This will allow secure, decentralized access to Solana's history data.
+
+## CIDs
+
+To retrieve data from Filecoin you will need knowledge about the [content identifiers](/archives/cid/) for the epochs, blocks or transactions you are interested in. In theory, you can clone an entire epoch via Filecoin, but you can also retrieve an individual block or transaction. When fetching blocs or transaction, you can fetch only the metdata or the full data of the transaction/block.
+
+## Fetching CIDs
+
+If you already know the CID of the data you are looking for you can fetch it via `faithful-cli fetch <cid>`. This requires no further indexes and can also be used to recursively fetch data for example for an epoch. To avoid fetching the full dataset for an epoch (100s of GB) you probably want to pass the parameter `--dag-scope=block` to fetch only the particular CID entity that you are interested in.
+
+## Retrieving Epoch CIDs
+
+You can fetch the latest Epoch CIDs uploaded from old-faithful.net:
+
+```
+curl https://files.old-faithful.net/100/epoch-100.cid
+```
+
+## Filecoin RPC
+The filecoin RPC server allows provide getBlock, getTransaction and getSignaturesForAddress powered by Filecoin. This requires access to indexes. The indexes allow you to lookup transaction signatures, block numbers and addresses and map them to Filecoin CIDs.
+
+You can run it in the following way:
+
+```
+faithful-cli rpc-server-filecoin -config 455.yml
+```
+
+The config file points faithful to the location of the required indexes (`455.yaml`):
+```
+indexes:
+  slot_to_cid: './epoch-455.car.bafyreibkequ55hyrhyk6f24ctsofzri6bjykh76jxl3zju4oazu3u3ru7y.slot-to-cid.index'
+  sig_to_cid: './epoch-455.car.bafyreibkequ55hyrhyk6f24ctsofzri6bjykh76jxl3zju4oazu3u3ru7y.sig-to-cid.index'
+  gsfa: './epoch-455.car.gsfa.index'
+```
+
+Due to latency in fetching signatures, typically the getSignaturesForAddress index needs to be stored in a local directory, but the other indexes can be fetched via HTTP or via local file system access. If you provide a URL, you need to make sure that the url supports HTTP Range requests. S3 or similar APIs will support this.
+
+You can enter URLs from old-faithful.net in these config files.
+
+There is a mode in which you can use a remote gSFA index, which limits it to only return signatures and not additional transaction meta data. In this mode, you can use a remote gSFA index. To enable this mode run faithful-cli in the following way:
+
+```
+faithful-cli rpc-server-filecoin -config 455.yml -gsfa-only-signatures=true
+```

site/hosting.markdown

@@ -0,0 +1,11 @@
+---
+layout: page
+title: Hosting
+permalink: /archives/hosting/
+parent: Archives
+nav_order: 4
+---
+
+# Hosting Faithful archives
+
+You can download and host the faithful archives using any file system data source or HTTP endpoint that supports range requests. To host archives or indexes all you need to do is [generate](/archives/data-prep/) or [download](/rpc-server/old-faithful-net/) Epoch car files or indexes. You can then use the config file syntax or command line to serve these car files.

site/index.markdown

@@ -3,4 +3,15 @@
 # To modify the layout, see https://jekyllrb.com/docs/themes/#overriding-theme-defaults
 
 layout: home
+nav_order: 1
 ---
+# :hotsprings: Old Faithful
+
+Old Faithful is the project to make all of Solana's history accessible, content addressable and available via a variety of means. The goal of this project is to completely replace the Bigtable dependency for Solana history access with a self-hostable, decentralized history archive that is usable for infrastructure providers, individual Solana users, explorers, indexers, and anyone else in need of historical access.
+
+{: .important-title }
+> Requests for comments release
+>
+> This is currently in RFC stage, which means that it is not intended for production use and that there may be breaking changes to the format, the CLI utilities or any other details related to the project.
+>
+> We appreciate all and any testing, questions, feedback, code submissions as issues via [github.com/rpcpool/faithful-cli](https://github.com/rpcpool/yellowstone-faithful/issues). We can also be found on the [Solana tech discord](https://discord.com/invite/kBbATFA7PW) (e.g. @linuskendall).

site/indexes.markdown

@@ -0,0 +1,51 @@
+---
+layout: page
+title: Indexes
+permalink: /archives/indexes/
+parent: Archives
+nav_order: 3
+---
+
+# Indexes
+
+Indexes will be needed to map Solana's block numbers, transaction signatures and addresses to their respective Content Identifiers. These indexes will be developed as part of this project. There are three content identifier indexes kinds that the Old Faithful index generation can provide:
+
+ - slot-to-cid: Lookup a CID based on a slot number
+ - tx-to-cid: Lookup a CID based on a transaction signature
+ - cid-to-offset: Index for a specific CAR file, used by the local rpc server (see above) to find CIDs in a car file
+
+In addition to these Old Faithful supports an index called `gsfa` that maps Solana addresses to a list of transaction signatures.
+
+## Index generation
+
+Once the radiance tooling has been used to prepare a car file (or if you have downloaded a car file externally) you can generate indexes from this car file by using the `faithful-cli`:
+
+```
+NAME:
+   faithful index
+
+USAGE:
+   faithful index command [command options] [arguments...]
+
+DESCRIPTION:
+   Create various kinds of indexes for CAR files.
+
+COMMANDS:
+   cid-to-offset
+   slot-to-cid
+   sig-to-cid
+   all
+   gsfa
+   help, h        Shows a list of commands or help for one command
+
+OPTIONS:
+   --help, -h  show help
+```
+
+For example, to generate the three indexes cid-to-offset, slot-to-cid, sig-to-cid you would run:
+
+```
+faithful-cli index all epoch-107.car .
+```
+
+This would generate the indexes in the current dir for epoch-107.

site/installation.markdown

@@ -0,0 +1,14 @@
+---
+layout: page
+title: Installation
+permalink: /installation/
+nav_order: 2
+---
+
+# Installing faithful-cli
+
+The easiest way to install faithful-cli is to download the pre-built binaries for Linux, Windows or Mac OS provided at [github.com/rpcpool/yellowstone-faithful](https://github.com/rpcpool/yellowstone-faithful/releases).
+
+## Building from source
+
+You can also build from source by cloning [the github repo](https://github.com/rpcpool/yellowstone-faithful/releases) and then running ```make```/