[Proposal] Project Structure and DAS Operations

[kespinola]

Goal

Adopt a project structure that is well organized and has clear points of extension for new features so as reduce the time to onboard new developers as well as release features.

Reduce the operational complexity of running DAS by reduce the number of deployable binaries that need to be managed in order to run ingestion, api, and ops tools.

Proposal

• Expose all functionality for das via a single cli and thus binary using the below folder structure.

core/ # contains shared modules used in ops, api, and ingest libraries
  src/
    types/ # move das_asset_types to this module
ops/ # subcommands for performing operational commands 
api/ # subcommand for starting api server
ingest/ # subcommand for running ingestion processes
cmd/ # single das binary that provides cli interface for running any das subcommand
  src/
    main.rs

An example main file for the cmd crate which exposes every other available command supported by das.

use anyhow::Result;
use das_api::{subcommand as api_subcommand, ApiCommand};
use das_ops::{subcommand as ops_subcommand, OpsCommand};
use das_ingest::{subcommand as ingest_subcommand, IngestCommand};
use das_sdk::{subcommand as sdk_subcommand, SdkCommand};
use clap::{Parser, Subcommand};

#[derive(Debug, Parser)]
#[clap(author, version)]
struct Args {
    #[command(subcommand)]
    command: Command,
}

#[derive(Debug, Subcommand)]
enum Command {
   Api(ApiCommand),
   Ops(OpsCommand),
   Ingest(IngestCommand),
   # potential new command for a read sdk
  Sdk(SdkCommand)
}


#[tokio::main]
async fn main() -> Result<()> {
    let args = Args::parse();

    env_logger::init();

    match args.command {
        Command::Api(subcommand) => api_subcommand(subcommand).await?,
        Command::Ops(subcommand) => ops_subcommand(subcommand).await?,
        Command::Ingest(subcommand) => ingest_subcommand(subcommand).await?,
        Command::Sdk(subcommand) => sdk_subcommand(subcommand).await?,
    }

    Ok(())
}

Some example commands exposed by the das binary.

das api serve # start read api server
das ops bubblegum backfill # backfill bubblegum transactions
das ops bubblegum report # compare index with chain and emit metric report
das ops push account {account} # push an account to the queue to be reindexed
das ops push transaction {signature} # push a transaction to the queue by its signature
das ingest accounts start --stream=ACCFILL # ingest accounts from the ACCFILL stream
das ingest transactions start --stream=TXN # ingest transactions from the TXN stream
das ingest metadata_json start # start workers for fetching and saving metadata json
das sdk get asset {} --das-api-endpoint= # print the response of getAsset using specified endpoint

• Switch to clap for managing configuration arguments for all commands. Clap does not natively support file based config but could extend it though my recommendation is use clap as is meaning argument and environment based configuration.

Outcomes

• Single binary to house all das functionality means das operators only need to manage a single container or build for performing all available tasks. Single binary simplifies the build process while allowing new subscommands or actions to be added by developers.
• Clap makes exploring the available functionality of das intuitive and automatically document through help commands.

[kespinola]

example of this structure but isolated to just the ops commands.

#163

[kespinola]

If operators still desire to run independent binaries then can scope subcommands using cargo features.

[kespinola]

From discussions in slack with @NicolasPennie and Pedro of Helius is to keep ingest, api, and ops as separate crates but refactor config and task management to make more use of clap.