Skip to content

Commit

Permalink
Add support for generic clients (#49)
Browse files Browse the repository at this point in the history
* wip. parsing of generic arguments is weird and not working

* fixed arg parsing. next add shred_version and other validator ENV variables

* finish up generic client

* address comments: update client trait, propagate errors up
  • Loading branch information
gregcusack authored Aug 6, 2024
1 parent 2c1566b commit 978c446
Show file tree
Hide file tree
Showing 11 changed files with 638 additions and 295 deletions.
1 change: 1 addition & 0 deletions PROGRESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ By here:
- [ ] Other Features
- [x] Heterogeneous Clusters (i.e. multiple validator versions)
- [x] Deploy specific commit
- [x] Generic Clients
- [ ] Deploy with user-defined stake distribution

By here:
Expand Down
122 changes: 121 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,92 @@ curl -X POST \
}' \
http://<node-ip>:<external-port>
```
Note: you can deploy any client through validator-lab or just completely separately and have the client send TXs or query this RPC through the `http://<node-ip>:<external-port>`.

## Generic Clients
Bring your own client and deploy it in a Validator Lab cluster!
All you need is a containerized version of your client in an accessible docker registry.

Key points/steps:
1) [Containerize your client](#Containerize-your-Client)
2) Any client accounts should be built into the client container image
3) Client arguments are passed in similar to how they are passed into the bench-tps client. For the generic client, use `--generic-client-args`.

For example, let's assume we have a client sending spam. And it takes the following arguments:
```
/home/solana/spammer-executable --target-node <kubernetes_domain_name>:<port> --thread-sleep-ms <ms-between-spam-batches> --spam-mode <client-specific-mode>
```
where `<kubernetes_domain_name>:<port>` is the domain name and port of the kubernetes service running the validator you want to target. See: [Node Naming Conventions](#kubernetes_domain_name)

When we go to deploy the generic client, we deploy it in a similar manner to how we deploy the bench-tps client:
```
cargo run --bin cluster -- -n <namespace>
...
generic-client --docker-image <client-docker-image> --executable-path <path-to-executable-in-docker-image> --delay-start <seconds-after-cluster-is-deployed-before-deploying-client> --generic-client-args 'target-node=<kubernetes_domain_name>:<port> thread-sleep-ms=<ms-between-spam-batches> spam-mode=<client-specific-mode>'
```

4) Any flag or value the client needs that is cluster specific should be read in from an environment variable. For example, say the client requires the following arguments:
```
/home/solana/spammer-executable --target-node <kubernetes_domain_name>:<port> --shred-version <version>
```
Shred-version is cluster specific; it is not known when you deploy a cluster. Modify the shred-version argument in the client code to read in the environment variable `SHRED_VERSION` from the host.
Example:
```
let default_shred_version = env::var("SHRED_VERSION").unwrap_or_else(|_| "0".to_string());
...
.arg(
Arg::with_name("shred_version")
.long("shred-version")
.takes_value(true)
.default_value(&default_shred_version)
.help("Shred version of cluster to spam"),
)
...
```
When you deploy a cluster with your client, leave the `--shred-version` command out since it will be read via environment variable:
```
cargo run --bin cluster -- -n <namespace>
...
generic-client --docker-image <client-docker-image> --executable-path <path-to-executable-in-docker-image> --delay-start <seconds-after-cluster-is-deployed-before-deploying-client> --generic-client-args 'target-node=<ip:port>'
```

The following environment variables are available to each non-bootstrap pod:
```
NAMESPACE # cluster namespace
BOOTSTRAP_RPC_ADDRESS # rpc address of bootstrap node
BOOTSTRAP_GOSSIP_ADDRESS # gossip address of bootstrap node
BOOTSTRAP_FAUCET_ADDRESS # faucet address of bootstrap node
SHRED_VERSION # cluster shred version
```
^ More environment variables to come!

<a name="kubernetes_domain_name"></a>
### Node Naming Conventions in Kubernetes
Say you want to launch your client and send transactions to a specific validator. Kubernetes makes it easy to identify deployed nodes via `<kubernetes_domain_name>:<port>`. Node naming conventions:
```
<node-name>-service.<namespace>.svc.cluster.local:<port>
```
e.g. bootstrap validator RPC port can be reached with:
```
bootstrap-validator-service.<namespace>.svc.cluster.local:8899
```
and a standard validator can be reached with:
```
validator-service-<8-char-commit-or-version>-<validator-index>.<namespace>.svc.cluster.local:<port>
```
examples:
```
# w/ commit
validator-service-bd1a5dfb-7.greg.svc.cluster.local:8001
# or with version
validator-service-v1.18.16-4.greg.svc.cluster.local:8001
```
Say you want to deploy your client with `--target-node <validator-4>` which is running v1.18.16:
```
cargo run --bin cluster -- -n <namespace>
...
generic-client --docker-image <registry>/<image-name>:<tag> --executable-path <path-to-executable-in-docker-image> --delay-start <seconds-after-cluster-is-deployed-before-deploying-client> --generic-client-args 'target-node=validator-service-v1.18.16-4.greg.svc.cluster.local:8001'
```

## Kubernetes Cheatsheet
Create namespace:
Expand Down Expand Up @@ -207,4 +293,38 @@ kubectl exec -it -n <namespace> <pod-name> -- /bin/bash
Get information about pod:
```
kubectl describe pod -n <namespace> <pod-name>
```
```

## Containerize your Client
### Dockerfile Template
```
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y iputils-ping curl vim && \
rm -rf /var/lib/apt/lists/* && \
useradd -ms /bin/bash solana && \
adduser solana sudo
USER solana
COPY --chown=solana:solana ./target/release/<client-executable> /home/solana/
COPY --chown=solana:solana ./client-accounts/ /home/solana/client-accounts/
RUN chmod +x /home/solana/<client-executable>
WORKDIR /home/solana
```

### Build client image
```
cd <client-directory>
docker build -t <registry>/<image-name>:<tag> -f <path-to-Dockerfile>/Dockerfile <context-path>
# e.g.
cd client-spam/
docker build -t test-registry/client-spam:latest -f docker/Dockerfile .
```

### Push client image to registry
```
docker push <registry>/<image-name>:<tag>
# e.g.
docker push test-registry/client-spam:latest
```
115 changes: 110 additions & 5 deletions src/client_config.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,117 @@
use solana_sdk::pubkey::Pubkey;
use {
solana_sdk::pubkey::Pubkey,
std::{error::Error, path::PathBuf},
strum_macros::Display,
};

#[derive(Clone, Debug)]
pub struct ClientConfig {
#[derive(Clone, PartialEq, Debug)]
pub struct BenchTpsConfig {
pub num_clients: usize,
pub client_duration_seconds: u64,
pub client_type: String,
pub client_to_run: String,
pub bench_tps_args: Vec<String>,
pub client_wait_for_n_nodes: Option<usize>,
pub client_to_run: String,
pub client_target_node: Option<Pubkey>,
}

impl ClientTrait for BenchTpsConfig {
fn executable_path(&self) -> Result<Vec<String>, Box<dyn Error>> {
let command = vec!["/home/solana/k8s-cluster-scripts/client-startup-script.sh".to_string()];
Ok(command)
}

fn generate_client_command_flags(&self) -> Vec<String> {
let mut flags = vec![];

flags.push(self.client_to_run.clone()); //client to run
if !self.bench_tps_args.is_empty() {
flags.push(self.bench_tps_args.join(" "));
}

flags.push(self.client_type.clone());

if let Some(target_node) = self.client_target_node {
flags.push("--target-node".to_string());
flags.push(target_node.to_string());
}

flags.push("--duration".to_string());
flags.push(self.client_duration_seconds.to_string());

if let Some(num_nodes) = self.client_wait_for_n_nodes {
flags.push("--num-nodes".to_string());
flags.push(num_nodes.to_string());
}

flags
}
}

#[derive(Default, Clone, PartialEq, Debug)]
pub struct GenericClientConfig {
pub num_clients: usize,
pub client_duration_seconds: u64,
pub client_wait_for_n_nodes: Option<u64>,
pub args: Vec<String>,
pub image: String,
pub executable_path: PathBuf,
pub delay_start: u64,
}

impl ClientTrait for GenericClientConfig {
fn executable_path(&self) -> Result<Vec<String>, Box<dyn Error>> {
let exec_path_string = self
.executable_path
.clone()
.into_os_string()
.into_string()
.map_err(|err| {
std::io::Error::new(
std::io::ErrorKind::InvalidData,
format!("Invalid Unicode data in path: {:?}", err),
)
})?;
Ok(vec![exec_path_string])
}
fn generate_client_command_flags(&self) -> Vec<String> {
self.args.clone()
}
}

#[derive(Debug, Clone, PartialEq, Display)]
pub enum ClientConfig {
#[strum(serialize = "bench-tps")]
BenchTps(BenchTpsConfig),
#[strum(serialize = "generic")]
Generic(GenericClientConfig),
None,
}

impl ClientConfig {
pub fn num_clients(&self) -> usize {
match self {
ClientConfig::BenchTps(config) => config.num_clients,
ClientConfig::Generic(config) => config.num_clients,
ClientConfig::None => 0,
}
}

pub fn build_command(&self) -> Result<Vec<String>, Box<dyn Error>> {
match self {
ClientConfig::BenchTps(config) => config.build_command(),
ClientConfig::Generic(config) => config.build_command(),
ClientConfig::None => Err("Client config is None".into()),
}
}
}

pub trait ClientTrait {
fn executable_path(&self) -> Result<Vec<String>, Box<dyn Error>>;
fn generate_client_command_flags(&self) -> Vec<String>;
/// Build command to run on pod deployment
fn build_command(&self) -> Result<Vec<String>, Box<dyn Error>> {
let mut command = self.executable_path()?;
command.extend(self.generate_client_command_flags());
Ok(command)
}
}
38 changes: 19 additions & 19 deletions src/cluster_images.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use {
crate::{validator::Validator, ValidatorType},
crate::{node::Node, NodeType},
std::{error::Error, result::Result},
};

Expand All @@ -11,41 +11,41 @@ use {

#[derive(Default)]
pub struct ClusterImages {
bootstrap: Option<Validator>,
validator: Option<Validator>,
rpc: Option<Validator>,
clients: Vec<Validator>,
bootstrap: Option<Node>,
validator: Option<Node>,
rpc: Option<Node>,
clients: Vec<Node>,
}

impl ClusterImages {
pub fn set_item(&mut self, item: Validator, validator_type: ValidatorType) {
match validator_type {
ValidatorType::Bootstrap => self.bootstrap = Some(item),
ValidatorType::Standard => self.validator = Some(item),
ValidatorType::RPC => self.rpc = Some(item),
ValidatorType::Client(_) => self.clients.push(item),
pub fn set_item(&mut self, item: Node) {
match item.node_type() {
NodeType::Bootstrap => self.bootstrap = Some(item),
NodeType::Standard => self.validator = Some(item),
NodeType::RPC => self.rpc = Some(item),
NodeType::Client(_, _) => self.clients.push(item),
}
}

pub fn bootstrap(&mut self) -> Result<&mut Validator, Box<dyn Error>> {
pub fn bootstrap(&mut self) -> Result<&mut Node, Box<dyn Error>> {
self.bootstrap
.as_mut()
.ok_or_else(|| "Bootstrap validator is not available".into())
}

pub fn validator(&mut self) -> Result<&mut Validator, Box<dyn Error>> {
pub fn validator(&mut self) -> Result<&mut Node, Box<dyn Error>> {
self.validator
.as_mut()
.ok_or_else(|| "Validator is not available".into())
}

pub fn rpc(&mut self) -> Result<&mut Validator, Box<dyn Error>> {
pub fn rpc(&mut self) -> Result<&mut Node, Box<dyn Error>> {
self.rpc
.as_mut()
.ok_or_else(|| "Validator is not available".into())
}

pub fn client(&mut self, client_index: usize) -> Result<&mut Validator, Box<dyn Error>> {
pub fn client(&mut self, client_index: usize) -> Result<&mut Node, Box<dyn Error>> {
if self.clients.is_empty() {
return Err("No Clients available".to_string().into());
}
Expand All @@ -54,23 +54,23 @@ impl ClusterImages {
.ok_or_else(|| "Client index out of bounds".to_string().into())
}

pub fn get_validators(&self) -> impl Iterator<Item = &Validator> {
pub fn get_validators(&self) -> impl Iterator<Item = &Node> {
self.bootstrap
.iter()
.chain(self.validator.iter())
.chain(self.rpc.iter())
.filter_map(Some)
}

pub fn get_clients(&self) -> impl Iterator<Item = &Validator> {
pub fn get_clients(&self) -> impl Iterator<Item = &Node> {
self.clients.iter()
}

pub fn get_clients_mut(&mut self) -> impl Iterator<Item = &mut Validator> {
pub fn get_clients_mut(&mut self) -> impl Iterator<Item = &mut Node> {
self.clients.iter_mut()
}

pub fn get_all(&self) -> impl Iterator<Item = &Validator> {
pub fn get_all(&self) -> impl Iterator<Item = &Node> {
self.get_validators().chain(self.get_clients())
}
}
Loading

0 comments on commit 978c446

Please sign in to comment.