Skip to content

Commit

Permalink
Add tag and repo overrides, fix spelling
Browse files Browse the repository at this point in the history
  • Loading branch information
@GildedPleb
GildedPleb committed Dec 29, 2022
1 parent 29904d5 commit b21be37
Show file tree
Hide file tree
Showing 6 changed files with 63 additions and 40 deletions.
6 changes: 3 additions & 3 deletions charts/hab/Chart.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v2
name: hab
description: A Helm chart for a high availability bitcoin node
description: A Helm chart for a highly available bitcoin node
type: application
version: 1.0.2
appVersion: "22.0"
version: 1.1.1
appVersion: "1"
8 changes: 5 additions & 3 deletions charts/hab/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
# High Availability Bitcoin Node - Helm Chart

This is a rudimentary Helm Chart for a simple Highly Available Bitcoin node. At
present it supports multiple bitcoin implementations (bitcoind, btcd, bcoin),
architectures (arm64, amd64), deployment environments (cloud, bare-metal), and
present, it supports multiple bitcoin implementations (bitcoind, btcd, bcoin),
architectures (arm64, amd64), deployment environments (cloud, bare-metal), versions (latest, 0.22, 0.19, etc), and
networks (prod, test, simnet, etc).

## Why?
Expand All @@ -13,7 +13,7 @@ Bitcoin, at a macro level, is a fundamentally highly available system: though
one node might fail, the incentives in the system ensure that there are many
nodes still available. So why make a highly available node? Bitcoin is not,
however, highly available at the micro node level: a single node may experience
all kinds of individual down time, disruptions, and failures.
all kinds of individual downtime, disruptions, and failures.

## Install

Expand Down Expand Up @@ -59,6 +59,7 @@ helm install hab gildedpled/hab -f values.hab.yaml
- Full tor integration, like tor services, or ingress
- Add autoscaling... probably infeasible due to resource constraints
- Reduce log pollution
- Investigate making nodes that have 1 replica, and are not intending to scale, being deployed as a deployment instead of a stateful set.

## Contributors

Expand All @@ -67,3 +68,4 @@ gildedpleb
---

Be a Gilded Pleb. Run a HAB node.

44 changes: 23 additions & 21 deletions charts/hab/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ This is the configuration context, values, and justification for a HAB node.

Presently, there are three node types in [`node-types.yml`](node-types.yml). For
the sake of greedy development, you can not use `values.hab.yaml` to add new
types, and must either directly edit `node-types.yml` or submit a RP with a new
types, and must either directly edit `node-types.yml` or submit a PR with a new
node type definition--please review our
[contributing documentation](./contributing.md) and submit a PR!

#### Node Type Specification

The general format is intended to be limited to values that are constant accross
all node implamentations of the same type, as such:
The general format is intended to be limited to values that are constant across
all node implementations of the same type, as such:

```yml
# Req? | Type | Default | Notes
Expand All @@ -25,18 +25,18 @@ bitcoind: # Yes | Key |
pullPolicy: "IfNotPresent" # No | String | IfNotPresent |
tag: "" # Maybe | String | | If the Bitcoin repo to use with this node type does not follow the Bitcoin Core semver, then tag will need to be defined
pullSecrets: [] # No | List | nil |
command: [] # No | List of Strings | nil | Entry command to run on startup, if the image does not provide one or it needs revising
command: [] # No | List of Strings | nil | Entry command to run on startup, if the image does not provide one, or it needs revising
env: # No | Dict | nil |
- name: BITCOIN_DATA # No | String | nil |
value: &data "/data/" # No | String | nil |
mount: # Yes | Dict | | Mount point path mapped against the command line option for setting the data directory to use
path: *data # Yes | String / path | |
setParam: "-datadir=" # Yes | String | | Must match valid commandline arg for this type of node
setParam: "-datadir=" # Yes | String | | Must match valid command line arg for this type of node
peers: # Yes | Dict | | The way the implementations allows you to add peers
multiArg: true # Yes | Bool | | With or without repeating the same argument or prodiving a list of peers to one argument
addParam: "-addnode=" # Yes | String | | Must match valid commandline arg for this type of node
addParam: "-addnode=" # Yes | String | | Must match valid command line arg for this type of node
ports: # Yes | String | | The Bitcoin port
p2pParam: "-port=" # Yes | String | | Must match valid commandline arg for this type of node
p2pParam: "-port=" # Yes | String | | Must match valid command line arg for this type of node
p2pPort: "8333" # Yes | String | | The Bitcoin port (unless it needs changing)
...
```
Expand Down Expand Up @@ -126,15 +126,17 @@ nodeList: # ----- | --------------- | ---------
name: "simnet" # Yes | String | | {type}-{name} must be universally unique
group: "test" # No | String | "{name}-group" | A group to open peer connections with in the HAB node
type: "btcd" # Yes | String | | Must be from the list of node types in node-types.yml
repositoryOverride: "gildedpleb/btcd" # No | String | See Notes | Using the "type" defined above, it is possible to override the image repository and tag (below). This allows targeting Bitcoin release versions, **provided those images exist** AND that they match the type (don't mix 'btcd' images with type 'bitcoind'). If the images can not be pulled, the deploy will fail. Defaults are working images around Bitcoind 0.22, but depend on the 'type' above.
tagOverride: "v0.22.0-beta" # No | String | See Notes | See "repositoryOverride" immediately above
replicas: 1 # No | Number | 1 | The number of nodes which meet this node definition (statefulset replicas)
targetHosts: ["nuc1"] # No | List of Strings | nil | In the case that we want to match bitcoin nodes to kubernetes hosts, matching `kubernetes.io/hostname`
targetHosts: ["nuc1"] # No | List of Strings | nil | In the case that we want to match bitcoin nodes to Kubernetes hosts, matching `kubernetes.io/hostname`
isolateNode: false # No | Bool | True | If True, no other node, of any group, that is also isolateNode: True, will be scheduled to the same host.
participateInNPlusOne: false # No | Bool | False | Whether or not this set can utilize the nPlusOne host(s) if a host in this set is under duress or dead.
participateInNPlusOne: false # No | Bool | False | Whether this set can utilize the nPlusOne host(s) if a host in this set is under duress or dead.
participateInBackups: false # No | Bool | False | See note below on Backing up
storageAmt: "30Gi" # Yes | String | |
args: # No | List of Strings | nil | All args to pass to the bitcoin node running on the pod. Bad args can prevent pods from starting. Any command line node arguments native to the bitcoin node instance can be passed in the additional arguments section, provided they are formatted correctly
- "--txindex"
- "--rpcuser=$(SIMNET_RPC_USER)" # NOTE: To pass a secret to a command line arg, set an ENV (below) and either add a secret cleartext, or reference it in a k8s secret, which is added before installing the chart.
- "--rpcuser=$(SIMNET_RPC_USER)" # NOTE: To pass a secret to a command line arg, set an ENV (below) and either add a secret clear text, or reference it in a k8s secret, which is added before installing the chart.
- "--rpcpass=$(SIMNET_RPC_PASS)"
- "--simnet"
- "--miningaddr=sb1q..."
Expand All @@ -152,7 +154,7 @@ nodeList: # ----- | --------------- | ---------
- name: SIMNET_RPC_PASS
value: password
readinessCMD: "btcclt" # No | String | nil | See note on readiness below
externalAccess: # No | Dict | nil | Whether or not we can access this node from outside the cluster
externalAccess: # No | Dict | nil | Whether we can access this node from outside the cluster
enabled: true # | The mapping here is the standard mapping for, generally speaking, any standard ingress, more docs here for the recommended nginx ingress : https://kubernetes.github.io/ingress-nginx/
ingress:
className: ""
Expand Down Expand Up @@ -181,13 +183,13 @@ nodeList: # ----- | --------------- | ---------

#### The rest..

The rest of `values.yml` is boilerplate Helm templateing.
The rest of `values.yml` is Helm boilerplate.

## HAB Node Configuration Notes and Justification

### Deployment Type

Each `nodeList` node is deployed as a kubernetes
Each `nodeList` node is deployed as a Kubernetes
[StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/).
Generally speaking, this is more desirable than a standard deployments type,
after all, Bitcoin is a stateful application. What is more, it may be required
Expand All @@ -206,7 +208,7 @@ StatefulSets have some other advantages and drawbacks as well:
**CON**:

- Scaling compute resources are dramatically constrained
- Node identity (such as rpc tls certs and the like) are not unified accross
- Node identity (such as rpc tls certs and the like) are not unified across
resources without intervention
- (Unless there is some serious coding wizardry) nodes who we wish to connect
with all other nodes in their group must also connect to themselves, causing
Expand All @@ -221,20 +223,20 @@ move forward by making longhorn optional in a future release.

#### Backups

`participateInBackups: true/false` determines whether or not we should back up
`participateInBackups: true/false` determines whether we should back up
the nodes volumes to NFS/S3. This presently requires Longhorn and a working
NSF/S3 bucket set up in Longhorn. If `participateInBackups: True`, we don't
restore volumes from backups automatically because its computationally
impractical; it might take 20 min, 2 hours, 10 hours, or 2 days for the chart to
deploy as it waits for resources to download from accross the internet or local
deploy as it waits for resources to download from across the internet or local
network--where it would usually take a few seconds, depending on hab node
resources. To restore from backup, consult
[Longhorn documentation](https://longhorn.io/docs/1.2.4/snapshots-and-backups/).

### Pod Management Policy

We define a `podManagementPolicy: "Parallel"` along with
`publishNotReadyAddresses: true` and we then check to make sure that all pods
`publishNotReadyAddresses: true`, and we then check to make sure that all pods
have addresses in an initContainer before launching the nodes to ensure that
there are no DNS lookup failures when launching a node. Some implementations
will exit upon DNS lookup failure, thereby crashlooping that pods deployment,
Expand All @@ -245,11 +247,11 @@ we want statefulset scaling-up assurances, we must do it manually.

Lastly, we expose a readiness prob command to allow the user to make use of
readiness as they see fit for each deployment. Unfortunately, it does not seem
worthwhile, at this point in time, to determin readiness against all the various
worthwhile, at this point in time, to determine readiness against all the various
types of deployment networks (livenet, simnet, testnet, etc) and also allow the
user to define what readiness means: does it mean that the network is fully
synced? Does it mean that the node is ready to validate a transaction? Or does
it simply mean that nothing is broken? etc.
it simply mean that nothing is broken? Etc.

Below are suggested readiness probes for each deployment type which would switch
Kubernetes reporting to "Ready" once and only if the node is fully synced.
Expand Down Expand Up @@ -278,10 +280,10 @@ readinessCMD:
m | getline t2;close(m); if( (t1 - t2) > 60) exit 1 }'''
```

> My god this one-liner is ugly:
> This one-liner is ugly:
>
> - `tail -n 100 /root/.btcd/logs/mainnet/btcd.log` Take the last 100 lines
> from the log
> from the log--this reduces command execution time
> - `grep ''SYNC: Processed''` Of those, keep only the lines that have "SYNC:
> Processed" in them
> - `tail -n 1` Take the last of those
Expand Down
41 changes: 30 additions & 11 deletions charts/hab/examples.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Examples

A standard single raspberry pi node, 1 host, 1 node:
A standard single Raspberry Pi node, 1 host, 1 node:

```yml
nodeList:
Expand Down Expand Up @@ -70,7 +70,7 @@ nodeList:
---
A 4 host, 3 node cluster with very uneven storage distribution per host, where
only bitcoind and bcoin can participate in nPlusOne
only bitcoind and bcoin can participate in N+1
```yml
nPlusOneHosts:
Expand Down Expand Up @@ -98,7 +98,7 @@ nodeList:
---
A 3 node, 1 - 3 host simnet cluster with rpc enabled, and mapped in envs. Will
require a kubernetes secret `simnet-rpc-secret`
require a Kubernetes secret `simnet-rpc-secret`

```yml
nodeList:
Expand Down Expand Up @@ -295,9 +295,9 @@ nodeList:
2 nearly identical bitcoind nodes, on 2 hosts, that are not networked. This is
essentially the same as adding `replicaCount: 2` to the first node, and deleting
the second, with one important difference: because these nodes are in different
peer groups, they will not talk to eachother (unless they talk to a remote peer
peer groups, they will not talk to each other (unless they talk to a remote peer
who shares their neighbors address). This is how to isolate peers from what
would normally be the same statefulset newtworking defaults. Also note that the
would normally be the same statefulset networking defaults. Also note that the
names need to change to pass validation.

```yml
Expand All @@ -311,6 +311,25 @@ nodeList:
storageAmt: 3000Gi
group: cluster2
```
---

Diversify bitcoind release risk by deploying three nodes, each targeting a different release.

```yml
nodeList:
- name: livenet1
type: bitcoind
storageAmt: 3000Gi
tagOverride: "latest"
- name: livenet2
type: bitcoind
storageAmt: 3000Gi
tagOverride: "22.0"
- name: livenet3
type: bitcoind
storageAmt: 3000Gi
tagOverride: "0.19"
```

---

Expand Down Expand Up @@ -475,7 +494,7 @@ nodeList:

Yes. For this repo, we are assuming a varied definition of High Availability (HA).
Broadly speaking, HA usually refers to software architecture that continues to
function regardless of certain, tollerable, degradations. As it would apply
function regardless of certain, tolerable, degradation. As it would apply
here, it would traditionally mean a replicated Bitcoind node statefulset, one
replica on each host, with N+1 tolerances, public exposure via a load balancer
and ingress, and each node mapping this publicly reachable address to the
Expand All @@ -485,16 +504,16 @@ simply route around the failed host.

This is an excellent pattern, but we are not sure if that is the best/only way
to play this wholistically speaking. For instance, a load balancer may disrupt
the way that bitcoin handles in-transit peer-to-peer messaging accross
the way that bitcoin handles in-transit peer-to-peer messaging across
nodes/hosts--I could be wrong here, it bares putting more research into this
project and this v1 is excellent groundwork to that end. But more broadly
speaking, even if we achieve fully robust traditional HA, I think that it only
amounts to a tactic in the larger strategy of robust node operation; a tactic
along side running multiple implementations of the same bitcoin protocol,
alongside running multiple implementations of the same bitcoin protocol,
running on a diversity or power sources, using a diversity of internet
providers, using a variety of host computers, ect.
providers, using a variety of host computers, etc.

As such, some of the examples above are indeed not HA on traditional metrics in
As such, some examples above are indeed not HA on traditional metrics in
the slightest. However, they remain highly available on different metrics. For
instance, any node which runs multiple implementations of the bitcoin protocol
on multiple hardware architectures from multiple supply chains, dramatically
Expand All @@ -505,6 +524,6 @@ may affect one dependency in one repo, but not others.
If the above traditional HA ideals are not reachable in this version, it is
already on the road map to make some kind of k8s controller that can map traffic
to nodes without worry about their underlying implementation language, logic, or
idiosyncrasies. Though the bitcoin protocol is universal accross all
idiosyncrasies. Though the bitcoin protocol is universal across all
implementations, running and interacting with any individual node is far from
unified.
2 changes: 1 addition & 1 deletion charts/hab/node-types.yml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
bitcoind:
repository: "ruimarinho/bitcoin-core"
pullPolicy: "IfNotPresent"
tag: ""
tag: "22.0"
pullSecrets: []
command: []
env:
Expand Down
2 changes: 1 addition & 1 deletion charts/hab/templates/statefulset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ spec:
- name: {{ $.Chart.Name }}
securityContext:
{{- toYaml $.Values.securityContext | nindent 12 }}
image: "{{ .repository }}:{{ .tag | default $.Chart.AppVersion }}"
image: "{{ $nodeSet.repositoryOverride | default .repository }}:{{ $nodeSet.tagOverride | default .tag }}"
imagePullPolicy: {{ .pullPolicy }}
{{- if len .command }}
command: {{ .command }}
Expand Down

0 comments on commit b21be37

Please sign in to comment.