diff --git a/charts/hab/.gitignore b/charts/hab/.gitignore new file mode 100644 index 0000000..bf1e7d6 --- /dev/null +++ b/charts/hab/.gitignore @@ -0,0 +1,6 @@ +values.hab.yaml +# Various IDEs +.project +.idea/ +.vscode +.DS_Store \ No newline at end of file diff --git a/charts/hab/.helmignore b/charts/hab/.helmignore new file mode 100644 index 0000000..0e8a0eb --- /dev/null +++ b/charts/hab/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/charts/hab/Chart.yaml b/charts/hab/Chart.yaml new file mode 100644 index 0000000..205db51 --- /dev/null +++ b/charts/hab/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: hab +description: A Helm chart for a high availability bitcoin node +type: application +version: 1.0.0 +appVersion: "22.0" diff --git a/charts/hab/LICENSE b/charts/hab/LICENSE new file mode 100644 index 0000000..54fa4a4 --- /dev/null +++ b/charts/hab/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2022 GildedPleb + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/charts/hab/README.md b/charts/hab/README.md new file mode 100644 index 0000000..2f98d74 --- /dev/null +++ b/charts/hab/README.md @@ -0,0 +1,67 @@ +# High Availability Bitcoin Node - Helm Chart + +This is a rudimentary Helm Chart for a simple Highly Available Bitcoin node. At +present it supports multiple bitcoin implementations (bitcoind, btcd, bcoin), +architectures (arm64, amd64), deployment environments (cloud, bare-metal), and +networks (prod, test, simnet, etc). + +## Why? + +Bitcoin, at a macro level, is a fundamentally highly available system: though +one node might fail, the incentives in the system ensure that there are many +nodes still available. So why make a highly available node? Bitcoin is not, +however, highly available at the micro node level: a single node may experience +all kinds of individual down time, disruptions, and failures. + +## Install + +### Requirements + +| | | +| ---------- | ----- | +| Kubernetes | >1.23 | +| Helm | >3.8 | +| Longhorn | >1.2 | + +> You may want to design a 3+ host Kubernetes Cluster with this chart in mind +> Additionally, having host names will be helpful. + +``` +helm repo add gildedpled/hab +helm repo update +helm show values gildedpled/hab > values.hab.yml +``` + +Edit `values.hab.yml` according to [configuration](./configuration.md) options. +And [examples](./examples.md) for examples. + +``` +helm install hab gildedpled/hab -f values.hab.yaml +``` + +## Roadmap Feature list + +- Build images from repo locally, and automatically, and push to remote, via: + https://flavio.castelli.me/2020/09/16/build-multi-architecture-container-images-using-kubernetes/ +- Store images locally in a registry like docker registry +- Mirror source code repos locally +- Make all officially supported repos interoperable, aka, the same RPC calls + return the same data, in the same format, no matter if the load balancer + directs you to a bcoin node or a bitcoind node. + - Would probably mean it needs to be a bitcoin controller, more here: + https://leftasexercise.com/2019/10/14/building-a-bitcoin-controller-part-vi-managing-secrets-and-creating-events/ + - Might even need a front end +- Integrate with higher layers, like lightning, block explorers, coinjoins, + etc. +- Diversify out of Longhorn +- Full tor integration, like tor services, or ingress +- Add autoscaling... probably infeasible due to resource constraints +- Reduce log pollution + +## Contributors + +gildedpleb + +--- + +Be a Gilded Pleb. Run a HAB node. diff --git a/charts/hab/configuration.md b/charts/hab/configuration.md new file mode 100644 index 0000000..c9bf3e5 --- /dev/null +++ b/charts/hab/configuration.md @@ -0,0 +1,298 @@ +# Configuration + +This is the configuration context, values, and justification for a HAB node. + +## Context + +### Node Types + +Presently, there are three node types in [`node-types.yml`](node-types.yml). For +the sake of greedy development, you can not use `values.hab.yaml` to add new +types, and must either directly edit `node-types.yml` or submit a RP with a new +node type definition--please review our +[contributing documentation](./contributing.md) and submit a PR! + +#### Node Type Specification + +The general format is intended to be limited to values that are constant accross +all node implamentations of the same type, as such: + +```yml + # Req? | Type | Default | Notes + # ----- | --------------- | -------------- | -------------------------------------- +bitcoind: # Yes | Key | | Repository, cmd, or project name used as reference in the `type` key below. + repository: "ruimarinho/bitcoin-core" # Yes | String / url | | Image name MUST be multi-arch image + pullPolicy: "IfNotPresent" # No | String | IfNotPresent | + tag: "" # Maybe | String | | If the Bitcoin repo to use with this node type does not follow the Bitcoin Core semver, then tag will need to be defined + pullSecrets: [] # No | List | nil | + command: [] # No | List of Strings | nil | Entry command to run on startup, if the image does not provide one or it needs revising + env: # No | Dict | nil | + - name: BITCOIN_DATA # No | String | nil | + value: &data "/data/" # No | String | nil | + mount: # Yes | Dict | | Mount point path mapped against the command line option for setting the data directory to use + path: *data # Yes | String / path | | + setParam: "-datadir=" # Yes | String | | Must match valid commandline arg for this type of node + peers: # Yes | Dict | | The way the implementations allows you to add peers + multiArg: true # Yes | Bool | | With or without repeating the same argument or prodiving a list of peers to one argument + addParam: "-addnode=" # Yes | String | | Must match valid commandline arg for this type of node + ports: # Yes | String | | The Bitcoin port + p2pParam: "-port=" # Yes | String | | Must match valid commandline arg for this type of node + p2pPort: "8333" # Yes | String | | The Bitcoin port (unless it needs changing) + ... +``` + +## Values + +### Failover Hosts + +```yml +nPlusOneHosts: + - 'pi5' + - 'nuc2' + ... +``` + +`nPlusOneHosts` is a list of hosts (`kubernetes.io/hostname`) which are to be +reserved for scheduling in the event that regular or primary hosts (n) fail. If +set, the scheduler will not schedule pods to these hosts unless, and only +unless, the primary hosts fail for those pods, and they have no other "targeted" +hosts that are available to them. To not define `nPlusOneHosts`, comment it out. + +### Additional Peers + +```yml +additionalPeers: + - addressAndPort: '8.8.8.8:8333' + group: 'prod' + ... +``` + +`additionalPeers` allows you to add external peers to peer groups to ensure that +all the nodes in that group attempt to connect to that peer. To not define +`additionalPeers`, comment it out. + +### NodeList Values + +`nodeList` is a list of all the nodes that you would like to run, as determined +in your `values.yaml`. If you have the resources for it, it is broadly +expandable, simply add a new node set to the list. For naming, the combination +of `type` and `name` of each node, in a `{type}-{name}` string must be +universally unique. e.g.: + +VALID: + +```yml +nodeList: + - type: bitcoind + name: test + ... + - type: btcd + name: test + ... +``` + +VALID: + +```yml +nodeList: + - type: btcd + name: test1 + ... + - type: btcd + name: test2 + ... +``` + +INVALID: + +```yml +nodeList: + - type: btcd + name: test + ... + - type: btcd + name: test + ... +``` + +#### Specification + +Each node in the nodeList expects this structure: + +```yml + # Req? | Type | Default | Notes +nodeList: # ----- | --------------- | -------------- | -------------------------------------- + - enabled: true # No | Bool | True | + name: "simnet" # Yes | String | | {type}-{name} must be universally unique + group: "test" # No | String | "{name}-group" | A group to open peer connections with in the HAB node + type: "btcd" # Yes | String | | Must be from the list of node types in node-types.yml + replicas: 1 # No | Number | 1 | The number of nodes which meet this node definition (statefulset replicas) + targetHosts: ["nuc1"] # No | List of Strings | nil | In the case that we want to match bitcoin nodes to kubernetes hosts, matching `kubernetes.io/hostname` + isolateNode: false # No | Bool | True | If True, no other node, of any group, that is also isolateNode: True, will be scheduled to the same host. + participateInNPlusOne: false # No | Bool | False | Whether or not this set can utilize the nPlusOne host(s) if a host in this set is under duress or dead. + participateInBackups: false # No | Bool | False | See note below on Backing up + storageAmt: "30Gi" # Yes | String | | + args: # No | List of Strings | nil | All args to pass to the bitcoin node running on the pod. Bad args can prevent pods from starting. Any command line node arguments native to the bitcoin node instance can be passed in the additional arguments section, provided they are formatted correctly + - "--txindex" + - "--rpcuser=$(SIMNET_RPC_USER)" # NOTE: To pass a secret to a command line arg, set an ENV (below) and either add a secret cleartext, or reference it in a k8s secret, which is added before installing the chart. + - "--rpcpass=$(SIMNET_RPC_PASS)" + - "--simnet" + - "--miningaddr=sb1q..." + additionalServicePorts: # No | List of Dicts | nil | Define any additional services/ports that you would like to expose + - port: 28333 # No | Number | nil | + targetPort: 28333 # No | Number | nil | + protocol: TCP # No | Type | nil | + name: zmqblock # No | String | nil | Must be unique + envs: # No | List of Dicts | nil | Define any additional envs, either directly or as references to k8s secrets + - name: SIMNET_RPC_USER + valueFrom: + secretKeyRef: + name: simnet-rpc-secret + key: user + - name: SIMNET_RPC_PASS + value: password + readinessCMD: "btcclt" # No | String | nil | See note on readiness below + externalAccess: # No | Dict | nil | Whether or not we can access this node from outside the cluster + enabled: true # | The mapping here is the standard mapping for, generally speaking, any standard ingress, more docs here for the recommended nginx ingress : https://kubernetes.github.io/ingress-nginx/ + ingress: + className: "" + annotations: + kubernetes.io/ingress.class: "nginx" + cert-manager.io/cluster-issuer: "selfsigned-issuer" + hosts: + - host: btc.gilded.lan + paths: + - path: / + pathType: Prefix + portNumber: 8333 + tls: + - secretName: bitcoin-tls + hosts: + - btc.gilded.lan + resources: # No | Dict | nil | This increases chances charts run on environments with little resources, such as raspberry pis. More docs here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 100m + memory: 128Mi + ... +``` + +#### The rest.. + +The rest of `values.yml` is boilerplate Helm templateing. + +## HAB Node Configuration Notes and Justification + +### Deployment Type + +Each `nodeList` node is deployed as a kubernetes +[StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). +Generally speaking, this is more desirable than a standard deployments type, +after all, Bitcoin is a stateful application. What is more, it may be required +as some clients if deployed as Deployments, will lock the database while in use, +preventing multiple pods from accessing the same volume without onerous +development overhead. + +StatefulSets have some other advantages and drawbacks as well: + +**PRO**: + +- Replication is dramatically simplified via storage classes +- Volumes are not ephemeral +- Each bitcoin node has its own private and verified copy of the blockchain + +**CON**: + +- Scaling compute resources are dramatically constrained +- Node identity (such as rpc tls certs and the like) are not unified accross + resources without intervention +- (Unless there is some serious coding wizardry) nodes who we wish to connect + with all other nodes in their group must also connect to themselves, causing + a lot of log pollution. + +### Storage Class + +For each node set, we define a new storage class to allow for varying +definitions around backups. However, seeing as how this is the only real +tangible benefit to using Longhorn with this chart (for the time being) we may +move forward by making longhorn optional in a future release. + +#### Backups + +`participateInBackups: true/false` determines whether or not we should back up +the nodes volumes to NFS/S3. This presently requires Longhorn and a working +NSF/S3 bucket set up in Longhorn. If `participateInBackups: True`, we don't +restore volumes from backups automatically because its computationally +impractical; it might take 20 min, 2 hours, 10 hours, or 2 days for the chart to +deploy as it waits for resources to download from accross the internet or local +network--where it would usually take a few seconds, depending on hab node +resources. To restore from backup, consult +[Longhorn documentation](https://longhorn.io/docs/1.2.4/snapshots-and-backups/). + +### Pod Management Policy + +We define a `podManagementPolicy: "Parallel"` along with +`publishNotReadyAddresses: true` and we then check to make sure that all pods +have addresses in an initContainer before launching the nodes to ensure that +there are no DNS lookup failures when launching a node. Some implementations +will exit upon DNS lookup failure, thereby crashlooping that pods deployment, +which in turn will crash-loop other pods. The drawback of this method is that if +we want statefulset scaling-up assurances, we must do it manually. + +### Readiness Prob + +Lastly, we expose a readiness prob command to allow the user to make use of +readiness as they see fit for each deployment. Unfortunately, it does not seem +worthwhile, at this point in time, to determin readiness against all the various +types of deployment networks (livenet, simnet, testnet, etc) and also allow the +user to define what readiness means: does it mean that the network is fully +synced? Does it mean that the node is ready to validate a transaction? Or does +it simply mean that nothing is broken? etc. + +Below are suggested readiness probes for each deployment type which would switch +Kubernetes reporting to "Ready" once and only if the node is fully synced. + +**bitcoind:** + +```yml +readinessCMD: + "tail -n 500 /data/debug.log | grep 'UpdateTip: new best=' | tail -3 | grep + progress=1.0" +``` + +**bcoin:** + +```yml +readinessCMD: 'tail -n 1000 /.bcoin/debug.log | grep progress=100' +``` + +**btcd:** + +```yml +readinessCMD: + 'tail -n 100 /root/.btcd/logs/mainnet/btcd.log | grep ''SYNC: Processed'' | + tail -n 1 | awk ''{print $1"T"$2"."$16"T"$17}'' | awk -F. ''{ c="date -jf + %FT%T " $1 " +%s"; c | getline t1; close(c); m="date -jf %FT%T " $3 " +%s"; + m | getline t2;close(m); if( (t1 - t2) > 60) exit 1 }''' +``` + +> My god this one-liner is ugly: +> +> - `tail -n 100 /root/.btcd/logs/mainnet/btcd.log` Take the last 100 lines +> from the log +> - `grep ''SYNC: Processed''` Of those, keep only the lines that have "SYNC: +> Processed" in them +> - `tail -n 1` Take the last of those +> - `awk ''{print $1"T"$2"."$16"T"$17}''` Format the two dates that are in +> that log: when the log was published, and when the block was published +> - `awk -F. ''{ c="date -jf %FT%T " $1 " +%s"; c | getline t1; close(c); m="date -jf %FT%T " $3 " +%s"; m | getline t2;close(m);` +> Covert each date to unix time in seconds +> - `if( (t1 - t2) > 60) exit 1` If they differ by more than 1 minute (60 +> seconds) return an error and fail the readiness prob (blocks can +> absolutely be outside this window, adjust accordingly) + +## Examples + +See [Examples](./examples.md) diff --git a/charts/hab/contributing.md b/charts/hab/contributing.md new file mode 100644 index 0000000..3633fd0 --- /dev/null +++ b/charts/hab/contributing.md @@ -0,0 +1,182 @@ +# Contributing to HAB + +First and foremost, thank you! We appreciate that you want to contribute to HAB, +your time is valuable, and your contributions mean a lot to us. + +## Important! + +By contributing to this project, you: + +- Agree that you have authored 100% of the content +- Agree that you have the necessary rights to the content +- Agree that you have received the necessary permissions from your employer to + make the contributions (if applicable) +- Agree that the content you contribute may be provided under the Project + license(s) +- Agree that, if you did not author 100% of the content, the appropriate + licenses and copyrights have been added along with any other necessary + attribution. + +## Getting started + +**What does "contributing" mean?** + +Creating an issue is the simplest form of contributing to a project. But there +are many ways to contribute, including the following: + +- Updating or correcting documentation +- Feature requests +- Bug reports + +If you'd like to learn more about contributing in general, the +[Guide to Idiomatic Contributing](https://github.com/jonschlinkert/idiomatic-contributing) +has a lot of useful information. + +**Showing support for HAB** + +Please keep in mind that open source software is built by people like you, who +spend their free time creating things the rest the community can use. + +Don't have time to contribute? No worries, here are some other ways to show your +support for HAB: + +- star the [project](https://github.com/gildedpleb/HAB) +- tweet your support for HAB + +## Issues + +Please only create issues for bug reports or feature requests. Issues discussing +any other topics may be closed by the project's maintainers without further +explanation. + +Do not create issues about bumping dependencies unless a bug has been identified +and you can demonstrate that it effects this library. + +**Help us to help you** + +Remember that we’re here to help, but not to make guesses about what you need +help with: + +- Whatever bug or issue you're experiencing, assume that it will not be as + obvious to the maintainers as it is to you. +- Spell it out completely. Keep in mind that maintainers need to think about + _all potential use cases_ of a library. It's important that you explain how + you're using a library so that maintainers can make that connection and + solve the issue. + +_It can't be understated how frustrating and draining it can be to maintainers +to have to ask clarifying questions on the most basic things, before it's even +possible to start debugging. Please try to make the best use of everyone's time +involved, including yourself, by providing this information up front._ + +### Before creating an issue + +Please try to determine if the issue is caused by an underlying library, and if +so, create the issue there. Sometimes this is difficult to know. We only ask +that you attempt to give a reasonable attempt to find out. Oftentimes the readme +will have advice about where to go to create issues. + +Try to follow these guidelines: + +- **Avoid creating issues for implementation help** - It's much better for + discoverability, SEO, and semantics - to keep the issue tracker focused on + bugs and feature requests - to ask implementation-related questions on + [stackoverflow.com][so] +- **Investigate the issue** - Search for existing issues (open or closed) that + address the issue, and might have even resolved it already. +- **Check the readme** - oftentimes you will find notes about creating issues, + and where to go depending on the type of issue. +- Create the issue in the appropriate repository. + +### Creating an issue + +Please be as descriptive as possible when creating an issue. Give us the +information we need to successfully answer your question or address your issue +by answering the following in your issue: + +- **description**: (required) What is the bug you're experiencing? How are you + using this library/app? +- **OS**: (required) what operating system are you on? +- **version**: (required) please note the version of HAB are you using +- **error messages**: (required) please paste any error messages into the + issue, or a [gist](https://gist.github.com/) +- **extensions, plugins, helpers, etc** (if applicable): please list any + extensions you're using + +### Closing issues + +The original poster or the maintainers of HAB may close an issue at any time. +Typically, but not exclusively, issues are closed when: + +- The issue is resolved +- The project's maintainers have determined the issue is out of scope +- An issue is clearly a duplicate of another issue, in which case the + duplicate issue will be linked. +- A discussion has clearly run its course + +## Pull Requests + +PRs are also welcome, and follow similar policies as Issues stated above. Please +employ the following steps to create a PR: + +1. Fork this repo +2. Create a feature branch +3. Commit/Push to that branch +4. Submit a PR to this upstream master branch + +Please follow this simple template for a Pull Requests: + +``` +## PR Name + +A description of the pull request. + +## Checklist + +- [ ] All tests are passing (if any) +- [ ] New tests were created to address changes in pr (and tests are passing) +- [ ] Updated README and/or documentation, if necessary +- [ ] Added myself to the `contributors` section of README.md +- [ ] If adding a new node implementation: + - [ ] Image that the new node references are multi-architecture + - [ ] Code used for image building is verifiable (if applicable) + - [ ] A one line `readinessCMD` has been provided in the README which yields a "READY 1/1" state when the livenet node is fully synced +``` + +## Next steps + +**Tips for creating idiomatic issues** + +Spending just a little extra time to review best practices and brush up on your +contributing skills will, at minimum, make your issue easier to read, easier to +resolve, and more likely to be found by others who have the same or similar +issue in the future. At best, it will open up doors and potential career +opportunities by helping you be at your best. + +The following resources were hand-picked to help you be the most effective +contributor you can be: + +- The + [Guide to Idiomatic Contributing](https://github.com/jonschlinkert/idiomatic-contributing) + is a great place for newcomers to start, but there is also information for + experienced contributors there. +- Take some time to learn basic markdown. We can't stress this enough. Don't + start pasting code into GitHub issues before you've taken a moment to review + this [markdown cheatsheet](https://gist.github.com/jonschlinkert/5854601) +- The GitHub guide to + [basic markdown](https://help.github.com/articles/markdown-basics/) is + another great markdown resource. +- Learn about + [GitHub Flavored Markdown](https://help.github.com/articles/github-flavored-markdown/). + And if you want to really go above and beyond, read + [mastering markdown](https://guides.github.com/features/mastering-markdown/). + +At the very least, please try to: + +- Use backticks to wrap code. This ensures that it retains its formatting and + isn't modified when it's rendered by GitHub, and makes the code more + readable to others +- When applicable, use syntax highlighting by adding the correct language name + after the first "code fence" + +[so]: http://stackoverflow.com/questions/tagged/HAB diff --git a/charts/hab/examples.md b/charts/hab/examples.md new file mode 100644 index 0000000..6835c83 --- /dev/null +++ b/charts/hab/examples.md @@ -0,0 +1,524 @@ +# Examples + +A standard single raspberry pi node, 1 host, 1 node: + +```yml +nodeList: + - name: livenet + type: bitcoind + storageAmt: 1000Gi + ... +``` + +--- + +An ultra simple BCoin cluster with 5 nodes, and 1 - 5 hosts (though, if it's 1 +host, it would need 7.5 tb of available space...): + +```yml +nodeList: + - name: livenet + type: bcoin + storageAmt: 1500Gi + replicaCount: 5 + ... +``` + +--- + +A 3 host, 3 node cluster with 1 node each of Bitcoin Core, BCoin, and BTCD, all +networked together, and trying to reach a node in Chicago. + +```yml +additionalPeers: + - addressAndPort: "47.185.51.150:8333" # a random public node in Chicago + group: cluster +nodeList: + - name: livenet + type: bitcoind + storageAmt: 1000Gi + group: cluster + - name: livenet + type: bcoin + storageAmt: 1500Gi + group: cluster + - name: livenet + type: btcd + storageAmt: 3000Gi + group: cluster + ... +``` + +--- + +A cluster of 2 - 4 hosts, 3 nodes, with a failover host + +```yml +nPlusOneHosts: + - 'host4' +nodeList: + - name: livenet + type: bitcoind + storageAmt: 1000Gi + group: cluster + replicaCount: 2 + participateInNPlusOne: true + - name: livenet + type: bcoin + storageAmt: 1500Gi + group: cluster + participateInNPlusOne: true + ... +``` + +--- + +A 4 host, 3 node cluster with very uneven storage distribution per host, where +only bitcoind and bcoin can participate in nPlusOne + +```yml +nPlusOneHosts: + - 'host4-mid-hd' +nodeList: + - name: livenet + type: bitcoind + storageAmt: 600Gi + group: cluster + targetHosts: ["host1-small-hd"] + participateInNPlusOne: true + - name: livenet + type: bcoin + storageAmt: 1500Gi + group: cluster + targetHosts: ["host2-mid-hd"] + participateInNPlusOne: true + - name: livenet + type: btcd + storageAmt: 4000Gi + group: cluster + targetHosts: ["host3-large-hd"] + ... +``` + +--- + +A 3 node, 1 - 3 host simnet cluster with rpc enabled, and mapped in envs. Will +require a kubernetes secret `simnet-rpc-secret` + +```yml +nodeList: + - name: simnet + type: btcd + storageAmt: 20Gi + replicaCount: 3 + args: + - "--rpcuser=$(SIMNET_RPC_USER)" + - "--rpcpass=$(SIMNET_RPC_PASS)" + - "--simnet" + - "--miningaddr=sb1qjwgq9pa4df4qv2dqdlkaf6mvzhel7lus4yhl80" + envs: + - name: SIMNET_RPC_USER + valueFrom: + secretKeyRef: + name: simnet-rpc-secret + key: user + - name: SIMNET_RPC_PASS + valueFrom: + secretKeyRef: + name: simnet-rpc-secret + key: pass + ... + +``` + +--- + +A 3 node, 3 host cluster with indexing, disabled wallet, zmq, rpc clients, and +also full re-indexing and re-validation on all reboots. Additionally, the nodes +are isolated to prevent them from being scheduled to the same host. Bitcoind is +able to pass in plaintext rpc auth because it is an encoded secret--still +probably not advisable though. + +```yml +nodeList: + - name: livenet + type: bitcoind + storageAmt: 1200Gi + replicaCount: 3 + isolateNode: true + args: + - "-listen=1" + - "-server=1" + - "-txindex=1" + - "-disablewallet" + - "-maxconnections=40" + - "-rest=1" + - "-assumevalid=0" + - "-reindex" + - "-rpcbind=0.0.0.0" + - "-rpcauth=rpcadmin:4228cacb482f444deb7bc0eb0131362b$fe7f1d00cb80892194940e78703e142002f726f91fb53d3bd067e8728e3f01ca" + - "-zmqpubrawblock=tcp://0.0.0.0:28333" + - "-zmqpubrawtx=tcp://0.0.0.0:28332" + additionalServicePorts: + - port: 28333 + targetPort: 28333 + protocol: TCP + name: zmqblock + ... + +``` + +--- + +A 5 host, 3 node cluster with dedicated failover for bitcoind, and additional +failover for all nodes. (For the most part you wouldn't need to do this as two +bitcoind instances would be far more suitable. However, if the underlying +hardware on the bitcoind hosts is fragile and resource constrained in storage +terms, this may be effective.) + +```yml +nPlusOneHosts: + - 'host5-failover-all' +nodeList: + - name: livenet + type: bitcoind + storageAmt: 1200Gi + replicaCount: 1 + targetHosts: ["host1", "host2"] + participateInNPlusOne: true + group: cluster + isolateNode: true + - name: livenet + type: bcoin + storageAmt: 1500Gi + group: cluster + participateInNPlusOne: true + isolateNode: true + - name: livenet + type: btcd + storageAmt: 3000Gi + group: cluster + participateInNPlusOne: true + isolateNode: true + ... +``` + +--- + +A 3 host, 3 node livenet cluster, and a 3 node simnet cluster, on top of it. + +```yml +nodeList: + - name: livenet + type: btcd + storageAmt: 3000Gi + group: live-cluster + isolateNode: true + replicaCount: 3 + - name: testnet + type: btcd + storageAmt: 10Gi + group: test-cluster + targetHosts: ["host1", "host2", "host3"] + replicaCount: 3 + args: + - "--testnet" + ... +``` + +--- + +A 5 node, 1 - 5 host, simnet cluster with full tls ingress and loadbalancer +(will need the certificate `bitcoin-tls` and ClusterIssuer `selfsigned-issuer` +for this to work, and may also require rpc cert management -- but really, I just +don't have time to test this) + +```yml + - enabled: true + name: simnet + type: btcd + storageAmt: 10Gi + replicaCount: 5 + args: + - "--rpcuser=user" + - "--rpcpass=$(SIMNET_RPC_PASS)" + - "--simnet" + - "--rpclisten=127.0.0.1:18556" + - "--miningaddr=sb1qjwgq9pa4df4qv2dqdlkaf6mvzhel7lus4yhl80" + envs: + - name: SIMNET_RPC_PASS + value: cleartextSimnetPass + additionalServicePorts: + - port: 18556 + targetPort: 18556 + protocol: TCP + name: simnetrpc + externalAccess: + enabled: true + ingress: + className: "" + annotations: + kubernetes.io/ingress.class: "nginx" + cert-manager.io/cluster-issuer: "selfsigned-issuer" + hosts: + - host: btc.gilded.lan + paths: + - path: / + pathType: Prefix + portNumber: 8333 + - path: /rpc + pathType: Prefix + portNumber: 18556 + tls: + - secretName: bitcoin-tls + hosts: + - btc.gilded.lan + ... +``` + +--- + +An "even race" cluster: 3 identical hosts, 3 nodes with 1 node each of Bitcoin +Core, BCoin, and BTCD, where bcoin and btcd are networked together, and bitcoind +is isolated, and has to index everything. Because why not? + +```yml +nodeList: + - name: livenet + type: bitcoind + storageAmt: 3000Gi + isolateNode: true + args: + - "-txindex=1" + - name: livenet + type: bcoin + storageAmt: 3000Gi + group: cluster + isolateNode: true + - name: livenet + type: btcd + storageAmt: 3000Gi + group: cluster + isolateNode: true + ... +``` + +--- + +2 nearly identical bitcoind nodes, on 2 hosts, that are not networked. This is +essentially the same as adding `replicaCount: 2` to the first node, and deleting +the second, with one important difference: because these nodes are in different +peer groups, they will not talk to eachother (unless they talk to a remote peer +who shares their neighbors address). This is how to isolate peers from what +would normally be the same statefulset newtworking defaults. Also note that the +names need to change to pass validation. + +```yml +nodeList: + - name: livenet1 + type: bitcoind + storageAmt: 3000Gi + group: cluster1 + - name: livenet2 + type: bitcoind + storageAmt: 3000Gi + group: cluster2 + ... +``` + +--- + +HAB node design for a 7 node cluster with prod and dev capabilities + +| Host | Space | Name | Group | Test Simnet | +| ---- | ----------- | -------------------------------------- | ---------- | -------------- | +| nuc1 | 2tb + 4tb | btcd full node and ctl - fn, ctl | prod, prod | | +| nuc2 | 2tb | available for test - | | btcd simnet1 | +| pi1 | 2tb | bitcoind1 Main HA Cluster - ha | prod | btcd simnet2 | +| pi2 | 2tb | bitcoind2 Main HA Cluster - ha | prod | btcd simnet3 | +| pi3 | 2tb | bitcoind3 Main HA Cluster - ha | prod | btcd simnet4 | +| pi4 | 2tb | bcoin1 - fn | prod | btcd simnet5 | +| pi5 | 2tb | n+1 node - | | | + +```yml +nPlusOneHosts: + - "pi5" + +nodeList: + - enabled: true + name: ctl + group: prod + type: bitcoind + targetHosts: [nuc2] + isolateNode: true + replicaCount: 1 + storageAmt: 1700Gi + participateInNPlusOne: true + participateInBackups: true + args: + - "-listen=1" + - "-server=1" + - "-disablewallet" + - "-dbcache=30000" + - "-maxconnections=40" + - "-maxuploadtarget=5000" + - "-assumevalid=0" + - "-reindex" + readinessCMD: "tail -n 500 /data/debug.log | grep 'UpdateTip: new best=' | tail -3 | grep progress=1.0" + - enabled: true + name: ref + group: prod + type: bitcoind + targetHosts: [nuc1] + isolateNode: true + replicaCount: 1 + storageAmt: 1700Gi + participateInNPlusOne: true + participateInBackups: true + args: + - "-listen=1" + - "-server=1" + - "-disablewallet" + - "-dbcache=30000" + - "-maxconnections=40" + - "-maxuploadtarget=5000" + - "-assumevalid=0" + - "-reindex" + readinessCMD: "tail -n 500 /data/debug.log | grep 'UpdateTip: new best=' | tail -3 | grep progress=1.0" + - enabled: true + name: test + group: prod + type: btcd + targetHosts: [nuc1] + isolateNode: false + replicaCount: 1 + storageAmt: 4000Gi + participateInNPlusOne: false + participateInBackups: false + externalAccess: + enabled: false + readinessCMD: + 'tail -n 100 /root/.btcd/logs/mainnet/btcd.log | grep ''SYNC: Processed'' | + tail -n 1 | awk ''{print $1"T"$2"."$16"T"$17}'' | awk -F. ''{ c="date -jf + %FT%T " $1 " +%s"; c | getline t1; close(c); m="date -jf %FT%T " $3 " +%s"; + m | getline t2;close(m); if( (t1 - t2) > 60) exit 1 }''' + resources: + limits: + cpu: 1000m + memory: 1000Mi + requests: + cpu: 1000m + memory: 1000Mi + - enabled: true + name: test + group: prod + type: bcoin + targetHosts: [pi1] + isolateNode: true + replicaCount: 1 + storageAmt: 1700Gi + participateInNPlusOne: true + participateInBackups: false + args: + - "--cache-size=4096" + readinessCMD: "tail -n 1000 /.bcoin/debug.log | grep progress=100" + - enabled: true + name: ha + group: prod + type: bitcoind + isolateNode: true + replicaCount: 3 + targetHosts: [pi2, pi3, pi4] + storageAmt: 1700Gi + args: + - "-listen=1" + - "-server=1" + - "-txindex=1" + - "-disablewallet" + - "-dbcache=4000" + - "-maxconnections=40" + - "-maxuploadtarget=5000" + - "-rest=1" + - "-rpcbind=0.0.0.0" + - "-rpcallowip=10.0.0.0/8" + - "-rpcauth=rpcadmin:4228cacb482f444deb7bc0eb0131362b$fe7f1d00cb80892194940e78703e142002f726f91fb53d3bd067e8728e3f01ca" + - "-zmqpubrawblock=tcp://0.0.0.0:28333" + - "-zmqpubrawtx=tcp://0.0.0.0:28332" + readinessCMD: "tail -n 500 /data/debug.log | grep 'UpdateTip: new best=' | tail -3 | grep progress=1.0" + - enabled: false + name: simnet + type: btcd + storageAmt: 10Gi + replicaCount: 5 + args: + - "--rpcuser=user" + - "--rpcpass=$(SIMNET_RPC_PASS)" + - "--maxpeers=40" + - "--simnet" + - "--rpclisten=127.0.0.1:18556" + - "--miningaddr=sb1qjwgq9pa4df4qv2dqdlkaf6mvzhel7lus4yhl80" + targetHosts: [nuc2, pi1, pi2, pi3, pi4] + envs: + - name: SIMNET_RPC_PASS + value: superSecretPass! + additionalServicePorts: + - port: 18556 + targetPort: 18556 + protocol: TCP + name: simnetrpc + externalAccess: + enabled: true + ingress: + className: "" + annotations: + kubernetes.io/ingress.class: "nginx" + cert-manager.io/cluster-issuer: "selfsigned-issuer" + hosts: + - host: btc.gilded.lan + paths: + - path: / + pathType: Prefix + portNumber: 8333 + tls: + - secretName: bitcoin-tls + hosts: + - btc.gilded.lan +``` + +## Aren't some of these not actually Highly Available? + +For this repo, we are assuming a varied definition of High Availability (HA). +Broadly speaking, HA usually refers to software architecture that continues to +function regardless of certain, tollerable, degradations. As it would apply +here, it would traditionally mean a replicated Bitcoind node statefulset, one +replica on each host, with N+1 tolerances, public exposure via a load balancer +and ingress, and each node mapping this publicly reachable address to the +network. Such that, any remote peer contact to our HA node will be unable to +discern service degradation in the event of local host failures as we, locally, +simply route around the failed host. + +This is an excellent pattern, but we are not sure if that is the best/only way +to play this wholistically speaking. For instance, a load balancer may disrupt +the way that bitcoin handles in-transit peer-to-peer messaging accross +nodes/hosts--I could be wrong here, it bares putting more research into this +project and this v1 is excellent groundwork to that end. But more broadly +speaking, even if we achieve fully robust traditional HA, I think that it only +amounts to a tactic in the larger strategy of robust node operation; a tactic +along side running multiple implementations of the same bitcoin protocol, +running on a diversity or power sources, using a diversity of internet +providers, using a variety of host computers, ect. + +As such, some of the examples above are indeed not HA on traditional metrics in +the slightest. However, they remain highly available on different metrics. For +instance, any node which runs multiple implementations of the bitcoin protocol +on multiple hardware architectures from multiple supply chains, dramatically +reduces external threats (be they unlikely) of compromised code or components +which may result in downtime or worse, or internal threats like zero-days which +may affect one dependency in one repo, but not others. + +If the above traditional HA ideals are not reachable in this version, it is +already on the road map to make some kind of k8s controller that can map traffic +to nodes without worry about their underlying implementation language, logic, or +idiosyncrasies. Though the bitcoin protocol is universal accross all +implementations, running and interacting with any individual node is far from +unified. diff --git a/charts/hab/index.yaml b/charts/hab/index.yaml new file mode 100644 index 0000000..932ab72 --- /dev/null +++ b/charts/hab/index.yaml @@ -0,0 +1,3 @@ +apiVersion: v1 +entries: {} +generated: "2022-04-07T14:36:50.212145069-07:00" diff --git a/charts/hab/node-types.yml b/charts/hab/node-types.yml new file mode 100644 index 0000000..eb9b561 --- /dev/null +++ b/charts/hab/node-types.yml @@ -0,0 +1,50 @@ +bitcoind: + repository: "ruimarinho/bitcoin-core" + pullPolicy: "IfNotPresent" + tag: "" + pullSecrets: [] + command: [] + env: + - name: BITCOIN_DATA + value: &data "/data/" + mount: + path: *data + setParam: "-datadir=" + peers: + multiArg: true + addParam: "-addnode=" + ports: + p2pParam: "-port=" + p2pPort: "8333" +btcd: + repository: "gildedpleb/btcd" + pullPolicy: "IfNotPresent" + tag: "v0.22.0-beta" + pullSecrets: [] + command: [] + env: [] + mount: + path: "/root/.btcd" + setParam: "--datadir=" + peers: + multiArg: true + addParam: "--addpeer=" + ports: + p2pParam: "--listen=0.0.0.0:" + p2pPort: "8333" +bcoin: + repository: "gildedpleb/bcoin" + pullPolicy: "IfNotPresent" + tag: "v2.2.0" + pullSecrets: [] + command: ["bcoin"] + env: [] + mount: + path: "/.bcoin" + setParam: "--prefix=" + peers: + multiArg: false + addParam: "--nodes=" + ports: + p2pParam: "--port=" + p2pPort: "8333" diff --git a/charts/hab/templates/NOTES.txt b/charts/hab/templates/NOTES.txt new file mode 100644 index 0000000..2172250 --- /dev/null +++ b/charts/hab/templates/NOTES.txt @@ -0,0 +1,24 @@ +PRC Access + +There are a few ways to get RPC access to any node. Assuming that appropriate rpc creds have been set +when deploying the nodes: + +1. Get access via executing comands in the Pod directly. Assuming no custom RPC server is defined - + + kubectl -n bitcoin exec hab-btcd-simnet-0 -- btcctl --rpcuser=user --rpcpass=pass --simnet getinfo + +2. Additionally, you can port forward from the Pod you wish to connect to. Below will map a btcd + instances default RPC simnet port (18556) to localhost:18556 - you may need to adjust tls settings + or copy certs - + + Terminal 1: + kubectl -n bitcoin port-forward hab-btcd-simnet-0 18556:18556 + Terminal 2: + btcctl --rpcuser=user --rpcpass=pass --simnet getinfo + +3. Lastly, you can access RPC services by setting 'externalAccess.enabled: true' on a node set, defining + an ingress, adding a certificate (if needed), then copying the btcd generated RPC cert on the pod + to your local machine, and using it in RPC calls to the exposed service. However, if the replica + count for the node set is > 1 the loadbalancer will direct traffic all the nodes in the set, where + only one of the nodes will allow access for that cert. Alternatively, '--notls` options exist on + various node implementations, but this is less secure, and certainly not recomened for public facing nodes. diff --git a/charts/hab/templates/_helpers.tpl b/charts/hab/templates/_helpers.tpl new file mode 100644 index 0000000..09fcb9f --- /dev/null +++ b/charts/hab/templates/_helpers.tpl @@ -0,0 +1,131 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "HAB.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "HAB.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "HAB.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "HAB.labels" -}} +helm.sh/chart: {{ include "HAB.chart" . }} +{{ include "HAB.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "HAB.selectorLabels" -}} +app.kubernetes.io/name: {{ include "HAB.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "HAB.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "HAB.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +Create a map of groups, and for each group, create a kubernetes recognizable node address for each node in that group, and return all addresses as a list +*/}} +{{- define "HAB.peersInGroup" -}} +{{- $groups := dict }} +{{- range .Values.nodeList }} +{{- $defaultGroup := print .name "-group" -}} +{{- if (or .enabled ( not ( hasKey . "enabled" ))) }} +{{- $_ := set $groups ( .group | default $defaultGroup ) list }} +{{- end }} +{{- end }} +{{- range $_, $nodeSet := .Values.nodeList }} +{{- if (or .enabled (not (hasKey . "enabled"))) }} +{{- $defaultGroup := print .name "-group" -}} +{{- range $index := until (int (.replicaCount | default 1)) }} +{{- $current := get $groups ( $nodeSet.group | default $defaultGroup ) }} +{{- $new := append $current (printf "%s-%s-%s-%d.%s-%s-%s.bitcoin.svc.cluster.local:%s" (include "HAB.fullname" $) $nodeSet.type $nodeSet.name $index (include "HAB.fullname" $) $nodeSet.type $nodeSet.name (get ($.Files.Get "node-types.yml" | fromYaml ) $nodeSet.type).ports.p2pPort) }} +{{- $_ := set $groups ( $nodeSet.group | default $defaultGroup ) $new }} +{{- end }} +{{- end }} +{{- end }} + +{{- range .Values.additionalPeers }} +{{- if hasKey $groups .group }} +{{- $current := get $groups .group }} +{{- $new := append $current .addressAndPort }} +{{- $_ := set $groups .group ( $new | uniq ) }} +{{- end }} +{{- end }} + +{{ $groups | toYaml}} +{{- end }} + +{{/* +Validate that naming is unique +*/}} +{{- define "HAB.validateNames" -}} +{{- $names := list }} +{{- range .Values.nodeList }} +{{- if (or .enabled ( not ( hasKey . "enabled" ))) }} +{{- $names = append $names (printf "%s-%s-%s" (include "HAB.fullname" $) .type .name) }} +{{- end }} +{{- end }} +{{- if ne (len ($names | uniq)) (len $names) }} +{{ print "Error: Vallidation Error, each active node name must be unique, nodes names supplied: " ($names | toString) }} +{{- end }} +{{- end }} + +{{/* +Validate that nPlusOne hosts, if given, do not also have nodes that select the same hosts. +*/}} +{{- define "HAB.validateHosts" -}} +{{- $hosts := list }} +{{- range .Values.nodeList }} +{{- if (or .enabled ( not ( hasKey . "enabled" ))) }} +{{- if .targetHosts }} +{{- range .targetHosts }} +{{- $hosts = append $hosts . }} +{{- end }} +{{- end }} +{{- end }} +{{- end }} +{{- range .Values.nPlusOneHosts}} +{{- if has . $hosts }} +{{ print "Error: Vallidation Error, If a nPlusOneHosts is defined, each active node host must not include that host in its targetHosts list. Target Hosts supplied: " ($hosts | uniq | toString) ", nPlusOneHost: " . }} +{{- end }} +{{- end }} +{{- end }} diff --git a/charts/hab/templates/backuprecuringjob.yml b/charts/hab/templates/backuprecuringjob.yml new file mode 100644 index 0000000..043170b --- /dev/null +++ b/charts/hab/templates/backuprecuringjob.yml @@ -0,0 +1,13 @@ +apiVersion: longhorn.io/v1beta1 +kind: RecurringJob +metadata: + name: bitcoin-everyblock-1day + namespace: longhorn-system + labels: + {{- include "HAB.labels" . | nindent 4 }} +spec: + # Every 10 minutes starting 0 min after the hour, kept for 1 day + cron: "0/10 * * * ?" + task: "backup" + retain: 5 + concurrency: 1 diff --git a/charts/hab/templates/cert.yaml b/charts/hab/templates/cert.yaml new file mode 100644 index 0000000..b6349b5 --- /dev/null +++ b/charts/hab/templates/cert.yaml @@ -0,0 +1,13 @@ +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: bitcoin-tls + namespace: bitcoin +spec: + secretName: bitcoin-tls + dnsNames: + - "btc.gilded.lan" + issuerRef: + name: selfsigned-issuer + kind: ClusterIssuer + group: cert-manager.io diff --git a/charts/hab/templates/ingress.yaml b/charts/hab/templates/ingress.yaml new file mode 100644 index 0000000..bf36ad2 --- /dev/null +++ b/charts/hab/templates/ingress.yaml @@ -0,0 +1,66 @@ +{{- range $idx, $nodeSet := .Values.nodeList }} +{{- if (or .enabled (not (hasKey . "enabled"))) }} +{{- $name := printf "%s-%s-%s" (include "HAB.fullname" $) $nodeSet.type $nodeSet.name -}} +{{- if (($nodeSet.externalAccess).enabled) -}} +--- +{{- if and $nodeSet.externalAccess.ingress.className (not (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion)) }} + {{- if not (hasKey $nodeSet.externalAccess.ingress.annotations "kubernetes.io/ingress.class") }} + {{- $_ := set $nodeSet.externalAccess.ingress.annotations "kubernetes.io/ingress.class" $nodeSet.externalAccess.ingress.className}} + {{- end }} +{{- end }} +{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion -}} +apiVersion: networking.k8s.io/v1 +{{- else if semverCompare ">=1.14-0" $.Capabilities.KubeVersion.GitVersion -}} +apiVersion: networking.k8s.io/v1beta1 +{{- else -}} +apiVersion: extensions/v1beta1 +{{- end }} +kind: Ingress +metadata: + name: {{ $name }} + labels: + {{- include "HAB.labels" $ | nindent 4 }} + app: {{ $name }} + {{- with $nodeSet.externalAccess.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if and $nodeSet.externalAccess.ingress.className (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }} + ingressClassName: {{ $nodeSet.externalAccess.ingress.className }} + {{- end }} + {{- if $nodeSet.externalAccess.ingress.tls }} + tls: + {{- range $nodeSet.externalAccess.ingress.tls }} + - hosts: + {{- range .hosts }} + - {{ . | quote }} + {{- end }} + secretName: {{ .secretName }} + {{- end }} + {{- end }} + rules: + {{- range $nodeSet.externalAccess.ingress.hosts }} + - host: {{ .host | quote }} + http: + paths: + {{- range .paths }} + - path: {{ .path }} + {{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }} + pathType: {{ .pathType }} + {{- end }} + backend: + {{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }} + service: + name: {{ $name }} + port: + number: {{ .portNumber }} + {{- else }} + serviceName: {{ $name }} + servicePort: {{ .portNumber }} + {{- end }} + {{- end }} + {{- end }} +{{- end }} +{{- end }} +{{- end }} diff --git a/charts/hab/templates/service.yaml b/charts/hab/templates/service.yaml new file mode 100644 index 0000000..81ec885 --- /dev/null +++ b/charts/hab/templates/service.yaml @@ -0,0 +1,41 @@ +{{- range $idx, $nodeSet := .Values.nodeList }} +{{- if (or .enabled (not (hasKey . "enabled"))) }} +{{- $name := printf "%s-%s-%s" (include "HAB.fullname" $) $nodeSet.type $nodeSet.name -}} +--- +apiVersion: v1 +kind: Service +metadata: + name: {{ $name }} + labels: + {{- include "HAB.labels" $ | nindent 4 }} + app: {{ $name }} +spec: + {{- if (($nodeSet.externalAccess).enabled) }} + type: "LoadBalancer" + {{- else }} + type: "ClusterIP" + {{- end }} + ports: + {{- with (get ($.Files.Get "node-types.yml" | fromYaml) $nodeSet.type) }} + - port: {{ .ports.p2pPort }} + targetPort: {{ .ports.p2pPort }} + protocol: TCP + name: p2p + {{- end }} + {{- if $nodeSet.additionalServicePorts }} + {{- range $_, $p := $nodeSet.additionalServicePorts }} + - port: {{ $p.port }} + targetPort: {{ $p.targetPort }} + protocol: {{ $p.protocol }} + name: {{ $p.name }} + {{- end }} + {{- end }} + publishNotReadyAddresses: true + selector: + {{- include "HAB.selectorLabels" $ | nindent 4 }} + app: {{ $name }} + {{- if not $nodeSet.externalAccess }} + clusterIP: None + {{- end }} +{{ end }} +{{ end }} \ No newline at end of file diff --git a/charts/hab/templates/serviceaccount.yaml b/charts/hab/templates/serviceaccount.yaml new file mode 100644 index 0000000..fd1b5bf --- /dev/null +++ b/charts/hab/templates/serviceaccount.yaml @@ -0,0 +1,12 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "HAB.serviceAccountName" . }} + labels: + {{- include "HAB.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/hab/templates/snapshotrecuringjob.yml b/charts/hab/templates/snapshotrecuringjob.yml new file mode 100644 index 0000000..19e2f0d --- /dev/null +++ b/charts/hab/templates/snapshotrecuringjob.yml @@ -0,0 +1,13 @@ +apiVersion: longhorn.io/v1beta1 +kind: RecurringJob +metadata: + name: bitcoin-everyblock-1day-snap + namespace: longhorn-system + labels: + {{- include "HAB.labels" . | nindent 4 }} +spec: + # Every 10 minutes starting 0 min after the hour, kept for 1 day + cron: "0/10 * * * ?" + task: "snapshot" + retain: 90 + concurrency: 1 diff --git a/charts/hab/templates/statefulset.yaml b/charts/hab/templates/statefulset.yaml new file mode 100644 index 0000000..4cef1f2 --- /dev/null +++ b/charts/hab/templates/statefulset.yaml @@ -0,0 +1,204 @@ +{{- range $idx, $nodeSet := .Values.nodeList }} +{{- if (or .enabled (not (hasKey . "enabled"))) }} +{{- $name := printf "%s-%s-%s" (include "HAB.fullname" $) $nodeSet.type $nodeSet.name -}} +{{- $peerList := get (include "HAB.peersInGroup" $ | fromYaml) ($nodeSet.group | default (print $nodeSet.name "-group")) }} +--- +{{- include "HAB.validateNames" $ }} +{{- include "HAB.validateHosts" $ }} +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: {{ $name }} + labels: + {{- include "HAB.labels" $ | nindent 4 }} + app: {{ $name }} +spec: + serviceName: {{ $name }} + replicas: {{ $nodeSet.replicaCount | default 1 }} + podManagementPolicy: "Parallel" + selector: + matchLabels: + {{- include "HAB.selectorLabels" $ | nindent 6 }} + app: {{ $name }} + {{- if (or $nodeSet.isolateNode (not (hasKey $nodeSet "isolateNode"))) }} + type: hab-isolated + {{- end }} + template: + metadata: + {{- with $.Values.podAnnotations }} + annotations: + {{- toYaml $ | nindent 8 }} + {{- end }} + labels: + {{- include "HAB.selectorLabels" $ | nindent 8 }} + app: {{ $name }} + {{- if (or $nodeSet.isolateNode (not (hasKey $nodeSet "isolateNode"))) }} + type: hab-isolated + {{- end }} + spec: + {{- with (get ($.Files.Get "node-types.yml" | fromYaml) $nodeSet.type) }} + {{- with .pullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ include "HAB.serviceAccountName" $ }} + securityContext: + {{- toYaml $.Values.podSecurityContext | nindent 8 }} + {{- if (or $nodeSet.isolateNode (not (hasKey $nodeSet "isolateNode"))) }} + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + type: hab-isolated + {{- end }} + affinity: + nodeAffinity: + {{- if $nodeSet.targetHosts }} + {{- if and $nodeSet.participateInNPlusOne $.Values.nPlusOneHosts }} + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 50 + preference: + matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + {{- range $nodeSet.targetHosts }} + - {{ . }} + {{- end }} + - weight: 1 + preference: + matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + {{- range $.Values.nPlusOneHosts }} + - {{ . }} + {{- end }} + {{- else }} + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + {{- range $nodeSet.targetHosts }} + - {{ . }} + {{- end }} + {{- end }} + {{- else }} + {{- if and $nodeSet.participateInNPlusOne $.Values.nPlusOneHosts }} + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 50 + preference: + matchExpressions: + - key: kubernetes.io/hostname + operator: NotIn + values: + {{- range $.Values.nPlusOneHosts }} + - {{ . }} + {{- end }} + - weight: 1 + preference: + matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + {{- range $.Values.nPlusOneHosts }} + - {{ . }} + {{- end }} + {{- else }} + {{- if $.Values.nPlusOneHosts }} + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: NotIn + values: + {{- range $.Values.nPlusOneHosts }} + - {{ . }} + {{- end }} + {{- end }} + {{- end }} + {{- end }} + {{- if gt (len $peerList) 1 }} + initContainers: + - name: "init-{{ $.Chart.Name }}" + image: busybox:latest + command: + - 'sh' + - '-c' + - 'until {{ range $peerList }}nslookup {{ first (regexSplit ":" . -1) }} && {{ end }}true; do sleep 2; done' + {{- end }} + containers: + - name: {{ $.Chart.Name }} + securityContext: + {{- toYaml $.Values.securityContext | nindent 12 }} + image: "{{ .repository }}:{{ .tag | default $.Chart.AppVersion }}" + imagePullPolicy: {{ .pullPolicy }} + {{- if len .command }} + command: {{ .command }} + {{- end }} + {{- if or (len .env) (len ($nodeSet.envs | default list )) }} + env: {{ if len .env }}{{ .env | toYaml | nindent 12}}{{ end }}{{ if len ($nodeSet.envs | default list )}}{{ $nodeSet.envs | toYaml | nindent 12}}{{ end }} + {{- end }} + args: + # Add peers from group "{{ $nodeSet.group | default (print $nodeSet.name "-group") }}" (if any) + {{- if gt (len $peerList) 1 }} + {{- with .peers }} + {{- $ctx := . -}} + {{- if .multiArg }} + {{- range $peerList }} + - "{{ $ctx.addParam }}{{ . }}" + {{- end }} + {{- else }} + {{- $peers := "" }} + {{- range $peerList }} + {{- $peers = print $peers . "," }} + {{- end }} + - {{$ctx.addParam}}{{ trimAll "," $peers }} + {{- end }} + {{- end }} + {{- end }} + # Add additional args (if any) + {{- range $nodeSet.args }} + - {{ . | quote }} + {{- end }} + # Set appropriate ports + {{- with .ports }} + - "{{ .p2pParam }}{{ .p2pPort }}" + {{- end }} + # Set data directory + {{- with .mount }} + - "{{ .setParam }}{{ .path }}" + {{- end }} + volumeMounts: + - mountPath: {{ .mount.path }} + name: "volume-{{ $name }}" + {{- if $nodeSet.readinessCMD }} + readinessProbe: + initialDelaySeconds: 60 # 1 minute + periodSeconds: 600 # 10 minutes + exec: + command: + - sh + - -c + - {{ $nodeSet.readinessCMD | quote }} + {{- end }} + {{- if $nodeSet.resources }} + resources: + {{- toYaml $nodeSet.resources | nindent 12 }} + {{- end }} + volumeClaimTemplates: + - metadata: + name: "volume-{{ $name }}" + spec: + storageClassName: "sc-{{ $name }}" + accessModes: [ "ReadWriteOnce" ] + resources: + requests: + storage: {{ $nodeSet.storageAmt | default "Error: A valid storage amount is needed" }} + {{- end }} +{{- end }} +{{ end }} diff --git a/charts/hab/templates/storageclass.yml b/charts/hab/templates/storageclass.yml new file mode 100644 index 0000000..8425b5f --- /dev/null +++ b/charts/hab/templates/storageclass.yml @@ -0,0 +1,25 @@ +{{- range $idx, $nodeSet := .Values.nodeList }} +{{- if (or .enabled (not (hasKey . "enabled"))) }} +{{- $name := printf "%s-%s-%s" (include "HAB.fullname" $) $nodeSet.type $nodeSet.name -}} +--- +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: "sc-{{ $name }}" + labels: + {{- include "HAB.labels" $ | nindent 4 }} +provisioner: driver.longhorn.io +allowVolumeExpansion: true +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +parameters: + replicaAutoBalance: "disable" + dataLocality: "best-effort" + numberOfReplicas: "1" # Depending on implementation, a bitcoin instance may attain a permanent lock on the database as a security feature. This lock is replicated accross longhorn replication volume instances, meaning, even if you have a replication, you may not have access to it, depending on how the original node or host was shut down and how it comes back up. As such, it is far more reliable (and useful) to replicate full node instances than it is to only replicate the blockchain, and keep this set to `1`. + staleReplicaTimeout: "30" + fromBackup: "" + {{- if .participateInBackups }} + recurringJobSelector: '[{"name":"bitcoin-everyblock-1day","isGroup":false},{"name":"bitcoin-everyblock-1day-snap","isGroup":false}]' + {{- end }} +{{ end }} +{{ end }} \ No newline at end of file diff --git a/charts/hab/values.yaml b/charts/hab/values.yaml new file mode 100644 index 0000000..5fcc9e7 --- /dev/null +++ b/charts/hab/values.yaml @@ -0,0 +1,59 @@ +# # Default values for HAB. +# # This is a YAML-formatted file. +# # Declare variables to be passed into your templates. +# # See ./configuration.md and ./examples.md for a breakdown of how to use this values.yaml + + +####################### +## HAB NODE VALUES ## +####################### + + +# # WARNING: Depending on implementation, some nodes may fail to launch if dns names given here can not be resolved. +# # Leave commented out to not add any +# additionalPeers: +# - addressAndPort: "8.8.8.8:8333" +# group: "prod" + +# # Leave commented out to not add any +# nPlusOneHosts: +# - "host1" + +nodeList: + - enabled: true + name: simple + type: bitcoind + storageAmt: "1000Gi" + + +####################### +## BASIC BOILERPLATE ## +####################### + + +nameOverride: "" +fullnameOverride: "" + +serviceAccount: + # Specifies whether a service account should be created + create: true + # Annotations to add to the service account + annotations: {} + # The name of the service account to use. + # If not set and create is true, a name is generated using the fullname template + name: "" + +podAnnotations: {} + +podSecurityContext: + {} + # fsGroup: 2000 + +securityContext: + {} + # capabilities: + # drop: + # - ALL + # readOnlyRootFilesystem: true + # runAsNonRoot: true + # runAsUser: 1000