Error
Woops. Looks like this page doesn't exist ¯\_(ツ)_/¯.
diff --git a/404.html b/404.html index c25c322..8e21a5f 100644 --- a/404.html +++ b/404.html @@ -1,3 +1,3 @@ -
Woops. Looks like this page doesn't exist ¯\_(ツ)_/¯.
Woops. Looks like this page doesn't exist ¯\_(ツ)_/¯.
The Helper is a script or program written in any language that is in charge of handling the steps that are unique to your environment.
Essentially it receives Server Metadata on its STDIN and writes a response to STDOUT that is used to configure the server.
See the sections below for sample Helper scripts.
Let’s look at what you might receive as input on STDIN. The specifics will vary a bit between scenarios which we will call out in specific sections. The data you will get will all already have been validated. For example the JWT would have been parsed already and known to be valid.
{
"identity": "24bd22cdb279.choria.local",
"csr": null,
@@ -57,16 +57,16 @@
The configuration must set plugin.choria.server.provision
to disable provisioning, else the node will keep being reprovisioned forever.
Response keys used by scenarios:
Key | Description |
---|---|
defer | Defers the provisioning, this is a soft state meaning the server will come back and be retried later |
shutdown | Issues a shutdown on the server with exit code 0, systemd will not restart it. |
msg | A message to log on the Server to explain why it is being deferred or shut down |
configuration | A JSON Object of configuration items in key-value pairs, will be written to the server config |
action_policies | A JSON Object of Action Policy policies in key-value pairs, where the key is an agent name |
opa_policies | A JSON Object of Open Policy Agent policies in key-value pairs, where the key is an agent name or default |
When using the pki
feature used to enroll with a Certificate Authority:
Key | Description |
---|---|
key | An optional x509 private key that the server should use, will be encrypted using a unique one-time password in transit |
certificate | The signed certificate in PEM format |
ca | The Certificate Authority public key in PEM format |
ssldir | What directory to store the key, certificate and ca in on the server |
When using the jwt
feature to create server JWT tokens for Organization Issuer based networks:
Key | Description |
---|---|
server_claims | The Choria Server token claims to base the server JWT on |
When using the upgrades
feature to in-place upgrade servers:
Key | Description |
---|---|
upgrade | The version to upgrade the server to before provisioning |
Most typically you have a Enterprise Certificate Authority or you made your own using something like cfssl.
In this mode Choria Server will generate a private key on its disk, create a CSR and your helper will receive the CSR in PEM format. In your helper you then simply interact with your CA to sign the CSR and respond with the signed Certificate and CA public key. This way once provisioned your server will be fully enrolled for mTLS.
To enable the Provisioner to request the CSR the pki
feature needs to be enabled in the Provisioner Configuration.
The csr
input field will then be a JSON Object with:
Key | Description |
---|---|
csr | The PEM encoded CSR the node is sending |
public_key | The public part of the key the server created |
ssldir | The directory the server created the private key in so it can be used in the generated configuration |
Once you have this data you can use your CA API to enroll the node and get a signed certificate back. Simply put the resulting PEM data in the certificate
and ca
keys in the reply. You can set a SSL directory but typically just set ssldir
to what was received in the input.
A basic sample helper that enrolls in a cfssl
based CA can be seen in cfssl-helper.rb
In general the Private key stays on the node and you do not need it. Some Certificate Authorities require the private key to be accessible when signing a request. Provisioner support that, if you generate a key in the provisioner and add it to the reply in the key
JSON field a single use Shared Secret negotiated using Diffie-Hellman will be used to encrypt the key in transit.
I would not suggest ever to use a CA that requires you to transmit the Private Key during enrollment, it’s best to assume your CA is unusable at that point and consider a Organization Issuer based deployment.
In cases where a Certificate Authority is not available or it is operated in a way that makes it unsuitable for mTLS use you might opt to deploy Choria in an Organization Issuer based setup. The basic setup of that mode is out of scope for this document.
This applies only to Choria 0.27.0 and newer which is due to ship early 2023
To enrol nodes in an Organization Issuer based network you need to enable the ed25519
feature in the Provisioner Configuration.
Once enabled the ed25519_pubkey
field will hold a JSON Object with these values:
Key | Description |
---|---|
public_key | The ed25519 public key unique to the server |
directory | The directory that will hold server.seed and where server.jwt will be saved to later |
signature | A signature of the request made using the private key matching the public_key , this can be ignored as it would already be validated |
Once you received these you can include a server_claims
in your reply to give the server access to specific features in its JWT claims. The list here is correct for 0.27.0
. For an up-to-date list see the Go Documentation for your version of Choria.
The claims can include permissions
that have these properties.
Permission | Description |
---|---|
submission | Allows the server to use Choria Submission |
streams | Allows the server to access Choria Streams for example to read KV buckets from autonomous agents |
governor | Allows the server to access Choria Governor from autonomous agents |
service_host | Allows the server to host Services |
You can also use them to restrict it to a specific sub collective and more. Most values will default to sane defaults when not given.
Choria Server can be upgraded in-place to a new version. This is done by overwriting the binary at run-time with one downloaded from a specifically prepared repository.
This can only be done during Provisioning and requires the upgrades
feature to be enabled.
This applies only to Choria 0.27.0 and newer which is due to ship early 2023
Compatible nodes will have the upgradable
key in the inventory received and will be set to true
when upgrades are enabled. If you only want to support the latest nodes you can use this to determine if a upgrade is needed along with the version
key in the inventory. On nodes that are too old set shutdown
or defer
.
You would have to configure the Provisioner with a repository location and set up a repository on a HTTP server as per the guidelines from go-updater.
With all of this in place you can add the upgrade
key to the helper response that should just be a desired version like 0.28.0
. Provisioner will then attempt to upgrade the node.
Hosted at GitHub by R.I. Pienaar
Configuring the Provisioner requires exact knowledge of your deployment needs, we suggest you start with first configuring servers by hand while exploring your needs, once the specific configuration file contents and security needs are defined those can be configured into the Provisioner system.
Deploying a Provisioner should not be the first thing you do.
You must be running the Choria Broker and it will need a few extra configuration items.
You can enable Provisioning mode in the standard Open Source Choria Server by placing a provisining.jwt
in the right place. Those doing custom builds can also apply build-time defaults that has no external dependencies.
Provisioner calls into a user-supplier helper, written in a language like Ruby but any language can be used, to generate the per-node configuration properties.
If you have a x509 based Choria deployment (the default) you will need to be able to generate or sign certificates for your nodes. To do this your CA needs to have a API that you can call to issue certificates.
There are other options requiring JWT files to be signed also if that is not possible.
In this mode typically nodes do not have any Policies controlling who can invoke Agents and Actions. You will probably want to write a Open Policy Agent policy that the Provisioner will deploy
We expose metrics to Prometheus, for in-depth monitoring you will need Prometheus or a compatible system.
Hosted at GitHub by R.I. Pienaar
Provisioner is configured using /etc/choria-provisioner/choria-provisioner.yaml
typically. It’s a YAML format file with a few required settings and a number of optional ones. Changes to the file requires the process to be restarted.
As the Provisioner connects to the Choria Broker as a client it needs a configuration that allows it access. Create /etc/choria-provisioner/choria.cfg
with the following based on needs. These settings will augment those in /etc/choria/client.cfg
.
plugin.security.provider = file
plugin.security.file.certificate = /etc/choria-provisioner/ssl/cert.pem
plugin.security.file.key = /etc/choria-provisioner/ssl/key.pem
@@ -34,16 +34,16 @@
Item | Description | Default |
---|---|---|
workers | How many concurrent helpers to call while provisioning | number of cores |
interval | How often to perform a discovery against the network for new machines | 1m |
logfile | Where to write the log | |
loglevel | The level to log at, debug , info , warn or error | info |
helper | Path to the helper script | |
token | The value of the token set using --token in the provisioning.jwt | |
site | A unique name for this installation, surfaced in monitoring data | |
monitor_port | The post to listen on for monitoring requests | |
broker_provisioning_password | The password configured in the broker plugin.choria.network.provisioning.client_password | |
features.jwt | Enables fetching and validating provisioning.jwt , should almost always be true | false |
features.ed25519 | Enables JWT processing for Organization Issuer based networks | false |
features.pki | Enables x509 enrollment | false |
features.upgrades | Enables server version upgrades | false |
When enabled the Provisioner will fetch a CSR from the node and ask the node to create a private key that stays on the node. The helper can then get the certificate signed and the signed certificate will be sent to the node.
To enable set features.pki
to true
.
Here we reference our x509 key and certificate
Item | Description | Default |
---|---|---|
jwt_verify_cert | Full path to the public certificate used to sign provisioning.jwt | |
jwt_signing_key | Full path to our private key, also used in choria.conf |
When enabled the Provisioner will sign and issue server JWTs with custom claims and signatures. No x509 steps will be done.
To enable set features.ed25519
to true
.
Here we reference our JWT that gives us the right to provision and issue new JWTs
Item | Description | Default |
---|---|---|
jwt_verify_cert | The hex encoded public key of your Organization Issuer | |
jwt_signing_key | The private ed25519 key, also used in choria.conf | |
jwt_signing_token | The JWT token, also used in choria.conf |
Choria Provisioner can upgrade Choria Servers using a go-updated repository.
To enable set features.upgrades
to true
.
Here we configure the updates repository and say how failures are handled
Item | Description | Default |
---|---|---|
upgrades_repository | URL to your updates repository | |
upgrades_optional | Continue provisioning even if upgrading fails | false |
The Provisioner is generally fast enough to not need a cluster of active-active servers, so we support deploying multiple instances of Provisioner and using Leader Elections to elect one in the cluster as a leader that will be actively provisioning nodes.
To enable this the client needs access to Streams and Leader Elections so need --stream-user
and --elections-user
passed when creating its JWT.
Campaigning will be on a backoff schedule up to 20 second between campaigns, this means there can be up to a minute of downtime during a failover scenario, generally that’s fine for the Provisioner.
If a Provisioner was on standby and becomes leader it will immediately perform a discovery to pick up any nodes ready for provisioning.
Your broker must therefor have Choria Streams enabled.
Item | Description | Default |
---|---|---|
leader_election | Enables active-standby clusters | false |
Hosted at GitHub by R.I. Pienaar
The Choria Server has the required RPC Agent embedded and is ready to be used in a Provisioner managed network but it is disabled by default. Custom binaries can enable provisioning at compile time for an always-on experience.
Provisioning is enabled in the Open Source server by means of a JWT token that you create and place on the server. The JWT token holds all of the information the server needs to find it’s provisioning server and will present that token also to the provisioning server for authentication.
The token is signed using a trusted private key, the provisioner will only provision nodes presenting a trusted key.
If you are using a x509 based setup you really just need any RSA Key pair to sign the JWT with:
$ openssl genrsa -out provisioning-jwt-signer-key.pem 2048
$ openssl rsa -in provisioning-jwt-signer-key.pem -outform PEM -pubout -out provisioning-jwt-signer.pem
The public file will be placed on all your brokers to enable provisioning.
For the Organization Issuer based deploys you would sign it using your Issuer.
Choria JWTs are creating using the choria jwt
command, for provisioning specifically choria jwt provisioning
.
Creating the JWT:
$ choria jwt provisioning.jwt provisioning-jwt-signer-key.pem --srv choria.example.net --token toomanysecrets
@@ -46,16 +46,16 @@
Now provisioning is on with the settings we provided in the token.
Hosted at GitHub by R.I. Pienaar
In a typical Choria environment Puppet is used to provision a uniform Choria infrastructure integrated into the Puppet CA. This does not really work in large enterprises or dynamic environments..
Choria supports a provisioning mode where the process of enrolling a node can be managed on a per-environment basis, this is where we manage the vast differences in environments, platforms, etc. Once Choria is installed we have a unified overlay interface:
Choria Provisioner owns the early lifecycle of Choria Servers:
In essence this can replace the role of traditional Configuration Management with a more dynamic process for the purpose of configuring Choria Server. This is equivalent to an IoT device and it’s management.
Choria Provisioner is a very high performance system capable of provisioning 1,000 servers per minute assuming corporate x509 infrastructure is performant enough. It can be deployed in an active-standby cluster mode for high availability.
This project and the Choria authentication landscape in general, is in a period of flux as we move to support a fully Certificate Authority free deployment strategy.
This project can be used today, even by users deploying with Puppet and has proven to be stable and scalable. In a future deployment scenario it will be central to the scalable operation of Choria.
Hosted at GitHub by R.I. Pienaar
We distribute an RPM or Docker container for the Provisioner package. Debian is not supported at present.
For RPMs we publish releases but also nightly builds to our repositories.
Users of our Puppet modules will already have these repositories available.
[choria_release]
name=Choria Orchestrator Releases
mirrorlist=http://mirrorlists.choria.io/yum/release/el/$releasever/$basearch.txt
@@ -24,16 +24,16 @@
There is a docker container choria-io/provisioner
that has releases only.
Hosted at GitHub by R.I. Pienaar
Choria Server that is enabled for Provisioning expose a number of endpoints that can be used to interact with it in ways that is not otherwise possible.
These features are almost all extremely dangerous and should be used with great caution.
These abilities are not possible without enabling Provisioning in the server, so they are off by default. You should ensure your site security policies have adequate control over these actions.
This requires Choria 0.27.0 and newer which is due to ship early 2023
If you are running a new enough CLI and Server you can check if a running server supports version upgrades:
$ choria inventory 73ce620b5a21.choria.local
Inventory for 73ce620b5a21.choria.local
@@ -14,16 +14,16 @@
In this case the token
is the same as supplied when creating the JWT in the --token
argument. You can use any discovery filters and other options as shown here.
Servers will shut down after a short delay.
Hosted at GitHub by R.I. Pienaar
Choria Servers publish lifecycle
events, these can be viewed using choria tool event --component provision_mode_server
.
Servers will publish startup
, shutdown
and provisioned
events that will be shown in the rolling display. These lifecycle events are published in JSON mode on the network and could be stored in a Stream for analysis.
$ choria tool event --component provision_mode_server
Waiting for events from topic choria.lifecycle.event.> on nats://broker.choria.local:4222
14:35:46 [startup] n1.choria.local: provision_mode_server version 0.26.15
@@ -25,16 +25,16 @@
A re-election can be forced using choria election evict provisioner
.
The Provisioner keep statistics in the Prometheus format, using this you can observe deployed counts, performance of individual RPC calls and more.
All the statics have a site
label allowing you to easily aggregate a global cluster of AAA Services.
Statistic | Descriptions |
---|---|
choria_provisioner_rpc_time | How long each RPC request takes |
choria_provisioner_helper_time | How long the helper takes to run |
choria_provisioner_discovered | How many nodes are discovered using the broadcast discovery |
choria_provisioner_event_discovered | How many nodes were discovered due to events being fired about them |
choria_provisioner_discover_cycles | How many discovery cycles were ran |
choria_provisioner_rpc_errors | How many times a RPC request failed |
choria_provisioner_helper_errors | How many times the helper failed to run |
choria_provisioner_discovery_errors | How many times the discovery failed to run |
choria_provisioner_provision_errors | How many times provisioning failed |
choria_provisioner_paused | 1 when the backplane paused operations, 0 otherwise |
choria_provisioner_busy_workers | How many workers are busy processing servers |
choria_provisioner_provisioned | Host many nodes were successfully provisioned |
We have a published Grafana Dashboard for this statistics.
Hosted at GitHub by R.I. Pienaar
Hosted at GitHub by R.I. Pienaar