Note: This is documentation for the current development branch, see Stool 5 for the last stable branch.
Stool is a tool to manage stages - create, publish, configure, delete. A stage is a Kubernetes workload, typically a web application.
Changes: https://github.com/mlhartme/stool/blob/stool-7.0/CHANGELOG.md
Here's an example what you can do with Stool.
You invoke Stool from the command-line with sc followed by a command and specific arguments.
Open a terminal and run
sc context
to see available contexts, i.e. places where you can create stages. Notes:
- if you get a
command not found: scerror: Stool is not installed on your machine. Please refer to the install section below. - if you get a
settings not founderror: Stool is installed, but it's not set up. Please runsc setup.
Choose one of the available contexts by running
sc context <yourcontext>
Depending on your context you'll be asked to authenticate.
Create a new stage called tour running a hello application with
sc create tour hello
You can run
sc status tour
to see status information like urls or available pods. If available is above 0, you can point your browser
to one of the urls.
To delete the stage run
sc delete tour
You can create any number of stages. Invoke
sc list
to see them all.
Stool is technically a Helm wrapper, a stage is a Helm releases, and directions evaluate to Helm values. Like other tools (e.g. Helmfile, Stool tries to simplify Helm. Stool puts an emphasize on powerful value definitions because in our use cases we have many workloads using the same Helm chart with different values.
Stool 5 runs stages in Docker, Stool 6 switches to Kubernetes. The general goal is to shrink Stool by replacing Stool functionality with standard Kubernetes features/tools. Here's an outline what has already been replaced and what's planned or open:
| Functionality | Moved to; obsolete after | |
|---|---|---|
| Stool 5 - current stable | Process Management | Docker |
| Stool 6 - superseded by 7 | Port Management | Kubernetes |
| (Cpu limits) | Kubernetes Pod Limits | |
| (node selection) | Kubernetes | |
| Stool 7 - current development | Image Building | any tool, e.g. Maven Plugin |
| (K8s Resources implementing a stage) | Helm chart | |
| Stage lifecycle | Helm | |
| Memory limits | Kubernetes Pod Limits | |
| Vault integration | configurable script | |
| Certificate generation | configurable script | |
| Stool X - thoughts only | Basic monitoring | Prometheus? |
| Disk Limits | Kubernetes ephemeral storage limits | |
| Expire Stages | K8s Cronjob? | |
| Simple cli: reduce cognitive load | train developers + operating | |
| Dashboard: None-technical UI | ? | |
| Proxy: restricted access | All stages by Jenkins? | |
| Directions | Helmfile? Shell scripts? |
- Stool is written with a capital S
type writer fontmarks things to type or technical terms from Stool.- italics mark text to be replaced by the user
- bold face highlights term in definition lists
- synopsis syntax:
[]for optional,|for alternatives,...for repeatable,type writerfor literals, italics for replaceables) - WORKSPACE denotes the workspace currently used
This term is overloaded, depending on the context it may refer to the command line client sc, a Stool server running
in Kubernetes, or the whole Github project.
A stage is a Kubernetes workload, typically running a web application like a Tomcat servlet container with a Java web application.
You can create, list and delete stages with the respective Stool command. Every stage has a configuration, which is a set of
variables you can get and set with sc config. Every stages has a status, which is a set of fields; you can check it
with sc status.
A stage has a context and a unique name in that context. A stage is referenced by name@context or just the name
if it's in the current context. You define the stage name and context when you create the stage, neither can be changed
later.
Technically, a stage is a Helm release -- sc create installs a Helm chart, sc delete uninstalls it, sc publish and sc config
upgrade it. Stage variables are Helm release values (or Helm release variables, as they occasionally call it).
It's safe to use helm uninstall instead of sc delete.
A context specifies a place that can host stages together with the necessary authentication. Stool supports Kubernetes contexts and proxy contexts.
Use Kubernetes contexts if you have direct access to Kubernetes. ~/.kube/config defines the available Kubernetes contexts,
each specified with a name and a cluster url.
A proxy context has a name, an optional token, and a URL pointing to a Stool server. It's to give restricted access to Kubernetes
only, and to centrally manage Stool settings. sc manages a list of proxy contexts in its settings.
Stool manages a current context in its settings. You can change it permanently with sc context or per-invocation
with the -context global option.
Directions augment charts to define workloads.
Directions (or, more explicitly: the direction list) define how to create and publish stages. Stage directions are the directions the stage was created or last published with.
Directions look something like this:
DIRECTIONS: "hello"
EXTENDS: "kutter"
image: "myregistry/hello:1.0.0"
cert:
private: true
expr: "script.cert(stage)"
Directions have a subject (to allow others to reference it) and can extend other direction (i.e. inherit all directions). Each direction has a name and an expression.
Directions define the available variables of the stage, the expression is evaluated to determine the initial values.
The expression is denoted as a string. You can use Freemarker templating in it, and every chartkit script is available as a Freemarker function.
Applications usually define directions in a directions.yaml file and attach them to images in a directions label.
In addition, Stool is usually configured with pre-defined directions in its chartkit.
Technically, directions specify a Helm chart and how to evaluate its values. I.e. directions supply all info needed for Helm install/upgrade.
The chartkit defines a set of charts, scripts and directions. TODO
Stages are configured via variables, the set of variables of a stage is called its configuration. You can inspect and adjust variables with stool config.
A variable has a name, a value, and an associated direction. Stage directions define the available variables.
The value is evaluated by the associated direction expression when the stage is created, published
or configured. However, you can configure a fixed value by passing an assignment to create, publish or config.
In this case, the value remains unchanged until you configure a new value or remove it with key-
Variables apply to one stage only, every stage has its own set of variables, even if the associated directions are the same.
Distinguish variables from status fields: every stage has status fields, you can view them with sc status.
Status fields are key/values pairs like variables, but they are read-only.
TODO: common variables: metadataExpire, metadataContact, metadataComment, replicas, ...
Technically, variable values are Helm values of the respective Helm release.
Every stage has an metadataExpire variable that specifies the date until the stage is needed. You can see the expire date with sc config metadataExpire.
If this date has passed, the stage is called expired, and it is automatically stopped, a notification email is sent, and you cannot start it
again unless you specify a new date with sc config metadataExpire=yyyy-mm-dd.
Depending on the autoRemove setting, an expired stage will automatically be removed after the configured number of days.
Stage expiring helps to detect and remove unused stages, which is handy (and sometimes even crucial) if you are not the only user of a server. If you receive an email notification that your stage has expired, please check if your stage is still needed. If so, adjust the expire date. Otherwise, remove the stage.
A workspace is a list of stages referenced by a workspace name. Use workspaces if you need to run the same commands on the a list of stages,
Add stages to workspace by passing a workspace to sc create or with sc attach. Remove stages from workspaces with sc detach .
To use a workspace for a command, invoke it with @ workspace instead of the stage name
Example: Suppose you have two stage, you can run status on them like this:
sc attach one @ws
sc attach two @ws
sc status @ws
Stool is configured via settings specified in its settings.yaml file. A setting is a key/value pair. Value has a type
(string, number, date, boolean, list (of strings), or map (string to string)). Settings are global, in contrast to variables,
they are not specific for a stage. Settings are usually adjusted by system administrators.
TODO: more available settings, chartkit etc ...
Local settings
- chartkit: path (starting with a
/) to a local chartkit checkout, or image containing the chartkit
The dashboard is the UI that's part of Stool server.
Stage control
sc global-option... [command argument...]
sc is a command line tool to manage stages. A stage is a Kubernetes workload, typically a web application.
command defaults to help. sc stands for stage control.
Technically, a stage is a Helm release; sc is a wrapper for Helm that adds directions, proxying and a dashboard.
sc global-option... [command argument...]
sc global-option... help [command]
sc global-option... version
sc global-option... setup [-proxyPrefix prefix][key=value ...]
sc global-option... context [-q][-offline][context]
sc global-option... auth [-batch]
sc global-option... list [stage] (field|variable)...
sc global-option... create [-optional][-wait] name directions ['@'workspace] [key=value | key- ...]
sc global-option... publish ['-dryrun'] stage [directions] [key=value | key- ...]
sc global-option... config stage [key | key=value | key- ...]
sc global-option... status stage (field|value)...
sc global-option... delete stage [-batch]
sc global-option... history stage
sc global-option... attach stage '@'workspace
sc global-option... detach stage '@'workspace
sc global-option... images repository
sc global-option... port-forward stage [local-port] remote-port
sc global-option... ssh [-timeout minutes] stage [shell]
sc global-option... validate [-email] [-repair] stage
sc server
stage = %all | @workspace | predicate
Global options are:
-v enables verbose output
-e prints stacktrace for all errors
-context context sets the current context for this invocation
-fail mode see below
If you specify multiple stage for one command, you might want to specify what to do if the command
fails for some of them. That's what -fail mode is for.
Mode normal reports problems immediately and aborts execution, Stool does not try to run the command
on remaining matching stages. This is the default.
after reports problems after the command was invoked on all matching stages.
never is similar to after, but reports warnings instead of errors (and thus, Stool always returns with exit code 0).
SC_OPTS to configure arguments sc passes to the underlying JVM.
SC_HOME to configure Stool home directory. Defaults to $HOME/.sc
Homepage: https://github.com/mlhartme/stool
Invoke sc help command to get help for the specified command.
Display man page
sc global-option... help [command]
Display help about the specified command. Or, if command is not specified, display general sc help.
Display version info
sc global-option... version
Prints sc's version info and, if the current context is a proxy context, the server version.
See sc help for available global options
Setup Stool
sc global-option... setup [-proxyPrefix prefix][key=value ...]
Creates a fresh Stool home directory or reports an error if it already exists. The location of the home directory is configurable with
the SC_HOME environment variable, it defaults to ~/.sc. The main configuration file inside this directory is settings.yaml.
Stool checks for an environment variable SC_SETUP_SETTINGS. If it exists, the referenced file defines the initial settings (using prefix
to prefix all proxy names (default is ""). Otherwise, default settings are generated.
Use key/values pairs to configure local settings or proxies.
Create a home directory with defaults: sc setup
Create a home directory with a custom chartkit: sc setup chartkit=/my/path/to/chartkit
Manage current context
sc global-option... context [-q][-offline][context]
When called without arguments: lists all contexts with an arrow pointing to the current one.
Prints just the current context when called with -q.
Changes the current context when invoked with a context argument. If the new context requires authentication, this command implicitly
runs sc auth to get the respective token. This can be disabled by specifying -offline.
Authenticate to current proxy context
sc global-option... auth [-batch]
Asks for username/password to authenticate to the current context. Reports an error when used with a Kubernetes context.
Proxy contexts authenticate with username/password against ldap. If successful, the referenced Stool server returns an api token that will be stored in the settings file and used for future access to this context.
Use the -batch option to omit asking for username/password and instead pick them from the environment
variables STOOL_USERNAME and STOOL_PASSWORD.
See sc help for available global options
List stages
sc global-option... list [stage] (field|variable)...
Displays status of all stages of the current context (or the stages specified by stage) as a table.
See the status command for a list of available fields. Default fields/variables are name origin last-deployed.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Create a new stage
sc global-option... create [-optional][-wait] name directions ['@'workspace] [key=value | key- ...]
Creates a new stage as defined by directions: evaluates all direction expressions of the stage, except those key-value pairs specified explicitly. The resulting values define the stage configuration. This configuration is passed to Kubernetes to setup the workload.
name specifies the stage name. It must contain only lower case ascii characters or digit or dashes, it's rejected otherwise because it would cause problems with urls or Kubernetes objects that contain the name.
directions is a reference to the directions for this stage. Directions can be referenced in three ways:
- a path pointing to a local yaml file containing the directions; this path has to start with a
/or a. - an image with a
directionslabel containing a base64 encoded directions yaml - a directions name defined in the configured chartkit
Note that previous changes by the config command get lost unless you repeat them in the assignment arguments.
You'll normally call publish without arguments and thus get the configuration as defined by the directions.
Except for temporary assignments, you should adjust the directions.
If a workspace is specified, the resulting stage is added to it.
Specify -wait to wait for pods to become available before returning from this command.
Reports an error if a stage already exists. Or omits stages creation if the -optional option is specified.
See sc help for available global options
Publish a stage
sc global-option... publish ['-dryrun'] stage [directions] [key=value | key- ...]
Similar to create, but updates stage instead of creating a new one. Fixed values (either specified by an assignment or fixed by a previous call) remain unchanged, all other values are newly evaluates by the associated direction. The workload is updated without downtime.
directions is optional - if not specified, the stage directions are used. This is useful to cause a re-start (depends on the workload if that works).
Publish prints the variables modified by the command. Invoke with -dryrun to see
this diff without actually changing anything.
Publishing is refused if your stage has expired. In this case, publish with a new expire value.
TODO: Publishing is refused if the user who built the image does not have access to all fault projects referenced by the image.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Manage stage configuration
sc global-option... config stage [key | key=value | key- ...]
Stage configuration is the set of its variables. This command gets or sets stage variables.
When invoked without arguments, all variables are printed, together with the documentation available. When invoked with one or more keys, the respective variables are printed. When invoked with one or more assignments or minus, the respective variable values a fixed to the specified value or a fixed value is removed.
A fixed variable sticks to specified value; in contrast: none-fixed variables are re-evaluated by the publish or config commands.
str may contain {} to refer to the previous value. You can use this, e.g., to append to a value:
sc config "metadataComment={} append this".
If you want to set a variable to a string with spaces, you have to use quotes around the assignment.
Variables have a type: boolean, number, date, string, or list of strings.
Boolean values may be true or false, case sensitive.
Date values have the form yyyy-mm-dd, so a valid metadataExpire value is - e.g. -2016-12-31. Alternatively,
you can specify a number which is shorthand for that number of days from now (e.g. 1 means tomorrow).
List values (e.g. metadataContact) are separated by commas, whitespace before and after an item is ignored.
If you change a variable a that's referenced by the direction of another variable b, variable b is re-evaluate and might
change as well, even if it's not explicitly mentioned in the command line.
Technically, variables are modified by running Helm upgrade with both modified an unmodified values.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
The following variable are mandatory for all stages: https://github.com/mlhartme/stool/blob/stool-7.0/src/main/resources/stage.yaml
To see all variables for a given stage, along with the documentation available, run sc config givenStage
sc config metadataComment prints the current comment value.
sc config metadataComment=42 sets the comment to 42.
Display stage status
sc global-option... status stage (field|value)...
Prints the specified status fields or values. Default: all status fields except directions.
Available fields:
- name Name of the stage.
- available Number of available replicas.
- urls Urls for this stage. Point your browser to one of them to access your stage.
- first-deployed When this stage was created.
- last-deployed When this stage was last published or re-configured.
- cpu Cpu usage reported by Docker: percentage of this container's cpu utilisation relative to total system utilisation.
- mem Memory usage reported by Docker
- images Available images in repository for this stage.
- chart Helm chart the defines this stage.
- directions The directions for this stage.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Delete a stage
sc global-option... delete stage [-batch]
Deletes the stage, i.e. deletes it from the respective cluster. If stage is specified as a workspace, it is removed from the workspace as well.
Before actually touching anything, this command asks if you really want to delete the stage.
You can suppress this interaction with the -batch option.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Display commands invoked on this stage
sc global-option... history stage
Prints the sc commands that affected the stage. Invoke it -v to see more details for each invocation.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Attach stage to a workspace
sc global-option... attach stage '@'workspace
Attaches the specified stage to workspace. Creates a new workspace if the specified one does not exist.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Detach a stage from a workspace
sc global-option... detach stage '@'workspace
Removes stages from workspace without modifying the stage itself. Removes the workspace if it becomes empty.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Display images with labels
sc global-option... images repository
Display info about the images in the specified repository.
TODO: this command is awkward, dump and suggest some standard-tool instead?
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Start port forwarding
sc global-option... port-forward stage [local-port] remote-port
Starts port-forwarding of port local-port on localhost (default: remote port) to remote-port on the currently running port of the stage. Reports an error, if the stage is not running. Forwarding is refused if the current user does not have access to all fault projects of the image. Forwarding is terminated manually by pressing ctrl-c or automatically after timeout minutes (default: 30).
Examples: debug your application: ssh port-forwarding 5005 and start your debugging session on localhost:5050
Examples: attach to a JMX console via jmxmp: ssh port-forwarding 5555 and
jconsole -J-Djava.class.path=${CISOTOOLS_HOME}/stool/opendmk_jmxremote_optional_jar-1.0-b01-ea.jar service:jmx:jmxmp://localhost:5555
(Note: if the connections crashes and jconsole asks to reconnect: make sure your jconsole java version matches the application's Java version;
and make sure the jar file is referenced properly)
Ssh into the running stage
sc global-option... ssh [-timeout minutes] stage [shell]
Executes an interactive shell in the main container (i.e. the container running the web app) of this stage.
The default shell is /bin/sh. Reports an error if the stage is not running.
The shell is refused if the current user does not have access to all fault projects of the image.
Validate the stage
sc global-option... validate [-email] [-repair] stage
Checks if the metadataExpire date of the stage has passed. If so, and if
-repair is specified, the stage is stopped (and also removed if expired for more than autoRemove days). And
if -email is specified, a notification mail is sent as configured by the metadataContact value.
Note: Use sc help stage-argument to read about the stage argument,
use sc help for available global options
Start server
sc server
Starts a Stool server, which is a web application implementing the proxy and the dashboard. The server runs in the current console until it's stopped with ctrl-c.
Stage argument
stage = %all | @workspace | predicate
Most Stool commands take a stage argument to specify the stages to operate on. The general form of stage is:
%all specifies all stages in the current context
@workspace specifies all stages in the respective workspace
predicate specifies all matching stages in the current context. The syntax for predicates is as follows:
predicate = and {',' and}
and = expr {'+' expr}
expr = NAME | cmp
cmp = (FIELD | VARIABLE) ('=' | '!=') (STR | prefix | suffix | substring)
prefix = VALUE '*'
suffix = '*' STR
substring = '*' STR '*'
NAME # name of a stage
FIELD # name of a status field
VARIABLE # name of a variable
STR # arbitrary string
The most common predicate is a simple NAME that refers to the respective stage in the current context.
Next, a predicate FIELD=STR matches stages who's status field has the specified string.
VALUE=STR is similar, it matches stage values.
sc status foo prints the status of stage foo.
sc config replicas=0 replicas=1 sets one replica for all stages that have none.
sc delete %all -fail after deletes all stages. Without -fail after, the command would abort after the first
stage that cannot be deleted.
Prerequisites:
- Linux or Mac
- Java 16 or higher.
- If you want to use Kubernetes Contexts: Helm 3.
Install steps
- Download the latest
application.shfile from Maven Central - Make it executable, rename it to
scand add it to your $PATH. - run
sc setup - adjust
~/.sc/settings.yaml
To build Stool, you need:
- Linux or Mac
- Java 15+
- Maven 3+
- Helm 3
- Git
- Docker (with api 1.26+) and Kubernetes.
- TODO: depends on fault and cisotools
The rough layering of packages is
cli, server
core
directions
registry, kubernetes
util
Upper level user lower level, but not neighbours.
Docker on macOS should be fine out of the box (it sets up a Linux VM, and all docker containers run as the development user).
With Linux, you either setup Docker to run as your user or as a root user.
This is the normal setup: install Docker for you Linux distribution and make sure that the development user has permissions to access the docker socket (usually be adding him/her to the Docker group).
In addition, you need to make sure that root has access to the Fault workspace:
- add
user_allow_otherto/etc/fuse.conf - append
-Dfault.mount_opts=allow_roottoFAULT_OPTS
TODO: that's all to make it work?
Running Containers as a none-root user is more secure and saves some trouble when Stool Server needs access to the Fault Workspace. To configure this you have to a) execute containers as the development user, and b) adjust docker socket permissions that are accessible from containers (which is needed for Stool server).
- edit
/lib/systemd/system/docker.socketand changegroupentry to the primary group of your development user (note that this can't be done in/etc/docker/daemon.jsonbecause systemd sets up the socket) - setup user ns mapping for your Docker daemon to run all containers as your development user.
This is a common security measure and, it makes sure that files in bind mounted directories are creates with normal development
user and group. To do so:
(replace with your developer user (e.g. mlhartme), with its primary group (usually the same as ,
with your developer user id (e.g. 1000), and gid with your development user's primary group id (e.g. 1000);
run
idto see your values)-
/etc/docker/daemon.json
{ "userns-remap": "<user>:<group>" } -
/etc/subuid
<user>:<uid>:1 <user>:100000:65535 -
/etc/subgid
<group>:<gid>:1 <group>:100000:65535
-
Apply changes with
systemctl daemon-reload
systemctl restart docker
Check if /var/run/docker.sock is created with the correct primary group. Also check
journalctl -u docker for errors. Note that the following warnings seem to be ok for Stool (i've seen them on Debian and Majaro):
Your kernal does not support cgroup rt period
Your kernel does not support cgroup rt runtime
Next, run docker version, you should get both the server and the client version. If Docker reports a permission problem, check
the above group configuration for /var/run/docker.sock.
Finally, make sure you have a stool network, it's needed for tests and to run the server:
docker network create stool
-
tomcat.p12
puppet.exec("openssl", "pkcs12", "-export", "-in", chain.getAbsolute(), "-inkey", key.getAbsolute(), "-out", p12.getAbsolute(), "-name", "tomcat", "-passout", "pass:changeit");
-
generate fault key:
ssh-keygen -t rsa -b 4096 -N "" -m pem -C "[email protected]" -f ./test-pearl.key
-
add hostkey to fault
fault host-add -fqdn=public_test_pearl -public-key test-pearl.key.pub
Get Stool sources with
git clone https://github.com/mlhartme/stool.git
Stool has pretty standard Maven build, run
cd stool
mvn clean install
The main build result is target/sc. Add it to your path and make sure that
sc -v version
print the correct build date.
To enable the integration tests: touch it.properties and add the following entries
kubernetes: context to use for integration testschartkit: reference to your chartkitportus: Portus registry with credentials to use
Note: you might want to store this file in a different location an create a symlink
Maven releases go to Sonatype, you need the respective account. After running mvn release:prepare and mvn release:perform,
go to the staging repository and promote the release.