Space requirements for a small step certificate authority and how to reduce disk space usage

[andrei-kondakov]

Hello!

First of all, I want to say thank you for your work done. You have developed an excellent product for easy certificate administration 👍 🥇

We want to integrate small step ca solution into our infrastructure.

For testing, I created a cloud server with 32 GB of disk space.

root@ca:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal

root@ca:~# step version
Smallstep CLI/0.15.16 (linux/amd64)
Release Date: 2021-04-15 22:36 UTC

root@ca:~# sudo lshw -c cpu
  *-cpu                     
       description: CPU
       product: Intel Core Processor (Skylake, IBRS)
       vendor: Intel Corp.
       physical id: 400
       bus info: cpu@0
       version: pc-q35-5.2
       slot: CPU 0
       size: 2GHz
       capacity: 2GHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp x86-64 constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
       configuration: cores=1 enabledcores=1 threads=1

On this server, I have successfully set up a small step CA listening on port 4343.

I have issued certificates for all servers in our infrastructure (the default expiration is 1 day). And I configured the automatically renew of these certificates as described here: https://smallstep.com/docs/step-ca/certificate-authority-server-production#the-standalone-step-renewal-daemon

Thus, I have 30+ servers daily requesting a certification authority to update a certificate.

After some time with the system running, I found that all my disk space was full.

root@ca:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   28G  1.1G  97% /

So I would like to ask about the memory requirement of the certification authority.
And how can you reduce the consumption of server space?

My config file:

{
	"root": "/etc/step-ca/certs/root_ca.crt",
	"federatedRoots": [],
	"crt": "/etc/step-ca/certs/intermediate_ca.crt",
	"key": "/etc/step-ca/secrets/intermediate_ca_key",
	"address": ":4343",
	"dnsNames": [
		...
	],
	"logger": {
		"format": "text"
	},
	"db": {
		"type": "badgerV2",
		"dataSource": "/etc/step-ca/db",
		"badgerFileLoadingMode": "FileIO"
	},
	"authority": {
		"provisioners": [
			...
		]
	},
	"tls": {
		"cipherSuites": [
			"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
			"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
		],
		"minVersion": 1.2,
		"maxVersion": 1.3,
		"renegotiation": false
	}
}

I found that most of the space is taken up by these files (using ncdu):

--- /etc/step-ca/db -------------------------------------------------------------------------------------------------------------------------
                         /..
  313.4 MiB [##########]  000020.vlog                                                                                                        
  313.4 MiB [##########]  000029.vlog
  313.4 MiB [######### ]  000019.vlog
  313.4 MiB [######### ]  000028.vlog
  313.4 MiB [######### ]  000021.vlog
  313.4 MiB [######### ]  000027.vlog
  313.0 MiB [######### ]  000018.vlog
  312.9 MiB [######### ]  000007.vlog
  312.9 MiB [######### ]  000004.vlog
  312.9 MiB [######### ]  000011.vlog
  312.9 MiB [######### ]  000017.vlog
  312.9 MiB [######### ]  000003.vlog
  312.9 MiB [######### ]  000013.vlog
  312.9 MiB [######### ]  000008.vlog
  312.9 MiB [######### ]  000009.vlog
  312.9 MiB [######### ]  000012.vlog
  312.9 MiB [######### ]  000002.vlog
  312.9 MiB [######### ]  000005.vlog
  312.9 MiB [######### ]  000010.vlog
  312.9 MiB [######### ]  000015.vlog
  312.9 MiB [######### ]  000016.vlog
  312.9 MiB [######### ]  000006.vlog
  312.4 MiB [######### ]  000001.vlog
  312.3 MiB [######### ]  000000.vlog
  130.8 MiB [####      ]  000033.vlog
   76.5 MiB [##        ]  000030.vlog
   75.9 MiB [##        ]  000022.vlog
   62.7 MiB [##        ]  086221.sst
   62.0 MiB [#         ]  052332.sst
   59.6 MiB [#         ]  000014.vlog
   58.8 MiB [#         ]  034009.sst
   58.8 MiB [#         ]  034010.sst
   58.8 MiB [#         ]  034048.sst
   58.8 MiB [#         ]  002017.sst
   58.8 MiB [#         ]  033986.sst
 Total disk usage:  14.5 GiB  Apparent size:  14.5 GiB  Items: 3675           

I read in the documentation that this ".vlog" is meta information for the work of a certification authority.

Is it possible to somehow clean them and reduce the consumption of space?
Maybe I should use another database like bolt or etc?

Thank you <3

[maraino]

@andrei-kondakov My guess is that most of the data are ACME-related data, which is not really necessary after a sign has been completed. I think @dopey has some kind of script that drops some of this data and does some garbage collection to reduce the size on disk. Unfortunately, you will need to stop step-ca to do this.

A better alternative would be to use the MySQL backend on a different container that would be easy to clean if it grows.

[andrei-kondakov]

A better alternative would be to use the MySQL backend on a different container that would be easy to clean if it grows.

@maraino If i use MySQL backend how do i know what data i can safely delete?

[andrei-kondakov]

mysql> SHOW TABLES;
+----------------------------+
| Tables_in_smallstep        |
+----------------------------+
| acme_account_orders_index  |
| acme_accounts              |
| acme_authzs                |
| acme_certs                 |
| acme_challenges            |
| acme_keyID_accountID_index |
| acme_orders                |
| nonces                     |
| revoked_ssh_certs          |
| revoked_x509_certs         |
| ssh_certs                  |
| ssh_host_principals        |
| ssh_hosts                  |
| ssh_users                  |
| used_ott                   |
| x509_certs                 |
+----------------------------+
16 rows in set (0.01 sec)

[dopey]

First off, quick plug for our discord community - https://discord.gg/PWGZeKZB. I have a bunch of questions and it can be easier to ask those in a more real time environment.

Questions:

1. How long have you been running? your server is creating 30 certificates per day, so that amount of disk usage is very unexpected, even if you'd been running for years. I would want to see your step-ca logs to try to understand why so much db was being used. It's possible that an ACME request is failing causing you to get into some sort of loop. (just spitballing).
2. How are you requesting certificates? ACME? If so, which client are you using?
3. I ca definitely instruct you about which tables you can get rid of, but I think trying to dig into the root of what's going on might be more useful.

Here's an example script for a badger DB that goes through each table and prints out how many "rows" are in the table. Would love to run that against your db to try to understand where are the data is. https://gist.github.com/dopey/89ec20f22c66c1333bf38c9b19b89758.

[dopey]

Ended up working closely with @andrei-kondakov over on our discord. Seems like they had gotten into an unusual situation where they were requesting far too many certificates (likely renewals) from their DB. >11million over the course of 3 days. Looks like their usage has settled, but I've asked @andrei-kondakov to let us know if a similar situation begins to develop.