Running a PKI using Smallstep certificates with Docker

Recently, I had to set up a new PKI. I was going to go with the good old OpenSSL but it’s 2021, there must be a more userfriendly and, more importantly, automated approach.

There are many open-source possibilities: EJBCA, cfssl, Hashicorp Vault, Smallstep Certificates. I chose to use Smallstep certificates because it has all the features I need and they are not behind a pay-wall:

• lightweight: small Go binary, you can run it with a file-based database (similar to SQLite)
• user friendly CLI: compared to openssl commands
• ACME protocol: useful for Traefik reverse proxy
• OIDC authentication
• support: the guys are super friendly and available on their Discord channel

Be sure to check their website, they have other features that you might want, especially their Certificate Manager. They also have a SaaS offering if you do not want to get your hands dirty.

Let’s dive into it shall we?

Initialize the CA

First, create a volume to store step-ca data: docker volume create step-ca_step

Then start the container to manually initialize the CA: docker run -it --rm -v <code>step-ca_step:/home/step smallstep/step-ca:0.16.0 sh

Let’s initialize the base structure of step-ca. I could do it manually but this is faster and easier. We are going to specify a few parameters:

• Name of the PKI: not important, it will be overwritten with our manually created CA
• DNS names or IP address: not important, same reason
• IP:port binding: :8000 means port 8000 on all ports inside the container
• Name of the first provisioner: ca-admin, first user of the system for remote access
• Password: password of the first user

~ $ step ca init
✔ Deployment Type: Standalone
What would you like to name your new PKI?
✔ (e.g. Smallstep): not important
What DNS names or IP addresses would you like to add to your new CA?
✔ (e.g. ca.smallstep.com[,1.1.1.1,etc.]): ca.xentoo.local
What IP and port will your new CA bind to?
✔ (e.g. :443 or 127.0.0.1:4343): :8000
What would you like to name the CA's first provisioner?
✔ (e.g. you@smallstep.com): ca-admin
Choose a password for your CA keys and first provisioner.
✔ [leave empty and we'll generate one]: 
✔ Password: supersecretpassword

Generating root certificate...
all done!

Generating intermediate certificate...
all done!

✔ Root certificate: /home/step/certs/root_ca.crt
✔ Root private key: /home/step/secrets/root_ca_key
✔ Root fingerprint: superlongfingerprint
✔ Intermediate certificate: /home/step/certs/intermediate_ca.crt
✔ Intermediate private key: /home/step/secrets/intermediate_ca_key
✔ Database folder: /home/step/db
✔ Default configuration: /home/step/config/defaults.json
✔ Certificate Authority configuration: /home/step/config/ca.json

Your PKI is ready to go. To generate certificates for individual services see 'step help ca'.

At this point, the CA is fully functional with a root CA and an intermediate. We could use it as is, but we won’t. One reason is that the root CA private key passphrase, the intermediate CA private key passphrase and the ca-admin provisioner password are all the same.

Let’s overwrite all certificates with our own.

Replace autogenerated certificates

First, the root certificate using elliptic curve 384 bits for the private key and a duration of 20 years:

~ $ step certificate create --profile root-ca --kty=EC --curve=P-384 --not-after=175320h "Xentoo Root CA" certs/root_ca.crt secrets/root_ca_key
Please enter the password to encrypt the private key:
✔ Would you like to overwrite secrets/root_ca_key [y/n]: y
✔ Would you like to overwrite certs/root_ca.crt [y/n]: y
Your certificate has been saved in certs/root_ca.crt.
Your private key has been saved in secrets/root_ca_key.

Note: save the root CA private key file secrets/root_ca_key and its passphrase in a VERY safe location.

Next, create a new intermediate certificate using elliptic curve 384 bits for the private key as well and a duration of 10 years:

~ $ step certificate create --profile intermediate-ca --kty=EC --curve=P-384 --not-after=87660h --ca certs/root_ca.crt --ca-key secrets/root_ca_key "Xentoo Intermediate CA" certs/intermediate_ca.crt secrets/intermediate_ca_key
Please enter the password to decrypt secrets/root_ca_key:
Please enter the password to encrypt the private key:
✔ Would you like to overwrite secrets/intermediate_ca_key [y/n]: y
✔ Would you like to overwrite certs/intermediate_ca.crt [y/n]: y
Your certificate has been saved in certs/intermediate_ca.crt.
Your private key has been saved in secrets/intermediate_ca_key.

Write the intermediate CA key password in /home/step/secrets/password. Without this, step-ca cannot use the private key and so cannot generate certificates. Make sure the Docker host and container volume are secure.

Retrieve the fingerprint of the root certificate, you will need it to connect remotely to the CA lter: step certificate fingerprint certs/root_ca.crt.

At this point, if you saved the root CA private key file secrets/root_ca_key and passphrase in a safe location outside the container, you can delete it from the container. You will not need it anymore.

Exit the container.

Docker Compose file

Let’s create a basic compose file to be able to start our PKI.

We need a few things set up:

• expose the PKI port for step-cli, here I use port 8500
• re-use the volume step-ca_step for our configuration
• create a container for mysql and give it a volume for the database. The environment variables match the parameters given to step-ca in the example of section Use MySQL as database
• impose CPU and memory limits on containers to prevent any memory leaks (adjust to your needs)

version: "3"

services:
  ca:
    image: smallstep/step-ca:0.18.0
    container_name: step-ca
    restart: unless-stopped
    ports:
      - 192.168.0.10:8500:8500
    volumes:
      - step-ca_step:/home/step
    depends_on:
      - db
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 100M
    memswap_limit: 100M

  db:
    image: mysql:8.0.27
    container_name: step-db
    restart: unless-stopped
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_DATABASE=dbname
      - MYSQL_USER=user
      - MYSQL_PASSWORD=password
      - MYSQL_RANDOM_ROOT_PASSWORD=yes

    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M
    memswap_limit: 512M

volumes:
  step-ca_step:
    external: true
  db:

You can now start the stack using compose: docker compose up. Do not worry about other containers than ca starting right now, we will cover them later.

Create your first certificate

If you configured your DNS records to point to your Docker host, you should be able to access it using https and the port you configured. In my case: ca.xentoo.local:8000.

On your remote system, download step-cli from Github and bootstrap the connection with CA using the following command:

~ $ step ca bootstrap --ca-url https://ca.xentoo.local:8000 --fingerprint therootfingerprint
The root certificate has been saved in /home/step/certs/root_ca.crt.
Your configuration has been saved in /home/step/config/defaults.json.

You can check the connection is successful by checking the CA health: step ca health and if it returns ok, you’re good to go.

At this point, you can manually request certificates using step ca certificate. Let’s test it:

~ $ step ca certificate test1 test1.crt test1.key
✔ Provisioner: ca-admin (JWK) [kid: random_id]
✔ Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.xentoo.local:8000
✔ Certificate: test1.crt
✔ Private Key: test1.key

Inspect the certificate with step certificate inspect test1.crt:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: xxx (xxx)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=Xentoo Intermediate CA
        Validity
            Not Before: Jul 31 14:21:30 2021 UTC
            Not After : Aug 1 14:21:30 2021 UTC
        Subject: CN=test1
        ...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Subject Key Identifier:
                xxx
            X509v3 Authority Key Identifier:
                keyid:xxx
            X509v3 Subject Alternative Name:
                DNS:test1
            X509v3 Step Provisioner:
                Type: JWK
                Name: ca-admin
                CredentialID: xxx

The certificate has been generated successfully, however, it has a few issues.

Adding content and extensions to issued certificates

You can notice a few things:

• the private key is not encrypted
• the certificate subject is rather empty
• the certificate lifetime is set to 24h
• there is no Authority Information Access
• there is no CRL distribution point
• there is no OCSP responder URL
• the provisioner name is embedded in the issued certificate

Let’s change most of these by using a custom certificate template. In the container, create the file templates/certs/x509/leaf.tpl with the following content:

{
        "subject": {
                "commonName": {{ toJson .Subject.CommonName }},
                "country": "BE",
                "province": "BELGIUM",
                "locality": "BRUSSELS",
                "organization": "Xentoo",
                "organizationalUnit": "Operations"
        },
{{- if .SANs }}
        "sans": {{ toJson .SANs }},
{{- end }}
{{- if typeIs "*rsa.PublicKey" .Insecure.CR.PublicKey }}
        "keyUsage": ["keyEncipherment", "digitalSignature"],
{{- else }}
        "keyUsage": ["digitalSignature"],
{{- end }}
        "extKeyUsage": ["serverAuth", "clientAuth"],
        "ocspServer": "http://ocsp.xentoo.local/",
        "issuingCertificateURL": "http://pki.xentoo.local/intermediate_ca.crt",
        "crlDistributionPoints": "http://pki.xentoo.local/intermediate_ca.crl"
}

Let’s apply the template to our default provisioner, as well as some time constraints:

• minimum lifetime: 24 hours
• maximum lifetime: 5 years
• default lifetime: 5 years

In the container, edit config/ca.json and add the following content to the provisioner ca-admin secution:

                                "claims": {
                                        "minTLSCertDuration": "24h",
                                        "maxTLSCertDuration": "43800h",
                                        "defaultTLSCertDuration": "43800h"
                                },
                                "options": {
                                        "x509": {
                                                "templateFile": "templates/certs/x509/leaf.tpl"
                                        }
                                }

Restart the container: docker compose restart ca . Generate a new certificate and inspect it to validate all the settings have been set as defined.

Use MySQL as database

By default, step-ca uses Badger database. If you want to implement high-availability for step-ca, you will need a database that supports it. At the moment, there is only MySQL. So let’s go for it.

Edit config/ca.json and replace the db section with the following content:

        "db": {
                "type": "mysql",
                "dataSource": "user:password@tcp(hostname:3306)/",
                "database": "dbname",
                "badgerFileLoadingMode": ""
        },

Note: user, password, hostname and dbname must match the environment variables in your docker-compose.yml file for the db service. In my case:

• user=step
• password=iwillnottellyou
• hostname=db
• database=step

Restart the ca container to apply database changes: docker compose restart ca.

Generate another certificate to validate the database connection works fine. If you do not have errors, go on.

Enable ACME provisioner

Next, because I use Traefik internally and it is able to automatically request certificates using the ACME protocol, I am going to enable the ACME provisioner on my CA. In the container, edit config/ca.json and add a new provisioner by appending the following content to the provisioners section:

                        {
                                "type": "ACME",
                                "name": "acme",
                                "forceCN": true,
                                "claims": {
                                        "minTLSCertDuration": "24h",
                                        "maxTLSCertDuration": "2160h",
                                        "defaultTLSCertDuration": "2160h"
                                },
                                "options": {
                                        "x509": {
                                                "templateFile": "templates/certs/x509/leaf.tpl"
                                        }
                                }
                        }

Notice I am using the same template file, but I have change the duration settings to mimick Let’s Encrypt.

Restart the container and let’s try to create a new certificate using certbot.

# certbot certonly --standalone --domains test.xentoo.local --server https://ca.xentoo.local:8000/acme/acme/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): ca.xentoo.local
OCSP check failed for /etc/letsencrypt/archive/test.xentoo.local/cert1.pem (are we offline?)
Cert is due for renewal, auto-renewing...
Renewing an existing certificate for test.xentoo.local
Performing the following challenges:
http-01 challenge for test.xentoo.local
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/test.xentoo.local/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/test.xentoo.local/privkey.pem
   Your certificate will expire on 2021-10-08. To obtain a new or
   tweaked version of this certificate in the future, simply run
   certbot again. To non-interactively renew *all* of your
   certificates, run "certbot renew"

Note: make sure port 80/tcp is open before your start, cerbot challenge uses by default http-01 method.

Note: make sure to configure your DNS record to point to the server you are requesting the certificate from.

Certificate Revokation List

We can generate certificates but we also can revoke them using step ca revoke <serialnumber> or, if you have the certificate file and the private key file, using step ca revoke certificate_file privatekey_file.

However, our clients have no way to verify the certificates have been revoked yet. Let’s fix that.

As of today, 2021-09-12, it is not yet possible to generate a CRL file using step nor step-ca. They plan to develop it though.

However, others have found workarounds. Since all the information we need is in the database, we can manually generate the CRL using good old bash and openssl commands.

When you revoke a certificate, its serial number and date of revocation is added to the database table revoked_x509_certificates in “binary JSON” format. Using the bash script in the gist above, we will generate an index.txt file containing all the revoked certificates. This file can be ingested by openssl ca -gencrl command.

For convenience, I have packaged all the config files and scripts in a container image called crl . Whenever I need to generate the CRL, I simply start the container with docker compose up crl.

Let’s try that by first revoking our first certificate: step ca revoke test1.crt test1.key.

Let’s start the crl container to generate the CRL: docker compose up crl > crl.pem.

Let’s check the CRL contains our certificate: openssl crl -in crl.pem -text -noout.

It looks fine visually, let’s make sure the serial number is correct. We first need to concatenante our CA chain with the CRL file.

$ docker cp step-ca /home/step/certs/root_ca.crt .
$ docker cp step-ca /home/step/certs/intermediate_ca.crt .
$ cat root_ca.crt intermediate_ca.crt crl.pem > crl_chain.pem

Now that we have our CRL with CA chain, let’s verify test1.crt against it:

$ openssl verify -crl_check -CRLfile crl_chain.pem test1.crt

You can also check that our second cerificate test2.crt is still valid:

$ openssl verify -crl_check -CRLfile crl_chain.pem test2.crt

We can copy this file to our webserver to make it available as http://pki.xentoo.local/intermediate_ca.crl as described in our certificate template.

Authority Information Access: CA Issuers

Since we have defined a URL to check the intermediate CA, we need to publish it. Copy certs/intermediate_ca.crt from step-ca container to your webserver and make it available as described in the template.

For me, it’s http://pki.xentoo.local/intermediate_ca.crt .

Authority Information Access: OCSP responder

Coming soon.

Backups

Our CA is running fine and we need to make sure we can restore it if anything bad happens. So we need backups.

There are a few important things to backup:

• the compose files
• the configuration files
• the database

Regarding compose files, they are stored in a Git repository which should be backed up separately. So we’re good. The environment variables values are not included in the repository, but they are defined in step-ca config file config/ca.json, so we’re good too.

We will backup the files using good old tar and the database with mysqldump.

For convenience, I have packaged both in the container called backups. The container has read-only access to step-ca volume and will connect to db container using environment variables.

To perform backups, simply run docker compose up backups. The backup files are stored in Docker volume called step-ca_backups.

Conclusion

It was a bit long but we made it.

At this point, we have a fully functional PKI, capable of issuing certificates with a rather userfriendly CLI or ACME protocol. We have CRL and (soon) OCSP support. And we have backups in case something goes sideways.