Writing BPMN Let's Encrypt Kubernetes Operators in Python II

Writing BPMN Let’s Encrypt K8s Operators in Python II

Registering a certificate using the cert-bot from letsencrypt might be a little tricky. Especially without root. Inside a container. Here’s how to achieve that as part of our operator.

This article is the second one from our series:

1. How the operator architecture looks like,

2. How the registration of the new certificate works (this article),

3. Lessons learned.

The registration of the certificate happens in another BPMN process that gets launched as a job (in the operator):

@adhesive.task('Create/Renew Certificate for {event.id}')
def create_or_renew_certificate_for_event_id_(context: adhesive.Token[Data]) -> None:
    kubeapi = KubeApi(context.workspace)

    try:
        kubeapi.apply(f"""
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: {context.data.event.id}
            namespace: {context.data.event.namespace}
          spec:
            template:
              metadata:
                labels:
                  app: register-domain
              spec:
                containers:
                - name: register-domain
                  image: germaniumhq/certbot
                  imagePullPolicy: Always
                  command: ["python", "new-certificate.py"]
                  env:
                  - name: INGRESS_OBJECT
                    value: {context.data.event.id}
                  - name: KUBERNETES_NAMESPACE
                    value: {context.data.event.namespace}
                  - name: ADHESIVE_POOL_SIZE
                    value: "100"
                restartPolicy: Never
            backoffLimit: 4
        """)

Since this is also just an Adhesive process, it’s actually in the same operator image. This one reads its input from the environment (in the new-certificate.py job)

def main() -> None:
    kubernetes_namespace = read_env("KUBERNETES_NAMESPACE")
    ingress_object_name = read_env("INGRESS_OBJECT")

    adhesive.bpmn_build(
        "new-certificate.bpmn",
        initial_data={
            "namespace": kubernetes_namespace,
            "ingress_object": ingress_object_name,
        })

The new-certificate.bpmn is the one we’ll examine now.

Ok, let’s cut to the chase. To create the certificate, we’ll use the certbot script from letsencrypt. (https://certbot.eff.org/) This script is written in python, and wrapped in a shell script that can restart services, start an HTTP server, etc.

Why? Because for it to function, it needs to write some private challenge on the disk, so letsencrypt from outside knows that we are the actual owners of the website. This disk access is an immediate problem for us because we don’t know what’s behind that service. Maybe there’s just a REST endpoint without any storage.

To mitigate that we’ll do two things:

1. Start a pod with "disk" access that can run the certbot-auto so it can do its job.

2. Run an HTTP server that can serve the folder where certbot-auto writes its challenge.

3. Patch the Ingress object, so requests for the challenge go to our pod, not to the default service.

The sweet part about this? While we’re doing this, we don’t have downtime. Furthermore, we can start the HTTP server directly from the BPMN process, since that would block in its thread:

As we bring things up, we’re waiting for them to be active. These checks make debugging easier.

After, we patch the Ingress against the service that points to our job:

for rule in ingress.spec.rules:
    domain_names.add(rule.host)
    rule.http.paths._raw.insert(0, yamldict.create(f"""
        backend:
          serviceName: register-domain
          servicePort: http
        path: /.well-known/
    """))

We check if all the domains defined in the Ingress are accessible from outside, then we run the certbot-auto script:

@adhesive.task('Create Certificate for {domain_names}')
def create_certificate_for_domain_name_(context: adhesive.Token[Data]) -> None:
    domains_as_string = f"-d {' -d '.join(context.data.domain_names)}"

    context.workspace.run(f"""
        export LE_AUTO_SUDO=
        certbot-auto certonly \\
                --webroot \\
                --agree-tos \\
                --no-bootstrap \\
                --email bogdan.mustiata@gmail.com \\
                -n \\
                {domains_as_string} \\
                --config-dir /tmp/le/config \\
                --work-dir /tmp/le/work \\
                --logs-dir /tmp/le/logs \\
                --webroot-path /tmp/www
    """)

Now that we have the certificate, we throw it into a secret, repatch the Ingresses, cleanup, and done.

To create the secret, we just read the files and dump them:

@adhesive.task('Create Secret {ingress_object}')
def create_secret(context: adhesive.Token[Data]) -> None:
    kube = KubeApi(context.workspace)
    namespace = context.data.namespace
    domain_name = context.data.domain_names[0]

    tls_certificate = read_file_base64(f'/tmp/le/config/live/{domain_name}/cert.pem')
    tls_key = read_file_base64(f'/tmp/le/config/live/{domain_name}/privkey.pem')

    kube.apply(f"""
        apiVersion: v1
        kind: Secret
        type: kubernetes.io/tls
        metadata:
            name: {context.data.ingress_object}
            namespace: {namespace}
        data:
            tls.crt: {tls_certificate}
            tls.key: {tls_key}
    """)

And finally to add the secret:

@adhesive.task('Add TLS secret to ingress {ingress_object}')
def add_tls_secret_to_ingress(context: adhesive.Token[Data]) -> None:
    kubeapi = KubeApi(context.workspace, context.data.namespace)
    ingress = kubeapi.get(kind="ingress",
                          name=context.data.ingress_object,
                          namespace=context.data.namespace)

    remove_well_known_paths(ingress)

    ingress.spec.tls = [
        {
            "hosts": context.data.domain_names,
            "secretName": context.data.ingress_object
        }
    ]

    kubeapi.apply(ingress)

Article Photo from Pexels: https://www.pexels.com/photo/people-laptop-industry-internet-132700/