Injecting TLS Certs Into Containers

Increasingly, large companies are implementing transparent HTTPS proxy servers in order to enforce company policy on appropriate Internet use or detect unauthorized exfiltration of corporate IP. While I understand the motivation behind this trend, the implementation is, in my considered opinion, a bloody dumpster fire.

That’s because the proxies do it by executing a literal man-in-the middle attack. The proxy strips off the TLS encryption from the connection and re-encrypts traffic in both directions with its own certificate. This works for the typical office worker because there’s generally a locally installed bit of client software that overrides network routing to send all HTTP/S traffic to the proxy and injects the proxy’s root certificate into the operating system’s secure certificate store.

Developers, on the other hand, tend to use things like Docker and Podman and WSL2 and k3s and the AWS CLI and a host of other dev tools that 1) have to make HTTPS connections to work correctly and b) have their own dedicated certificate stores. Generally the first indication you have when something goes wrong is ERR_SSL_PROTOCOL_ERROR error messages or the equivalent.

In my experience when this happens during a docker build the first thing developers will do is assume it’s the Docker daemon that needs fixing, when in reality it’s the Linux distro inside the container that’s missing the proxy’s root cert.

I’ve collected here a set of snippets for injecting the proxy root cert into containers during build. Note that in some cases you might have to do this multiple times, for instance if you manually installed the AWS CLI in a container: the AWS CLI uses its own cert store from python’s certifi library, while other tools will use the system cert store of the Linux distro in the container

All of these snippets assume the root certificate is named tls-proxy-root-cert.crt, is stored in the root of the Docker build context, and is a Base64-encoded x.509 certificate (note: filename extensions are not an accurate indicator of the actual format of a certificate file. Open it up and verify it’s a Base64-encoded file.)

There’s a chicken-and-egg problem if your security admins haven’t whitelisted the package servers for your favourite base image’s distro: you need to install the ca-certificates package to get the update-ca-certificates command, but you can’t install ca-certificates because you can’t get to the package servers. The fix is to copy the TLS proxy root cert into /etc/ssl/certs/ca-certificates.crt, which is enough to get access to the package servers working. Once ca-certificates is installed, copying the TLS proxy root cert to the correct staging location and running update-ca-certificates regenerates the CA cert bundle with all the necessary certs for general use.

Debian/Ubuntu Link to heading

This should also work for Linux distributions based on Debian, like Linux Mint, but I haven’t tested any of those.

Debian-based distros require the certificate filename to have the .crt extension when copied to /usr/local/share/ca-certificates for some reason.

COPY ./tls-proxy-root-cert.crt /etc/ssl/certs/ca-certificates.crt

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update \
  && apt-get -y upgrade \
  && apt-get install -y --no-install-recommends \
    ca-certificates \
  && rm -rf /var/lib/apt/lists/*

COPY ./tls-proxy-root-cert.crt /usr/local/share/ca-certificates/tls-proxy-root-cert.crt
RUN update-ca-certificates

Fedora/RHEL Link to heading

This should also work for all current Red Hat derivatives - Rocky Linux, AlmaLinux, Oracle Linux, etc.

Fedora’s dnf uses curl under the hood, so you need to inject the cert first. Fortunately, Fedora doesn’t need the ca-certificates bundle installed to update the CA cert bundle.

COPY ./tls-proxy-root-cert.crt /etc/pki/ca-trust/source/anchors/tls-proxy-root-cert.crt
RUN update-ca-trust

RUN dnf makecache --refresh \
  && dnf -y install \
    ca-certificates \
  && dnf clean all

Alpine Linux Link to heading

Alpine has the same chicken-and-egg problem Debian does.

COPY ./tls-proxy-root-cert.crt /etc/ssl/certs/ca-certificates.crt

RUN apk update \
  && apk add ca-certificates \
  && rm -rf /var/cache/apk/*

COPY ./tls-proxy-root-cert.crt /usr/local/share/ca-certificates/tls-proxy-root-cert.crt
RUN update-ca-certificates

AWS and Azure CLIs Link to heading

Both the Azure and AWS CLIs are written in python and bundle their own python and dependent libraries. This includes a dedicated certificate store which they use by default. You can override the location of the certificate store with an environment variable:

To use these tools inside a Docker container, first install the proxy’s root certificate as described above, and then set an environment variable as listed in the table before running the command.

Note that these instructions are tested and correct as of the date of writing, but life comes at you fast. If you’re coming across this two years from now there’s a good chance all this will have changed.