Containers done right - providing configuration

Containers are ephemeral, but almost always require configuration and credentials. They are kept and maintained seperately from application code. This principal is very important part of containers world. By providing different configuration to the same, ephemeral container we create an app instance that should do the thing it was designed to do that is controlled with those parameters.

Why it’s important

It’s easy to change configuration in traditional world with an access to servers via ssh. This however often leads to snowflake antipattern when no one really knows what has been changed, by who and why. To keep your environments consistent you must manage configuration of your applications in a proper way.

Another important part is dealing with sensitive configs, like credentials to external services (e.g. databases). This part of containerized environment is confusing for many people. In Kubernetes we have ConfigMap and Secret objects that are designed to store such information, but sometimes they are not the best solution and certainly not the only one.

DON’Ts

• Don’t store credentials inside container image

Just don’t.

• Don’t store confidential data in ConfigMap

It’s obvious, but many people still use it as a single source for configs.

• Don’t store raw Secret objects in git

They are not encrypted, but only obfuscated. Just run them through base64 decrypt to see their values. They are treated in a special way by Kubernetes components (such as storing them on nodes in memory, not on peristsent storage), but current implementation doesn’t include any encryption. Personally I think that’s a missing piece that force many users to use external tools for such things (see below).

• Don’t pass secrets as environment variables.

Although you can use environment variables to pass it to an app, but you shouldn’t use that feauture. It’s because when you don’t handle them properly (i.e. unsetting them after usage) they migh be visible to an attacker - listing all environment variables is easy when there’s a vulnerability in app code. Remeber - that’s a serious security risk you need to be aware of.

DOs

• Do use encryption when storing real confidential data in git.

There are couple of solutions available that will manage encryption of your data stored in git:

• git-secret
• git-crypt
• BlackBox

• Ansible vault - mostly for yaml files, especially when you already use ansible

• Do provide default configuration in your container images

Don’t expect users to prepare a long list of all available config paramaters. Help them to launch your container by providing reasonable defaults and let them provide only a few of parameters or ideally even without.

• Do put proper labels on Secret and ConfigMap objects

For environments where you have multiple applications running in the same namespace, labels that match those attached to the application that is going to use it, help to identify it quickly.

• Do use multiple keys inside ConfigMap for a single application

In general use single configmap per application

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-myconfig
  app: myapp
data:
  firstfile.cfg: |+
    # config content here
    # ....
  anotherfile.cfg: |+
    # config content here
    # ....

• Do consider using much more secure solution for managing sensitive data

Currently the best solution available for containerized environments is HashiCorp Vault. Most interesting features are:

• dynamically generated credentials for databases, nodes (ssh authenitcation based on certificates)
• tighter access control based on RBAC and service accounts assigned to pod
• auditing capabilities - who, when and what accessed your secrets
• auto renewal of token with Agent’s Auto-auth feature (since version 0.11)