In this post, I’ll modify the pipeline from the previous posts to use a Docker registry powered by AWS ECR (Amazon Elastic Container Registry).

Creating AWS ECR repositories

First, we need to enable ECR in Amazon and create our repositories. We have two images that we need to publish, blog-helm and blog-helm-ci, so we need two repositories.

Enable ECR
Create the blog-helm repository
Success!

The second repository is created in the same way.

Getting Docker credentials

To be able to use this registry, we need to login with docker login. The AWS CLI offers a command which prints the necessary docker login command we need to run:


> aws ecr get-login --no-include-email --region eu-west-1
docker login -u AWS -p *** https://830988624223.dkr.ecr.eu-west-1.amazonaws.com

TeamCity changes

TeamCity in theory supports connecting to a Docker registry as a build feature. However, when I tried to setup the connection it complained that the password is too long (it is 1868 characters, so, yeah that’s long), so that didn’t work. This means that we need to do some extra work.

First, we’ll create a few parameters on the project level:

Configuration Parameters for the Docker registry

With this in place, I need to add two new build steps:

  • Before pushing to or pulling from the Docker registry, we need to be logged in. This is an extra step that will run docker login -u %docker.username% -p %docker.password% %docker.server%.
  • After the registry is no longer required, I'd like to be logged out. This is an extra step that will run docker logout %docker.server%.

Since I have my build pipeline in code, I can inject these steps easily wherever I need:

        script {
            name = "Login to Docker registry"
            scriptContent = "docker login -u %docker.username% -p %docker.password% %docker.server%"
        }
        script {
            name = "Push Docker production image"
            scriptContent = "docker push %docker.registry%/blog-helm:%env.IMAGE_TAG%"
        }
        script {
            name = "Push Docker CI image"
            scriptContent = "docker push %docker.registry%/blog-helm-ci:%env.IMAGE_TAG%"
        }
        script {
            name = "Logout from Docker registry"
            scriptContent = "docker logout %docker.server%"
            executionMode = BuildStep.ExecutionMode.ALWAYS
        }

Notice that in the logout step, I’ve configured the execution mode to be Always. This means that it will always run, even if some other step failed before. It’s a nice setting to have for cleanup tasks like these.

It might be interesting to see how the password is stored. Having defined the parameter as a password, TeamCity does not commit its actual value into the code repository. Instead, it commits some reference value:

        password("docker.password", "credentialsJSON:175b2d15-2353-475e-ab70-571d1e5843e9", label = "Docker registry password")

When I change the password via the UI, TeamCity commits a patch file that updates this reference value:

changeProject("d3c230cf-b4cd-4a9e-8017-4b4b945b3a3c") {
    params {
        expect {
            password("docker.password", "credentialsJSON:6a39fd43-0513-4ba4-a446-14843fa7c355", label = "Docker registry password")
        }
        update {
            password("docker.password", "credentialsJSON:175b2d15-2353-475e-ab70-571d1e5843e9", label = "Docker registry password")
        }
    }
}

Results in AWS ECR

With this in place, I’m able to publish the images to AWS ECR:

Production Image (blog-helm)
CI Image (blog-helm-ci)

You can see that the production image is much smaller than the ci image, because the latter contains dev dependencies and it’s not based on alpine, due to PhantomJS.

I also had a mistake in my .dockerignore, I should have excluded the ci-scripts folder from the Docker context; this unfortunately was creating a different CI image on every build. This is why the CI image has so many different tags compared to the production image.

Using the image in Kubernetes

My minikube needs to be able to pull from the AWS ECR too. First, I need to create a secret with the kubectl CLI tool:

$ kubectl create secret docker-registry myaws \
    --docker-username=AWS \
    --docker-password=*** \
    --docker-email=*** \
    --docker-server=830988624223.dkr.ecr.eu-west-1.amazonaws.com
secret "myaws" created

This will create a new secret named myaws. To use it, I need to modify the deployment template of my Helm chart:

    spec:
      imagePullSecrets:
        - name: myaws
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}

and of course I need to specify the new registry in the values.yaml:

replicaCount: 1
image:
  repository: 830988624223.dkr.ecr.eu-west-1.amazonaws.com/blog-helm
  tag: latest
  pullPolicy: IfNotPresent

And that’s it! Now my local Kubernetes is able to pull images from the AWS ECR as well.

AWS Limits and Lifecycle Policy

In the AWS UI, there’s a prominent warning: Amazon ECR limits the number of images to 1,000 per repository. Since we’re publishing an image on every commit, I can imagine that this limit is easy to reach. Let’s say 5 devs are working on a project (repository) and they each make 10 commits a day. In 20 working days, the limit is met.

Luckily AWS offers a cleanup policy. It can delete images based on their age or their count. Images are selected based on the tag prefix. Unfortunately we don’t have a specific catch-all prefix in the current setup, as the image tag is following semver. It would be perhaps interesting to implement a tag scheme that has a different prefix for feature branches and different for the master branch (e.g. prod-1.2.3 for master and feat-1.2.4-some-feature-branch.1 for the feature branches). It would be also great if AWS allowed for a regex match. In any case, since my code is at the moment on major version 2, I can setup a rule for all images starting with “2.” and specify I want to keep the 5 most recent:

Creating a lifecycle policy

These are the 7 images currently available:

Images sorted by date

If I do a dry-run on the rule, it correctly identifies the two oldest that will be deleted with this policy:

Termination candidates