Welcome Bottlerocket!

Publish date: Tue, Mar 31, 2020

Bottlerocket logo

What is Bottlerocket?

Bottlerocket is a free and open-source Linux-based operating system meant for hosting containers. Bottlerocket focuses on security and maintainability, providing a reliable, consistent, and safe platform for container-based workloads.


Bottlerocket is somewhat unique in the container ecosystem. It is not quite a microVM like firecracker, but it is signficantly different than a typical Linux distro. Bottlerocket may be considered as a replacement for customers currently running CoreOS Container Linux, which will reach its end of life on May 26, 2020. There is also an open-source fork of CoreOS, knows as flatcar. Flatcar has its own website, and the project is being supported by Kinvolk.

Philosophy behind Bottlerocket

The base operating system has just what you need to run containers reliably, and is built with standard open-source components. Bottlerocket-specific additions focus on reliable updates and on the API. Instead of making configuration changes manually, you can change settings with an API call, and these changes are automatically migrated through updates.


The OS is meant to be immutable, or as immutable as possible. You do not yum update packages, you update the entire OS at once. Therefore, there are available tools to easily self-update the entire OS. So, while you gain a minimally sized OS (small attack surface), with all sorts of cool security goodies, you may need to upgrade the OS even more often than required by an Orchestrator (i.e. Kubernetes fleets are usually updated at least once every three months).

Core components of Bottlerockets


The easiest way to run Bottlerocket is to use eksctl. This is the command tool which was built to specifically manage EKS clusters. In the latest release (0.15.0), there is built-in support for Bottlerocket.

Launch a cluster

eksctl create cluster -f bottlerocket.yaml

where the config files looks like this:

$ cat bottlerocket.yaml
# A simple example of ClusterConfig object with Bottlerocket settings:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

  name: bottlerocket
  region: us-west-2
  version: '1.15'

  id: "vpc-f31a4797"
  cidr: ""
        id: "subnet-da32a9ac"
        cidr: ""
        id: "subnet-b1f0b3d5"
        cidr: ""
        id: "subnet-fd5ef8a5"
        cidr: ""

  - name: ng1-public
    instanceType: m5.xlarge
    desiredCapacity: 2
    amiFamily: Bottlerocket
    ami: auto-ssm
      "network-locality.example.com/public": "true"
      enableAdminContainer: true
        motd: "Hello, eksctl!"

  - name: ng2-public-ssh
    instanceType: m5.xlarge
    desiredCapacity: 2
    amiFamily: Bottlerocket
    ami: auto-ssm
      # Enable ssh access (via the admin container)
      allow: true
      publicKeyName: aws-key

Update CNI

After the cluster comes up, you are advised to update the Container Network Interface to the latest version.

$ kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.6/config/v1.6/aws-k8s-cni.yaml

$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

Test, test, test

I ran many basic workloads (i.e. nginx, redis, small web-servers), and I did not run into any issues at all. Kubernets/EKS talked to the nodes without a problem.

Verifying the worker nodes are running Bottlerocket

$ kubectl get nodes -o wide
NAME                                          STATUS   ROLES    AGE     VERSION    INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                KERNEL-VERSION   CONTAINER-RUNTIME
ip-172-31-22-96.us-west-2.compute.internal    Ready    <none>   4h36m   v1.15.10   Bottlerocket OS 0.3.1   5.4.16           containerd://1.3.3+unknown
ip-172-31-26-118.us-west-2.compute.internal   Ready    <none>   4h36m   v1.15.10    Bottlerocket OS 0.3.1   5.4.16           containerd://1.3.3+unknown
ip-172-31-3-151.us-west-2.compute.internal    Ready    <none>   4h36m   v1.15.10    Bottlerocket OS 0.3.1   5.4.16           containerd://1.3.3+unknown
ip-172-31-4-202.us-west-2.compute.internal    Ready    <none>   4h36m   v1.15.10    Bottlerocket OS 0.3.1   5.4.16           containerd://1.3.3+unknown

Here is snapshot of my Bottlerocket cluster, using one of my favorite tools, kube-ops-view, by Henning Jacobs of Zalando. The documentation is here.

dashboard of running pods

Logging into Bottlerocket

Since it is very hard to login to a Bottlerocket instance, I thought I would show you how to do it. There are two ways to do it:

  1. Use AWS SSM Session Manager to login into the nodes. For security reasons, you must add the IAM policy AmazonSSMManagedInstanceCore to the instance you wish to access (update appropriate instance role). I also prefer to use the MacOS plug-in when working with SSM.
  2. SSH into the Admin container of Bottlerocket. From here you can su, or the Bottlerocket equivalent: sudo sheltie. This gives you a full root shell. For security reasons, this shell has to be enabled from the Control container, or configured during boot-up. I did so for node group ng-2 in my eksctl config file.

Using SSM/Session Manager to login into a control container

$ aws ssm start-session --target i-053e9d0a9018175e3

Starting session with SessionId: nick-macpro-023bcf5f7f19b20bc
Welcome to Bottlerocket's control container!

This container gives you access to the Bottlerocket API, which in turn lets you
inspect and configure the system.  You'll probably want to use the `apiclient`
tool for that; for example, to inspect the system:

   apiclient -u /settings

You can run `apiclient --help` for usage details, and check the main
Bottlerocket documentation for descriptions of all settings and examples of
changing them.

If you need to debug the system further, you can enable the admin container.
This enables SSH access to the system using the key you specified when you
launched the instance.  This environment has more debugging tools installed,
and allows you to get root access to the host.

To enable the admin container, run:


[ssm-user@ip-172-31-26-118 /]$ exit

Exiting session with sessionId: nick-macpro-023bcf5f7f19b20bc.

Logging into the admin container via SSH

$ ssh -i ~/.ssh/aws-key.pem ec2-user@
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:/1oFWYxpGkHSvGJISqrJubiMSlbEG3Dl6dLU/DS2iYE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
Welcome to Bottlerocket's admin container!

This container provides access to the Bottlerocket host filesystems (see
/.bottlerocket/rootfs) and contains common tools for inspection and
troubleshooting.  It is based on Amazon Linux 2, and most things are in the
same places you would find them on an AL2 host.

To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`).  When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
[ec2-user@ip-172-31-22-96 ~]$ sudo sheltie
bash-5.0# ls
bin   etc   lib64	media  proc  sbin  tmp	x86_64-bottlerocket-linux-gnu
boot  home  local	mnt    root  srv   usr
dev   lib   lost+found	opt    run   sys   var
bash-5.0# ls etc
chrony.conf  host-containerd  kubernetes   mtab		  resolv.conf  wicked
cni	     host-containers  ld.so.cache  nsswitch.conf  selinux
containerd   hosts	      ld.so.conf   os-release	  shadow
group	     iproute2	      machine-id   passwd	  systemd
gshadow      issue	      motd	   pki		  updog.toml
bash-5.0# exit
[ec2-user@ip-172-31-22-96 ~]$ logout

What is interesting, is that neither the control or admin containers show up in the pod view of the managing EKS cluster. These containers run outside of the kubelet and kubernetes.

Updating Bottlerocket

Since it is not possible to update packages, Bottlerocket has a tool to assist with the potentially frequent updates of the OS. Fortunately, there is an operator for EKS/Kubernetes. Information on the tool can be found in its Github repo. I have not used it, so I can’t address its benefits or drawbacks. For now, I think that I will simply update the node groups, like any other kubernetes worker nodes. Perhaps AWS will make Bottlerocket a possible default image for the managed worker node feature?!


I quickly explored setting up an EKS cluster using Bottlerocket as the container OS for the worker nodes. Bottlerocket is a highly secure container OS, with minimal size and attack surface.

Bottlerocket is in Preview, and should not be used for production workloads yet. It is available in the following regions:

Bottlerocket logo