Interacting with EKS via Lambda

Publish date: Mon, Sep 3, 2018

EKS authentications

One typically interacts with a Kubernetes cluster through kubectl. However, that only really works for interactive commands. When you want to automate something, you need to script it. Fortunately, there are several excellent kubernetes client libraries. The officially supported one is written in Go, simply because kubernetes is written in Go.

Now, there is a hurdle that must be overcome when scripting for an EKS cluster. EKS, like all AWS services, uses IAM for authentication. This means, that any scripts also need to somehow use IAM. The official Go client library supports external authenticators (now called credential plugins), as defined in your local KUBECONFIG file. Unfortunately, this support for external authenticators is not yet supported in other libraries (who doesn’t love Python these days?!). So, we will use the Go client library to demonstrate how to script interactions with your EKS cluster.

Using the Go client

Grab the sample code from the repository, for out-of-cluster-client-configuration.

Running this example

Make sure your kubectl is configured and pointed to a cluster. You also need your authenticator plugin installed on your system, and your kubectl config file referencing it. Notice how the arguments being passed to the authenticator references a role. That is an IAM role that the user must be able to execute for authentication to succeed. The authenticator repo and AWS docs describes how you must create a role to be used for authentication, and then use a ConfigMap to update your cluster to recognized that particular role.

Kubectl Configuration file

# [...]
- name: kubernetes-admin
      command: aws-iam-authenticator
        - "token"
        - "-i"
        - "CLUSTER_ID"
        - "-r"
        - "ROLE_ARN"
  # no client certificate/key needed here!

Kubernetes ConfigMap

apiVersion: v1
kind: ConfigMap
  name: aws-auth
  namespace: kube-system
  mapRoles: |
    # statically map arn:aws:iam::000000000000:role/KubernetesAdmin to cluster admin
    - roleARN: arn:aws:iam::000000000000:role/KubernetesAdmin
      username: kubernetes-admin
      - system:masters    
# [... for worker node authentication]

I created a kubernetesAdmin role for all my trusted colleagues who need access to the cluster by making us all part of an IAM group, which has the following role access.

# get your account ID
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account')

# define a role trust policy that opens the role to users in your account (limited by IAM policy)
POLICY=$(echo -n '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::'; echo -n "$ACCOUNT_ID"; echo -n ':root"},"Action":"sts:AssumeRole","Condition":{}}]}')

# create a role named KubernetesAdmin (will print the new role's ARN)
aws iam create-role \
  --role-name KubernetesAdmin \
  --description "Kubernetes administrator role (for AWS IAM Authenticator for Kubernetes)." \
  --assume-role-policy-document "$POLICY" \
  --output text \
  --query 'Role.Arn'

Run kubectl get nodes to confirm that everything is working.

Run this application with:

{% highlight bash %} $ cd out-of-cluster-configuration $ go build -o app main.go $ ./app There are 12 pods in the cluster Pod example-xxxxx in namespace default not found {% endhighlight %}

Creating a Lambda function

In order to run a Lambda function that can interact with our cluster, there is some preparation work required:

  1. Pass the appropriate EKS cluster name, AWS region and IAM authentication role required for EKS access via environmental variables into the lambda function
  2. Import the heptio authenticator - now a Kubernetes SIG - package into the lambda function, so the go client can generate an authentication token
  3. Create a RESTful call into your Kubernets API server, inserting the authentication Bearer Token
  4. Ensure the lambda function has appropriate access to assume the K8 necessary authentication role needed for token generation

Lambda role

When you create a lambda function, you must assign it an IAM role. The role I created to accompany my lambda function has 3 associated policies:

Bearer Token

It is possible to directly access the API server, if the RESTful call is in the right format. This is described in the kubernetes documentation. Here is an example using curl. Notice the Bearer Token. We use the same element to pass the encoded IAM role which is generated by the external/plug-in authenticator.

We create a similar RESTful call using the go client.

$ APISERVER=$(kubectl config view | grep server | cut -f 2- -d ":" | tr -d " ")
$ TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d '\t')
$ curl $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure
  "kind": "APIVersions",
  "versions": [
  "serverAddressByClientCIDRs": [
      "clientCIDR": "",
      "serverAddress": ""

The API server decodes the bearer token, and then uses AWS STS, to verify that the user/lambda function has legitimate access to the given role. This API server decoding is roughly equivalent to the following:

curl -X GET \
     -H "accept: application/json" \
     -H "x-k8s-aws-id: $CLUSTERNAME" \
     $(aws-iam-authenticator token -i $CLUSTERNAME | \
        jq -r ".status.token" | \
        sed 's/k8s-aws-v1\.//' | \
        base64 -D)

The role must be properly mapped via a ConfigMap (as previously shown above) to a kubernetes cluster role, before the command can be accepted.


Once everything is in place, and your function is uploaded - invoke it.

$ aws lambda invoke --function-name eksClient \
		--region us-east-1 \
		--log-type Tail - \
		| jq '.LogResult' -r | base64 -D

    START RequestId: b4e7770a-a961-11e8-9d7a-e18d853267af Version: $LATEST
    There are 31 pods in the cluster
    Pod example-xxxxx in namespace default not found
    END RequestId: b4e7770a-a961-11e8-9d7a-e18d853267af
    REPORT RequestId: b4e7770a-a961-11e8-9d7a-e18d853267af	Duration: 4859.56 ms	Billed Duration: 4900 ms 	Memory Size: 256 MB	Max Memory Used: 145 MB

So, we have demonstrated how it is possible to use a lambda function to interact with your EKS cluster. I would recommend additional error checking be added before you use this lambda for production use.

The working code is located in the following repository.


Go client package

AWS Go SDK version 1

AWS Go SDK version 2

AWS Lambda using Go

Repo for blog

Kubernetes Authorization