Kubernetes Basics for Developers
This is a document by the Platform Services and Infrastructure team originally internal use at Dollar Shave Club.
We believe that it’s important for communication to be efficient between the different engineering groups. The only way that we can do this effectively is if different groups have a shared understanding of the “slang” and concepts used by different groups.
This document is our attempt at giving a brief overview of the “slang” and concepts for developers who don’t have a lot of time to dig into the Kubernetes documentation. This way we can more easily express different ideas when speaking about specific technologies.
Now to the document…
This document attempts to give a quick overview of the different components of Kubernetes and how they interact. Though the title says this is just for developers, it is not. It is for anyone who is new to Kubernetes and might have heard a term they didn’t understand when discussing an app.
In an idealized world, developers wouldn’t need to care about this information. But it is always good to have a basic understanding of the platform your application is running on top of to better understand how your application functions.
What is Kubernetes?
Kubernetes (commonly abbreviated as k8s) allows us to schedule an application to run without caring which computer it is run on.
Kubernetes is a set of components that creates a resource scheduler used across a set of nodes or computers called a cluster. The user requests that one or more applications be run by Kubernetes and it determines which nodes have enough resources available and runs the applications on the nodes it found. Kubernetes does also provide us with some nice built-in systems such as service discovery and autoscaling.
Kubernetes constantly checks all of the configurations provided by users against the current state of the cluster and will do what is needed to get the configuration and the system in sync. This way if your application fails or if a new deployment is discovered, Kubernetes will make the updates as soon as it is discovered. Kubernetes is considered an eventually convergent system because it always works to get to the proper state.
Components of Kubernetes
Below are definitions for common terms that are used when discussing Kubernetes.
Pods
A pod is a unit of one or more containers deployed as a group onto a single node in the cluster. There will always be a container that runs the app along with zero or more helper containers called “sidecars”. They are always co-located onto the same node and the containers within a pod can reference other containers using localhost. Pods are usually created and managed through objects such as a Deployment or Job among others.
API Servers
The API servers are the public endpoint for users of the cluster when requesting that some action is to be performed. You should not be directly concerned with the API servers.
Node
A node is a computer that is part of the cluster that runs the kublet process. You should not need to worry about nodes for the majority of what you do on Kubernetes.
Kubelet
A kubelet is a daemon running on all of the worker nodes that communicates with the Kubernetes API servers. The node reports the amount of resources available on its node so that the API server can decide where to run a pod. You should not need to worry about the kubelet.
Namespace
A namespace is a logical grouping of different resources for a common application or environment in a Kubernetes cluster. We are able to group together a set of pods, services, ingresses, etc that all relate to the same group of application(s). For example, we currently group together applications based on the application as well as the environment it is running in (i.e. namespaces called “main-site-production” and “main-site-release”).
App
The app is the custom code that Kubernetes is being asked to run. This is domain specific logic for DSC and is usually written by engineers at DSC but could also be a third-party application that we’ve decided to run using Kubernetes.
Traditionally an app might be called a “service” but as you’ll see this can become confusing in Kubernetes. So, we’ve decided to refer to our custom applications as apps.
Sidecar
A container that performs a task for an app that is not custom business logic but is required for the proper function of the app. For example, if Consul is used to discover other apps inside the cluster, a Consul agent container will run as part of every pod to help with communicating with the Consul servers.
Job
A job is a pod that is run to complete a task that does not require the app to run at all times. Jobs are usually run for tasks such as a database migration that is needed before a new version of an app can properly function.
ReplicaSet
A ReplicaSet controls how many pods are running for a given app. The ReplicaSet will make sure that the correct number of pods is running for a given deployment at all times. If a pod fails and goes missing, the ReplicaSet will be used to launch another pod on the same or a different node. A ReplicaSet is usually not manually created, but is managed through the higher level Deployment object.
Deployment
A deployment defines a configuration for a ReplicaSet which includes how many pods should be running as well as the different configuration options to supply to the running pods.
A deployment will also define how the new pods are deployed. We are able to define how new pods should be created and how the old pods are deleted. For example, we can have Kubernetes slowly rollout a new version of an app by launching a few new pods and then deleting a few of the old pods repeatedly until all of the old pods have been removed. This allows us to get a new version of the app into the environment with little issue for the end users.
Service
A service provides apps with an IP address and a DNS name (provided through the CoreDNS service) to access the pods running for another app. This service endpoint will send our request to one of the running and available pods that are part of the service using round-robin selection. The service is able to identify pods that should be grouped by using a set of label attached to the pods. Services are the primary method of service discovery in Kubernetes.
Although you can get the IP address for a service, it is suggested that you use DNS as the IP has the possibility of changing. With the DNS name you will automatically pick that up but with an IP address you would most likely need to deploy your app before it can be updated.
For example, DNS names for services are in the form, “<name>.<namespace>.svc.cluster.local”. An example of a service DNS name for the service named “my-service” in the “my-service-production” namespace would be, “my-service.my-service-production.svc.cluster.local”. There are also other DNS names available for other parts of the system as well.
Ingress
An ingress allows us to define how traffic from outside of the cluster can reach a particular services inside the cluster. The Ingress generally takes HTTP traffic and routes it based on the hostname header of the request. This is similar to virtual hosts or OSI layer 7 based routing that has been traditionally done in web servers such as Apache or Nginx.
Ingress Controller
An ingress controller is the system that does the routing for the ingress of a service. The ingress controller is usually running a software app such as Envoy, Nginx or a custom built solution like Guardian. The ingress controller will dynamically update its configuration as services are updated and deployed. The ingress controller will create a configuration file based on all of the ingresses that have been discovered.
kubectl
kubectl is a tool for accessing the Kubernetes API through the command line. It is what you will usually use when looking for a namespace, a set of pods or for a service.
It is suggested that you install kubectl on your local machine using the homebrew package manager.
Helm
Helm is the tool we currently use for deployments. Helm allows us to template out the YAML configurations for the different parts of a app deployment on Kubernetes. The templating allows us to easily use the same configuration between different environments such as production and DQAs with little duplication.
App Network Flow
How do users access public domains such as dollarshaveclub.com?
For public apps, users will usually access the app through our Content Distribution Network (CDN) Fastly. Fastly will then interpret the request and forward the request to an AWS Elastic Load Balancer (ELB) that handles that service. Most services use a shared AWS ELB so the request is forwarded to the ingress controller that has been configured by the various app ingress configurations. The request is finally sent to the pods in the deployment using the service DNS name which sends the request to a single pod that is part of the service. The pod hopefully completes the request and the response is sent back up the chain to the original requester.
How does a service inside of Kubernetes talk to another?
When two apps running inside of Kubernetes want to interact they do so using the services attached to the two apps. Let’s say that App A has a service called “Service A” and App B has a service called “Service B”. For example, App A would talk to App B using the DNS name for Service B. The DNS name for Service B resolves to the IP address for Service B which then tells Kubernetes to send the request to an App B pod using round-robin selection. The App B pod will hopefully complete the request and return a response to the App A pod that made the request.
It is important to note that Kubernetes will randomly choose a pod for every connection. So, if each request opens and closes a connection, each request will most likely be serviced by a different running pod. If the app keeps the connection open, multiple requests can be set to the same pod as needed.
For more information, make sure to checkout the proper Kubernetes documentation.
