Advanced Topics

Note: This page assumes that you’re familiar with core Kubernetes concepts, and are comfortable deploying your own apps. If not, you should review the Intermediate App Developer topics first.

After checking out the current page and its linked sections, you should have a better understanding of the following: * Advanced features that you can leverage in your application * The various ways of extending the Kubernetes API

Now you know the set of API objects that Kubernetes provides. Understanding the difference between a DaemonSetEnsures a copy of a Pod is running across a set of nodes in a cluster. and a DeploymentAn API object that manages a replicated application. is oftentimes sufficient for app deployment. That being said, it’s also worth familiarizing yourself with Kubernetes’s lesser known features. They can be quite powerful when applied to the right use cases.

Container-level features

As you may know, it’s an antipattern to migrate an entire app (e.g. containerized Rails app, MySQL database, and all) into a single Pod. That being said, there are some very useful patterns that go beyond a 1:1 correspondence between a container and its Pod:

• Sidecar container: Although your Pod should still have a single main container, you can add a secondary container that acts as a helper (see a logging example). Two containers within a single Pod can communicate via a shared volume.
• Init containers: Init containers run before any of a Pod’s app containers (such as main and sidecar containers). Read more, see an nginx server example, and learn how to debug these containers.

Pod configuration

Usually, you use labelsTags objects with identifying attributes that are meaningful and relevant to users. and annotationsA key-value pair that is used to attach arbitrary non-identifying metadata to objects. to attach metadata to your resources. To inject data into your resources, you’d likely create ConfigMapsAn API object used to store non-confidential data in key-value pairs. Can be consumed as environment variables, command-line arguments, or config files in a volume. (for nonconfidential data) or SecretsStores sensitive information, such as passwords, OAuth tokens, and ssh keys. (for confidential data).

Below are some other, lesser-known ways of configuring your resources’ Pods:

• Taints and Tolerations - These provide a way for nodes to “attract” or “repel” your Pods. They are often used when an application needs to be deployed onto specific hardware, such as GPUs for scientific computing. Read more.
• Downward API - This allows your containers to consume information about themselves or the cluster, without being overly coupled to the Kubernetes API server. This can be achieved with environment variables or DownwardAPIVolumeFiles.
• Pod Presets - Normally, to mount runtime requirements (such as environmental variables, ConfigMaps, and Secrets) into a resource, you specify them in the resource’s configuration file. PodPresets allow you to dynamically inject these requirements instead, when the resource is created. For instance, this allows team A to mount any number of new Secrets into the resources created by teams B and C, without requiring action from B and C. See an example.

Additional API Objects

Note: Before setting up the following resources, check to see if they are the responsibility of your organization’s cluster operatorsA person who configures, controls, and monitors clusters. .

• Horizontal Pod Autoscaler (HPA)An API resource that automatically scales the number of pod replicas based on targeted CPU utilization or custom metric targets. - These resources are a great way to automate the process of scaling your application when CPU usage or other custom metrics spike. See an example to understand how HPAs are set up.

• Federated cluster objects - If you are running an application on multiple Kubernetes clusters using federation, you need to deploy the federated version of the standard Kubernetes API objects. For reference, check out the guides for setting up Federated ConfigMaps and Federated Deployments.

Kubernetes is designed with extensibility in mind. If the API resources and features mentioned above are not enough for your needs, there are ways to customize its behavior without having to modify core Kubernetes code.

Understand Kubernetes’s default behavior

Before making any customizations, it’s important that you understand the general abstraction behind Kubernetes API objects. Although Deployments and Secrets may seem quite different, the following concepts are true for any object:

• Kubernetes objects are a way of storing structured data about your cluster. In the case of Deployments, this data represents desired state (such as “How many replicas should be running?”), but it can also be general metadata (such as database credentials).
• Kubernetes objects are modified via the Kubernetes APIThe application that serves Kubernetes functionality through a RESTful interface and stores the state of the cluster. . In other words, you can make GET and POST requests to a specific resource path (such as <api-server-url>/api/v1/namespaces/default/deployments) to read and write the corresponding object type.
• By leveraging the Controller pattern, Kubernetes objects can be used to enforce desired state. For simplicity, you can think of the Controller pattern as the following continuous loop:

These states are obtained from the Kubernetes API.

Note: Not all Kubernetes objects need to have a Controller. Though Deployments trigger the cluster to make state changes, ConfigMaps act purely as storage.

Create Custom Resources

Based on the ideas above, you can define a new Custom Resource that is just as legitimate as a Deployment. For example, you might want to define a Backup object for periodic backups, if CronJobs don’t provide all the functionality you need.

There are two main ways of setting up custom resources: 1. Custom Resource Definitions (CRDs) - This method requires the least amount of implementation work. See an example. 2. API aggregation - This method requires some pre-configuration before you actually set up a separate, extension API server.

Note that unlike standard Kubernetes objects, which rely on the built-in kube-controller-manager, you’ll need to write and run your own custom controllers.

You may also find the following info helpful: * How to know if custom resources are right for your use case * How to decide between CRDs and API aggregation

Service Catalog

If you want to consume or provide complete services (rather than individual resources), Service CatalogAn extension API that enables applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider. provides a specification for doing so. These services are registered using Service BrokersAn endpoint for a set of Managed Services offered and maintained by a third-party. (see some examples).

If you do not have a cluster operatorA person who configures, controls, and monitors clusters. to manage the installation of Service Catalog, you can do so using Helm or an installer binary.

References

The following topics are also useful for building more complex applications:

• Other points of extensibility within Kubernetes - A conceptual overview of where you can hook into the Kubernetes architecture.
• Kubernetes Client Libraries - Useful for building apps that need to interact heavily with the Kubernetes API.

What’s next

Congrats on completing the Application Developer user journey! You’ve covered the majority of features that Kubernetes has to offer. What now?

• If you’d like to suggest new features or keep up with the latest developments around Kubernetes app development, consider joining a SIG (special interest group)Community members who collectively manage an ongoing piece or aspect of the larger Kubernetes open source project. such as SIG Apps.

• If you are interested in learning more about the inner workings of Kubernetes (e.g. networking), consider checking out the Cluster Operator journey.