Configuring Default Resource Requests, Limits, and Quotas on your new OpenShift 4 Cluster
There are numerous tasks that need to be done to ready a newly installed OpenShift cluster for use by developers. One such task is configuring default requests and limits, as well as project quotas.
Note: All examples here were developed/tested on OpenShift 4.5.5
Note: In bash shell code examples, the $ at the beginning is just used to indicate a command prompt. If you are copy/pasting commands with me, do not include the leading $ in your paste
Most developers will not set requests/limits on their project. This is not due to laziness or apathy. It’s more of an “out of sight, out of mind” problem. However, failure to do so makes the resources their pods are allowed to consume effectively unlimited, which as a system administrator is not what you want. This can allow some Pods and containers to starve the system of resources, which can negatively impact other teams and users of the system.
You can configure the default project template to include requests and limits, such that if a developer forgets, they will be given a sensible default which can be tailored by the system administrator to be appropriate for the environment and cluster.
Because this template is only invoked for new projects, it’s important to note that it will not affect older projects that may be missing explicit requests and limits.
If the bootstrap project template has not yet been created, you can generate one with the
$ oc adm create-bootstrap-project-template -o yaml > template.yaml
You can see that this is just a normal OpenShift Template. What makes it special is that this is the Template that will be used to create new projects, such as when a user runs
oc new-project <project-name>.
The objects defined here will get created in each new project space.
Notice that there are some helpful variables already defined. We will make use of these with the new objects we are going to create.
The first object we’ll create is a LimitRange that will apply to each Pod and/or Container. Let’s define a LimitRange object that includes
default settings. These will get applied to applied to any Pod which doesn’t explicitly declare its CPU and Memory requests:
In this Template we are saying that any Pods that don’t specify a request will be given half a CPU and 500Mi of memory. The limit will be set to 1 CPU and 1Gi of memory. Because the requests and limits are both set and they are not the same number, the Quality of Service here will be Burstable.
Next, we need to set a ResourceQuota for the new project. Unlike the LimitRange, the ResourceQuota applies to an entire namespace, and is the sum of all resources consumed. There are a number of different resources you can apply a quota to, but the most common ones are Requests/Limits CPU/Memory, storage, and number of Pods.
Let’s create a simple ResourceQuota that allows 10 Pods in the namespace, up to 6 CPUs, and 16Gi of memory.
Now that we have these two objects defined, we are ready to add them to the
objects array in our project template. Here is our complete new project template that we will be setting as the default template for creating new projects:
Notice that we did not change anything in the previous template, other than adding our LimitRange and ResourceQuota to it.
Let’s go ahead and apply this to the cluster. Make sure to apply this to the correct namespace. It must to go into the
The namespace can be specified either on the command line, or in the YAML file itself. I prefer the latter strategy personally, but since the generated Template does not include a namespace in the YAML, I followed suit and left it out and specified on the command line in this case:
$ oc apply -f template.yaml -n openshift-config
Lastly we need to configure the cluster to use our Template for bootstrapping new projects. Let’s do so using the
oc patch command:
Your new project template is installed!
But as they say, the proof is in the pudding, so let’s try creating a new project and verify the new objects are created:
Great! If you get an error on project creation, try a few times and possibly try running the
oc patch command again a few times too. For some reason I've seen it get stuck for a second or two and need another kick.
You can visually inspect the project config as well:
$ oc get project.config.openshift.io/cluster -o yaml
<YAML in here for you to inspect>
Make sure our template’s name “
project-request” is referenced at the bottom of the project config in the spec section:
Or just dump the whole project config to the terminal, and look for the spec section at the bottom:
$ oc get project.config.openshift.io cluster -o yaml
Things to look out for
1. This can affect Quality of Service
The requests and limits on a Pod will affect Quality of Service (QoS), so you should familiarize yourself with that concept to avoid creating a policy that has undesired side effects. For example, you may want to avoid setting the request and limits to the same number. This causes your Pods to run with “Guaranteed” QoS, which may not be what you want for every Pod in the cluster.
2. Slightly complicated rules
There are quite a few “rules” of logic regarding requests and limits, mins and maxes. I would recommend you familiarize yourself with them by reading the documentation for limit ranges.
Some cluster administrators like to set the requests very low as a means of “requiring” developers to specify. If they do not specify, then their Pods will be starved of resources. However, this can cause a couple of problems, a classic example is:
A. Developers will be surprised and confused if limits are super low, and will waste development time troubleshooting
You will want to do a good job disseminating knowledge about the defaults. Even with a respectable dissemination effort, there will still be plenty of people that miss the news and are surprised when their Pod struggles to run. These developers may not know how to fix their problem (particularly as many developers know only the bare minimum about Kubernetes), and may waste considerable time spinning their wheels.
3. Strict quotas can prevent deploys
Quotas are useful tools to prevent overuse and abuse of the system, particularly in limited capacity settings. However, much of the time the point of discovery that you have fulfilled a quota limit is when you are trying to deploy your application. This can be a most inconvenient time to be wrangling quotas, particularly if the team managing the deployment does not have control of the quota! It is good to keep this in mind to avoid causing problems for development teams. I highly recommend continually monitoring quotas, so that when a namespace is close to filling up, the proper team can be alerted ahead of time rather than having their deploy fail.
Default requests, limits, and quotas are not difficult to administer when one knows how. Hopefully this post has made it easy to understand what is going on.