Kubernetes

Kubernetes is a container orchestration system that automates software deployment, scaling, and management. The Cortex K8s agent is a lightweight agent that collects information from your cluster (Deployments, StatefulSets, Argo Rollouts, and CronJobs) and surfaces it in your Cortex workspace's catalog, Scorecards, and more.

Integrating Kubernetes with Cortex allows you to:

• Discover and import services directly from K8s clusters into Cortex, making it easy to keep the catalog in sync with what's actually running in production

• View Kubernetes data on entity pages in Cortex, giving you visibility into your infrastructure and how services are deployed

• Create Scorecards to track progress and drive alignment on projects relating to Kubernetes, and to enforce Kubernetes best practices

How to configure Kubernetes with Cortex

Prerequisites

Before getting started:

• Reach out to the Cortex customer engineering team for the Helm chart used for deployment and a username and password.

• Generate an API key in Cortex.

  • The API key should have the User (edit catalog entities) role at a minimum.

  • Note: It is also possible to programmatically create your API key via the Cortex API.

Security considerations

The Cortex k8s agent uses a push model that ensures you do not need to expose your cluster to the public internet.

Additionally, the Helm chart comes with a predefined ClusterRole that provides the correct RBACs:

• Permissions: ["get", "watch", "list"]

• Resources: ["deployments", "services", "pods", "replicationcontrollers", "statefulsets", "rollouts", "cronjobs"]

• API groups: ["apps", "argoproj.io", "batch"]

Communication out of the cluster to Cortex happens over HTTPS. There is no inbound traffic to the agent.

Install the Cortex k8s agent in your Kubernetes cluster

To connect Cortex to your Kubernetes instance, you’ll need to install the Cortex k8s agent in your Kubernetes cluster. The agent is lightweight and adds negligible impact to your cluster.

1. Create a Docker image pull secret:

   kubectl create secret docker-registry cortex-docker-registry-secret \
   --docker-server=ghcr.io \
   --docker-username={provided by Cortex} \
   --docker-password={token provided to you by the Cortex team} \
   --docker-email={email address}

2. Run the following command, replacing cortex-key with the value of your Cortex API key, to create a secret in your cluster:

   kubectl create secret generic cortex-key --from-literal api-key=

3. Run the following command to install the Helm chart provided by Cortex:

   helm install  ./helm-chart

Connecting Cortex entities to Kubernetes

Discovery

By default, Cortex will use the Cortex tag (e.g. my-entity) as the "best guess" for Kubernetes resource. For example, if your Cortex tag is my-entity, then the corresponding resource in Kubernetes should also be my-entity.

If your Kubernetes resource don’t cleanly match the Cortex tag, you can override this in the Cortex entity descriptor.

Methods for mapping Kubernetes resources

See the table below for the methods of mapping resources to entities:

Method	Use case	Action
Annotation-based mapping	Services that own their K8s infra	By default, Cortex maps Kubernetes deployments with a cortex.io/tag annotation to Cortex entities with the same tag. Annotation mapping should be at the default absolute path of .metadata.annotations."cortex.io/tag".
Label-based auto-mapping	Shared infra or external-managed services	Specify a list of label keys in the Kubernetes integration settings page of your Cortex workspace
Manual link in entity YAML	Complex or legacy workloads	Add the resource manually to your entity descriptor

See the tabs below to learn how to use each option:

Annotation

Annotation

You can link your Kubernetes deployment to a Cortex entity by adding an annotation to your k8s deployment metadata. By default, Cortex maps Kubernetes deployments with a cortex.io/tag annotation to Cortex entities with the same tag.

Use cortex.io/tag as the key and use the value of x-cortex-tag in the Cortex entity's cortex.yaml as the value.

For example, if the cortex.yaml file is:

openapi: 3.0.1
info:
  title: My Service
  x-cortex-tag: my-service
  x-cortex-type: service
  description: This is my cool service.

Then the deployment.yaml file should be configured as:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-name
  namespace: my-namespace
  annotations:
    cortex.io/tag: my-service

Customize annotation mapping

It is possible to customize annotation mapping in Cortex:

Customize annotation mapping

1. In Cortex, navigate to the Kubernetes settings page:

   • Click Integrations from the main nav. Search for and select Kubernetes.

2. Optionally enter a JQ mapping into the Annotation mapping field.

3. Click Save mapping.

Note that Cortex looks at the top-level metadata annotations on the Deployment object itself (metadata.annotations), not the pod template annotations (spec.template.metadata.annotations).

If your automapping is not working as expected, make sure the annotation mapping is at the correct default absolute path of .metadata.annotations."cortex.io/tag", or update the annotation mapping in the K8s configuration page to match the exact absolute path of the Cortex tag.

Example

Let's say, for example, your deployment.yaml includes my.service as the cortex.io/tag:

metadata:
  name: my-name
  namespace: my-namespace
  annotations:
    cortex.io/tag: my.service

If this deployment should be mapped to a Cortex entity with the tag my-entity, you can enter the following JQ expression to convert all periods in the deployment annotation tag to dashes:

.metadata.annotations."cortex.io/tag" | gsub("\\."; "-")

Label-based

Entity YAML

Import entities from Kubernetes

See the Create services documentation for instructions on manually importing entities.

Using the Kubernetes integration

View Kubernetes data on entity pages

Kubernetes deployment data will be available in the Kubernetes block on the entity details pages for entities imported from Kubernetes or linked to a k8s resource.

In the entity's sidebar, click Environments to see Kubernetes deployments, clusters, active replicas, and pending deployments, as well as:

• Replicas: Number of available, ready, and desired replicas.

• Containers: Resource containers, including requested memory, memory limit, and CPU data. Also includes the full container definition.

Eng Intelligence

Track and analyze your software delivery performance using real-time data from your Kubernetes environments in Eng Intelligence features.

Deployment data from your clusters is surfaced in Metrics Explorer and in dashboards, such as the DORA Dashboard, and can be used to assess key engineering metrics like deployment frequency, change failure rate, and more.

Scorecards and CQL

With the Kubernetes integration, you can create Scorecard rules and write CQL queries based on Kubernetes resources. For an example, see Cortex's prebuilt Kubernetes Deployment Baseline Scorecard template.

See more rule examples in the CQL Explorer in Cortex.

Cluster information

Data about k8s clusters associated with a given entity.

Definition: k8s.clusters()

Examples

You can use the k8s.clusters() expression in the Query Builder to find all clusters that start with "dev":

k8s.clusters.all((cluster) => cluster.name.matches("""dev-.*"""))

Or any cluster named "prod":

k8s.clusters.any((cluster) => cluster.name.matches("prod"))

Deployment labels

Checks deployment metadata.

Definition: k8s.metadata()

Examples

You can use this expression in a production readiness Scorecard to check ownership:

k8s.metadata().labels.any((label) => label.get("ownership") == "ownership_team")

This rule checks an entity's metadata labels for the ownership annotation and will pass if "ownership_team" is defined.

You can also use this expression in the Query Builder to find all k8s deployments with the label "environment":

k8s.metadata().labels.all((label) => label.containsKey("environment")) == true

Or you could refine the query further to find k8s deployments with an "environment" label and that are in production:

k8s.metadata().labels.all((label) => label.get("environment")?.matches("prod")") == true

K8s resource is set for entity

Checks whether a k8s resource of any type is associated with an entity.

Definition: k8s != null

Example

For a Scorecard focused on automation or development maturity, you can set a rule to make sure a k8s resource is mapped:

k8s != null

Kubernetes spec YAML

The Cortex k8s agent periodically sends the raw spec definitions for all entities. The spec JSON is equivalent to the root spec field of the entity descriptor (deployments, StatefulSet, etc.) and fully conforms to that format.

You can find the official documentation for these resource objects in the Kubernetes Workload Docs.

You can use this list of JSON specs combined with jq or Open Policy Agent (OPA) language to write complex assertions such as "all resources must have specific annotations set" or "all containers should have a CPU resource limit defined."

The list of JSON specs can also be filtered to only ones in a specific cluster by specifying the cluster name: k8s.spec("prod").

Definition: k8s.spec()

Examples

You can use this expression to write a wide range of rules. For a best practices Scorecard, you can make sure that resource definitions have set CPU requests:

jq(k8s.spec(), ".[].template.spec.containers[].resources.requests.cpu") != null

Or that all resource definitions expose only TCP ports:

jq(k8s.spec(), ".[].template.spec.containers[].ports[].protocol") == "TCP"

Replica information

Number of replicas available, current, desired, ready, unavailable, or updated.

Definition: k8s.replicas()

Example

You can use this expression in a development maturity Scorecard to make sure an entity has at least two available instances:

k8s.replicas().numAvailable >= 2

Background sync

The Cortex k8s agent is a cron job that runs every 5 minutes by default.

FAQs and troubleshooting

When I try to import entities, I don't see all the supported workload types (deployments, ArgoCD rollout, StatefulSet, CronJob)

Make sure that the types you expected to see are in the cluster you are attempting to import.

Missing namespaces from Kubernetes discovery

If you're using Cortex's k8s agent to import entities into Cortex but don't see all expected namespaces during the import process, make sure app.namespace is commented out in values.yaml:

app:
  # baseURL: 
  baseURL:
  keySecret:
  # namespace: exampleNamespace

If app.namespace is defined the Cortex k8s agent will only be able to discover services from that namespace. This behavior can be confirmed with a backend log similar to:

INFO 1 --- [ scheduling-1] k8sagent : Looking for stateful sets in namespace 

Once app.namespace is commented out, restart your pods. You will then be able to see all expected namespaces when importing new services.

Helm chart and deprecated Kubernetes Docker registry

If your Cortex agent in Kubernetes clusters is blocked due to deprecation of Docker registry after an upgrade, you can make these direct edits using the same credentials:

1. Access the image from ghcr.io instead of docker.pkg.github.com.

   image: ghcr.io/cortexapps/k8s-agent...

2. Update the registry secret, setting the server to https://ghcr.io.

If you are unable to make these changes, please reach out to [email protected] and request a new Helm chart with this change already reflected.

Failing ArcoCD rollouts error in the k8s agent

When running the self-hosted Kubernetes agent successfully, users may see failing ArgoCD rollouts errors while not using this tool.

Error polling argocd rollouts from Kubernetes API

io.kubernets.client.openapi.ApiException:
[...]
  at com.brainera.k8sSDKClient.getArgoRollouts(k8sClient.kt:101) ~[app:/na]

Cortex logs this exception for verbosity - this error is harmless if not using ArgoCD tool.

Can I deploy on prem if I don’t use Kubernetes?

Yes - the Cortex Helm chart deploys two Cortex-specific pods from images for the frontend and backend, as well as a data store. You can use these images to run Docker containers on other platforms, such as ECS.

Still need help?​

The following options are available to get assistance from the Cortex Customer Engineering team:

• Email: [email protected], or open a support ticket in the in app Resource Center

• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.