# Kubernetes {% hint style="info" %} Cortex connects to many third-party vendors whose system interfaces frequently change. As a result, integration behavior or configuration steps may shift without notice. If you encounter unexpected issues, check with your system administrator or refer to the vendor's documentation for the most current information. Additionally, integration sync times vary and are subject to scheduling overrides and timing variance. {% endhint %} [Kubernetes](https://kubernetes.io) is a container orchestration system that automates software deployment, scaling, and management. The Cortex K8s agent is a lightweight agent that collects information from your cluster (Deployments, StatefulSets, Argo Rollouts, and CronJobs) and surfaces it in your Cortex workspace's catalog, Scorecards, and more. Integrating Kubernetes with Cortex allows you to: * [Discover and import services](#connecting-cortex-entities-to-kubernetes) directly from K8s clusters into Cortex, making it easy to keep the catalog in sync with what's actually running in production * [View Kubernetes data on entity pages](#view-kubernetes-data-on-entity-pages) in Cortex, giving you visibility into your infrastructure and how services are deployed * Create [Scorecards](#scorecards-and-cql) to track progress and drive alignment on projects relating to Kubernetes, and to enforce Kubernetes best practices ## How to configure Kubernetes with Cortex ### Prerequisites Before getting started: * [Reach out to the Cortex customer engineering team](#still-need-help) for the Helm chart used for deployment and a username and password. * [Generate an API key](https://docs.cortex.io/configure/settings/api-keys) in Cortex. * The API key should have the `User (edit catalog entities)` role at a minimum. * Note: It is also possible to programmatically create your API key via the [Cortex API](https://app.gitbook.com/s/nPgS8L9MAPtoOtdWdeDp/readme/api-keys). #### Security considerations The Cortex k8s agent uses a push model that ensures you do not need to expose your cluster to the public internet. Additionally, the Helm chart comes with a predefined `ClusterRole` that provides the correct RBACs: * **Permissions:** `["get", "watch", "list"]` * **Resources:** `["deployments", "services", "pods", "replicationcontrollers", "statefulsets", "rollouts", "cronjobs"]` * **API groups:** `["apps", "argoproj.io", "batch"]` Communication out of the cluster to Cortex happens over HTTPS. There is no inbound traffic to the agent. ### Install the Cortex k8s agent in your Kubernetes cluster To connect Cortex to your Kubernetes instance, you’ll need to install the Cortex k8s agent in your Kubernetes cluster. The agent is lightweight and adds negligible impact to your cluster. 1. Create a Docker image pull secret: ``` kubectl create secret docker-registry cortex-docker-registry-secret \ --docker-server=ghcr.io \ --docker-username={provided by Cortex} \ --docker-password={token provided to you by the Cortex team} \ --docker-email={email address} ``` 2. Run the following command, replacing `cortex-key` with the value of your Cortex API key, to create a secret in your cluster: ``` kubectl create secret generic cortex-key --from-literal api-key= ``` 3. Run the following command to install the Helm chart provided by Cortex: ``` helm install ./helm-chart ``` ## Connecting Cortex entities to Kubernetes ### Discovery By default, Cortex will use the [Cortex tag](https://docs.cortex.io/entities#cortex-tag) (e.g. `my-entity`) as the "best guess" for Kubernetes resource. For example, if your Cortex tag is `my-entity`, then the corresponding resource in Kubernetes should also be `my-entity`. If your Kubernetes resource don’t cleanly match the Cortex tag, you can override this in the Cortex entity descriptor. ### Methods for mapping Kubernetes resources See the table below for the methods of mapping resources to entities:
MethodUse caseAction
Annotation-based mappingServices that own their K8s infraBy default, Cortex maps Kubernetes deployments with a cortex.io/tag annotation to Cortex entities with the same tag.

Annotation mapping should be at the default absolute path of .metadata.annotations."cortex.io/tag".
Label-based auto-mappingShared infra or external-managed servicesSpecify a list of label keys in the Kubernetes integration settings page of your Cortex workspace
Manual link in entity YAMLComplex or legacy workloadsAdd the resource manually to your entity descriptor
See the tabs below to learn how to use each option: {% tabs %} {% tab title="Annotation" %} **Annotation** You can link your Kubernetes deployment to a Cortex entity by [adding an annotation](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) to your k8s deployment metadata. By default, Cortex maps Kubernetes deployments with a `cortex.io/tag` annotation to Cortex entities with the same tag. Use `cortex.io/tag` as the key and use the value of `x-cortex-tag` in the Cortex entity's `cortex.yaml` as the value. For example, if the `cortex.yaml` file is: ```yaml openapi: 3.0.1 info: title: My Service x-cortex-tag: my-service x-cortex-type: service description: This is my cool service. ``` Then the `deployment.yaml` file should be configured as: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-name namespace: my-namespace annotations: cortex.io/tag: my-service ``` **Customize annotation mapping** It is possible to customize annotation mapping in Cortex:
Customize annotation mapping 1. In Cortex, navigate to the [Kubernetes settings page](https://app.getcortexapp.com/admin/integrations/k8s): * Click **Integrations** from the main nav. Search for and select **Kubernetes**. 2. Optionally enter a JQ mapping into the **Annotation mapping** field. 3. Click **Save mapping**. Note that Cortex looks at the top-level metadata annotations on the Deployment object itself (`metadata.annotations`), not the pod template annotations (`spec.template.metadata.annotations`). If your automapping is not working as expected, make sure the annotation mapping is at the correct default absolute path of `.metadata.annotations."cortex.io/tag"`, or update the annotation mapping in the K8s configuration page to match the exact absolute path of the Cortex tag. **Example** Let's say, for example, your `deployment.yaml` includes `my.service` as the `cortex.io/tag`: ```yaml metadata: name: my-name namespace: my-namespace annotations: cortex.io/tag: my.service ``` If this deployment should be mapped to a Cortex entity with the tag `my-entity`, you can enter the following JQ expression to convert all periods in the deployment annotation tag to dashes: ``` .metadata.annotations."cortex.io/tag" | gsub("\\."; "-") ```
{% endtab %} {% tab title="Label-based" %} **Label-based auto-mapping** You can override [Cortex tag](https://docs.cortex.io/entities#cortex-tag) discovery and have Cortex discover Kubernetes resources using their metadata labels instead:
Customize label-based auto-mapping 1. In Cortex, navigate to [**Integrations > Kubernetes**](https://app.getcortexapp.com/admin/integrations/k8s). 2. Under the **K8s auto-mapping customization**, specify a list of metadata label keys. 3. When you are finished, click **Save**. Once the list is saved, Cortex will discover all Kubernetes resources with metadata labels with the following criteria: * The resource's spec metadata key contains any of the specified labels * The key values match a Cortex entity tag **Example** For example, let's say you have two Cortex entities (`example` and `entity`), and the following Kubernetes JSON blob: ``` { "name": "Sample Kubernetes resource", "metadata": { "labels": { "app": "example", "another": "entity" } } } ``` By default, `example` and `entity` will have no Kubernetes resource mappings. If the list of metadata labels is set to `["app"]`, then entity `example` will be associated with "Sample Kubernetes resource." If the list is set to `["app", "another"]`, then both `example` and `entity` will be associated with the resource.
{% endtab %} {% tab title="Entity YAML" %} **Editing the entity descriptor** Cortex accepts several k8s resources, which can be on different clusters or of different types: deployments, ArgoCD rollout, StatefulSet, and CronJob. All of these resource types have the same field definitions: | Field | Description | Required | | ------------ | -------------------------------------------------------------- | :------: | | `identifier` | `namespace/name` as found in Kubernetes | **✓** | | `cluster` | The name of the cluster, which is set when deploying the agent | | **Deployments** ```yaml x-cortex-k8s: deployment: - identifier: namespace/name cluster: dev - identifier: experiment/scratch cluster: dev - identifier: default/cortex cluster: prod ``` **ArgoCD Rollout** ```yaml x-cortex-k8s: argorollout: - identifier: namespace/name cluster: dev ``` **StatefulSet** ```yaml x-cortex-k8s: statefulset: - identifier: namespace/name cluster: dev ``` **CronJob** ```yaml x-cortex-k8s: cronjob: - identifier: namespace/name cluster: dev ``` {% endtab %} {% endtabs %} ### Import entities from Kubernetes See the [Create services documentation](https://docs.cortex.io/entities/adding-entities/add-services#creating-services) for instructions on manually importing entities. ## Using the Kubernetes integration ### View Kubernetes data on entity pages Kubernetes deployment data will be available in the **Kubernetes** block on the [entity details pages](https://docs.cortex.io/ingesting-data-into-cortex/entities/details) for entities imported from Kubernetes or linked to a k8s resource.
Kubernetes data appears in the Kubernetes block on the entity details page overview.
In the entity's sidebar, click **Environments** to see Kubernetes deployments, clusters, active replicas, and pending deployments, as well as: * **Replicas:** Number of available, ready, and desired replicas. * **Containers:** Resource containers, including requested memory, memory limit, and CPU data. Also includes the full container definition.
### Scorecards and CQL With the Kubernetes integration, you can create Scorecard rules and write CQL queries based on Kubernetes resources. For an example, see Cortex's prebuilt [Kubernetes Deployment Baseline Scorecard](https://app.gitbook.com/s/iu4HGhtyt7D4jcoIrCBE/migrations-and-modernization/accelerate-migration-to-k8s#create-a-kubernetes-scorecard) template. See more rule examples in the [CQL Explorer](https://app.getcortexapp.com/admin/cql-explorer) in Cortex.
Cluster information Data about k8s clusters associated with a given entity. **Definition:** `k8s.clusters()` **Examples** You can use the `k8s.clusters()` expression in the Query Builder to find all clusters that start with "dev": ``` k8s.clusters.all((cluster) => cluster.name.matches("""dev-.*""")) ``` Or any cluster named "prod": ``` k8s.clusters.any((cluster) => cluster.name.matches("prod")) ```
Deployment labels Checks deployment metadata. **Definition:** `k8s.metadata()` **Examples** You can use this expression in a production readiness Scorecard to check ownership: ``` k8s.metadata().labels.any((label) => label.get("ownership") == "ownership_team") ``` This rule checks an entity's metadata labels for the ownership annotation and will pass if "ownership\_team" is defined. You can also use this expression in the Query Builder to find all k8s deployments with the label "environment": ``` k8s.metadata().labels.all((label) => label.containsKey("environment")) == true ``` Or you could refine the query further to find k8s deployments with an "environment" label and that are in production: ``` k8s.metadata().labels.all((label) => label.get("environment")?.matches("prod")") == true ```
K8s resource is set for entity Checks whether a k8s resource of any type is associated with an entity. **Definition:** `k8s != null` **Example** For a Scorecard focused on automation or development maturity, you can set a rule to make sure a k8s resource is mapped: ``` k8s != null ```
Kubernetes spec YAML The [Cortex k8s agent](#install-the-cortex-k8s-agent-in-your-kubernetes-cluster) periodically sends the raw spec definitions for all entities. The spec JSON is equivalent to the **root spec field** of the entity descriptor (deployments, StatefulSet, etc.) and fully conforms to that format. You can find the official documentation for these resource objects in the [Kubernetes Workload Docs](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/). You can use this list of JSON specs combined with jq or [Open Policy Agent (OPA) language](https://www.openpolicyagent.org/docs/v0.52.0/) to write complex assertions such as "all resources must have specific annotations set" or "all containers should have a CPU resource limit defined." The list of JSON specs can also be filtered to only ones in a specific cluster by specifying the cluster name: `k8s.spec("prod")`. **Definition:** `k8s.spec()` **Examples** You can use this expression to write a wide range of rules. For a best practices Scorecard, you can make sure that resource definitions have set CPU requests: ``` jq(k8s.spec(), ".[].template.spec.containers[].resources.requests.cpu") != null ``` Or that all resource definitions expose only TCP ports: ``` jq(k8s.spec(), ".[].template.spec.containers[].ports[].protocol") == "TCP" ```
Replica information Number of replicas available, current, desired, ready, unavailable, or updated. **Definition:** `k8s.replicas()` **Example** You can use this expression in a development maturity Scorecard to make sure an entity has at least two available instances: ``` k8s.replicas().numAvailable >= 2 ```
## Background sync The Cortex k8s agent is a cron job that runs every 5 minutes by default. ## FAQs and troubleshooting #### **When I try to import entities, I don't see all the supported workload types (deployments, ArgoCD rollout, StatefulSet, CronJob)** Make sure that the types you expected to see are in the cluster you are attempting to import. #### **Missing namespaces from Kubernetes discovery** If you're using [Cortex's k8s agent](#cortex-k8s-agent) to import entities into Cortex but don't see all expected namespaces during the import process, make sure `app.namespace` is commented out in `values.yaml`: ``` app: # baseURL: baseURL: keySecret: # namespace: exampleNamespace ``` If `app.namespace` is defined the Cortex k8s agent will only be able to discover services from that namespace. This behavior can be confirmed with a backend log similar to: ``` INFO 1 --- [ scheduling-1] k8sagent : Looking for stateful sets in namespace ``` Once `app.namespace` is commented out, restart your pods. You will then be able to see all expected namespaces when importing new services. #### **Helm chart and deprecated Kubernetes Docker registry** If your Cortex agent in Kubernetes clusters is blocked due to deprecation of Docker registry after an upgrade, you can make these direct edits using the same credentials: 1. **Access the image** from `ghcr.io` instead of `docker.pkg.github.com`. ``` image: ghcr.io/cortexapps/k8s-agent... ``` 2. **Update the registry secret**, setting the server to `https://ghcr.io`. If you are unable to make these changes, please reach out to and request a new Helm chart with this change already reflected. #### **Failing ArgoCD rollouts error in the k8s agent** When running the self-hosted Kubernetes agent successfully, users may see failing ArgoCD rollouts errors while not using this tool. ``` Error polling argocd rollouts from Kubernetes API io.kubernets.client.openapi.ApiException: [...] at com.brainera.k8sSDKClient.getArgoRollouts(k8sClient.kt:101) ~[app:/na] ``` Cortex logs this exception for verbosity - this error is harmless if not using ArgoCD tool. #### **Can I deploy on prem if I don’t use Kubernetes?** Yes - the Cortex Helm chart deploys two Cortex-specific pods from images for the frontend and backend, as well as a data store. You can use these images to run Docker containers on other platforms, such as ECS. ## Still need help?[​](https://docs.cortex.io/docs/reference/integrations/aws#still-need-help) The following options are available to get assistance from the Cortex Customer Engineering team: * **Email**: , or open a support ticket in the in app Resource Center * **Slack**: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a `:ticket:` reaction to a question in Slack, and the team will respond directly. Don’t have a Slack channel? Talk with your Customer Success Manager. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.