Self-managed Cortex

For organizations with strict security and compliance requirements, Cortex can be deployed on-premises into your own Kubernetes cluster.

The Cortex installation is provided as a Helm chart.

Due to the frequency of our releases and rapid changes to our product, we only offer dedicated support for releases up to two months prior to the current release version. For versions older than this we will provide best-effort support. If you are currently using a release older than two months, we strongly recommend updating to the latest release. If you encounter an issue while using a version of Cortex older than two months, you may be asked to upgrade as part of our ongoing support efforts.

Additional self-managed resources

After following the instructions on this page to get started with your self-managed Cortex account, see these additional guides to enable other features and integrations:

• User management: Enabling SSO, connecting users via GitHub OAuth, configuring email notifications

• Configuring integrations: Atlassian Connect app, AWS, GitHub app, GitHub OAuth, Microsoft Teams, Slack

• Upgrades and rollbacks: Upgrading or rolling back Self-Managed Cortex

Getting started with a Cortex self-managed account​

To deploy Cortex on-premises, you will:

• Make sure you have the required environment and infrastructure set up

• Create a PostgreSQL database where Cortex data will be stored

• Create Kubernetes secrets for pulling software images and connecting to the database

• Create a YAML file containing your configuration values for the Cortex Helm chart

• Preview the Kubernetes deployment

• Deploy using Helm

• Configure DNS records to allow users to access Cortex

Prerequisites

See the prerequisites below:

Prerequisites

Before getting started, you will need:

• A running Kubernetes cluster, configured with permission to make HTTPS requests to the public internet (our Docker images are hosted on GitHub) - you can use AWS/EKS, Azure/AKS, GCP/GKE, or your own locally installed Kubernetes distribution.

• Permission to add services, deployments, secrets and namespaces to the cluster

• kubectl installed and connected to your Kubernetes cluster

• Helm Package Manager installed

• A PostgresSQL database (15+) that is accessible from your k8s cluster, and has a dedicated database and user for Cortex.

  • Learn more under Step 2: Set up a persistent database.

• A GitHub personal access token (PAT) and a JWT license key, both provided to you by your Cortex account team

Required resources in Kubernetes

Below is a list of the containers that make up a minimum Cortex deployment, with their resource requirements:

• 2 instances of backend container, with 8GB memory per instance (2 cores recommended)

• 1 instance of worker container, with 8GB memory (2 cores recommended)

• 1 instance of frontend container

  • 500MB memory is generally sufficient, but you can adjust based on your available resources.

Ensure that your cluster has enough resources available to run the containers you plan to deploy.

DNS and TLS requirements

Before installing Cortex, you need to prepare:

• DNS Names: Create two DNS records in your domain:

  • One for the frontend (web UI): e.g., cortex-app.yourdomain.dom

  • One for the backend (API): e.g., cortex-api.yourdomain.dom

• TLS Certificate: Obtain a TLS certificate that covers both DNS names. You can use:

  • A wildcard certificate (e.g., *.yourdomain.dom)

  • A certificate with Subject Alternative Names (SANs) for both DNS names

  • For AWS: AWS Certificate Manager (ACM)

  • For Azure: Azure Key Vault certificates

• DNS Configuration: After installation, you'll need to point DNS records for the frontend and backend donain names to the two load balancers created by Kubernetes. The exact configuration depends on your environment:

  • For AWS: Point to each NLB's DNS name

  • For Azure: Point to each Load Balancer's public IP

  • For self-managed load balancers: Point to your load balancers' IPs or hostnames

Verify Kubernetes connection

Check kubectl installation and cluster access:

# Check kubectl version
kubectl version --client

# Verify cluster access
kubectl cluster-info

# List nodes to confirm connectivity
kubectl get nodes

Check Helm installation:

# Check Helm version (must be v3.x)
helm version

# List current Helm releases
helm list --all-namespaces

Step 1: Set up a persistent database

Cortex stores all persistent data in a PostgreSQL database. You will need to set up a database in order to install and use Cortex.​

Step 1.1: Database prerequisites

Cortex requires PostgreSQL 15 or higher and does not support other datastores. You have several options, depending on your environment:

• Self-managed PostgreSQL: Install PostgreSQL on your own infrastructure

• AWS RDS: Use Amazon RDS for PostgreSQL if running on AWS

• Azure Database for PostgreSQL: Use Azure's managed PostgreSQL service if running on Azure

You will need the following resources:

• 15GB minimum available storage, 40 GB recommended

• Maximum number of connections ≥100. The maximum number of connections should be ≥2x the backend connection pool size, which is 25 per instance, for a total of 50 when two backend containers are running.

Step 1.2: Create a Postgres database

1. On your PostgreSQL server, create a database with UTF-8 encoding that will be accessible from your instance.

2. Create a user with the following permissions on the public.schema within the database you created:

   • Add ALL on the public schema to create/modify tables

   • Add INSERT, SELECT, UPDATE, DELETE on all tables in the public schema

3. Make a note of the following database connection details, as you will need them in the next steps:

   • The hostname and port number for connecting to the database server

   • The name of the database you created (this guide assumes the database is named cortex)

   • The username and password to access the database

Here is an example of SQL that you might run to create the database and the user with sufficient permissions:

-- Create the cortex database with UTF-8 encoding
CREATE DATABASE cortex ENCODING 'UTF8';

-- Create the cortex user with a password
CREATE USER cortex_user WITH PASSWORD 'strongpassword';

-- Two options for permissions below: granular or admin
-- Permissions Option A: Granular permissions
\c cortex
GRANT USAGE, CREATE ON SCHEMA public TO cortex_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO cortex_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO cortex_user;

-- OR --
-- Permissions Option B: Database-level admin
-- This gives full control over the cortex database but not others.
GRANT ALL PRIVILEGES ON DATABASE cortex TO cortex_user;
\c cortex
GRANT ALL PRIVILEGES ON SCHEMA public TO cortex_user;

Step 1.3: Verify database connection

Verify that containers in your Kubernetes cluster will be able to connect to your Postgres database using the connection details you noted above:

# Set your database connection details
export DB_HOST="<your-postgres-host>"
export DB_PORT="5432"
export DB_NAME="<your-cortex-database>"
export DB_USERNAME="<your-database-user>"
export DB_PASSWORD="<your-database-password>"

# Test database connectivity and permissions
kubectl run pgtools-test --rm -i --restart=Never \
  --namespace cortex \
  --image=postgres:17 \
  --env="PGPASSWORD=${DB_PASSWORD}" \
  --command -- psql \
  -h "${DB_HOST}" \
  -p "${DB_PORT}" \
  -U "${DB_USERNAME}" \
  -d "${DB_NAME}" \
  -c "CREATE TABLE IF NOT EXISTS connection_test (id serial PRIMARY KEY, test_time timestamp DEFAULT NOW()); DROP TABLE connection_test;"

This command will:

• Connect to your database from within the cluster

• Create and drop a test table to verify write permissions

• Exit with an error if the database doesn't exist or the user lacks permissions

If it is successful, you will see:

CREATE TABLE
DROP TABLE

Step 2: Configure your Kubernetes cluster

Configure Kubernetes namespace and secrets

1. Create a Kubernetes namespace. A namespace is optional, but it's recommended to install Cortex in its own namespace for better organization and isolation:

kubectl create namespace cortex

1. Run the following command to add a Docker registry secret for pulling Cortex images from the GitHub Container Registry:

kubectl create secret docker-registry cortex-docker-registry-secret \
  --namespace cortex \
  --docker-server=ghcr.io \
  --docker-username=cortex-image-bot \
  --docker-password=<enter the GitHub PAT provided by Cortex> \
  --docker-email=<[email protected]>

• Replace <enter the GitHub PAT provided by Cortex> with the value of the GitHub PAT that was provided to you by your Cortex account team, and replace <[email protected]> with an email address.

1. Run the following command to create a secret for your Cortex license and your database connection details:

kubectl create secret generic cortex-secret \
    --from-literal DB_HOST=${DB_HOST} \
    --from-literal DB_PORT=${DB_PORT} \
    --from-literal DB_USERNAME=${DB_USERNAME} \
    --from-literal DB_NAME=${DB_NAME} \
    --from-literal DB_PASSWORD=${DB_PASSWORD} \
    --from-literal ENTITLEMENTS_JWT='your_Cortex_license_JWT'

• Substitute the placeholders in this command with the actual values of your port, username, password, database, and Cortex license JWT.

See the Kubernetes documentation for more information on managing secrets using kubectl.

​Step 3: Add the Cortex repo to Helm

Add the Cortex Helm repo

1. In command line, run the following command to add the Cortex repo to Helm:

helm repo add cortex https://helm-charts.cortex.io

1. Install the Helm diff plugin to preview changes before applying them:

helm plugin install https://github.com/databus23/helm-diff

Step 4: Configure and install the helm chart

Chart defaults

Note the following:

• The default values for resource allocation in the chart are provided as guidance, but you may need to adjust these depending on your resource consumption.

• User authentication is disabled by default. After installing Cortex, visiting the frontend URL in a browser will immediately log you in as a demo user. You will then be able to configure authentication using an OIDC provider.

• The chart does not ship with built in TLS or certificates. If you are deploying Cortex into a cluster behind your existing load balancer with TLS, change the protocol value in the chart.

Step 4.1: Configure the helm chart

It's not recommended to modify the Helm chart or values.yaml directly. Instead, we recommend that you create a separate overrides.yaml file for your configuration. This approach ensures:

• Configurations are preserved across upgrades

• Easy version control of your settings

• Clear separation between defaults and customizations

• Timely support from Cortex in the event of an issue

Cortex requires at minimum one frontend pod, one backend pod, and one worker pod. See the example overrides.yaml below demonstrating this.

In this step, you will create and configure the overrides.yaml file. In the next step, you will install the chart and include the overrides.yaml.

See a list of available settings

To see a complete list of available settings, you can review the values.yaml that is bundled in the Cortex Helm chart by running the following command:

helm show values cortex/cortex

Recommended configurations in overrides.yaml

In your overrides.yaml, adjust the replica counts and resource allocations based on your expected load and availability requirements.

At a minimum, the following values should be set:

• image.secrets.name : The name of the Docker registry secret you set in step 2

• app.service.type : Set this field based on your infrastructure. If you're using EKS, AKS, or GKE, set the type as LoadBalancer. If you're using your own Kubernetes distribution and plan to do your own load balancing, configure NodePort.

• app.ingress.type: The type of ingress controller to use with your load balancing setup. For EKS and AKS, use nginx . For GKE, use gcp. If you are using your own Kubernetes distribution, this setting is not used.

• app.secret : The name of the secret you set up in step 2 containing your Cortex license key and database connection details

• app.hostnames.frontend : The DNS name where users will visit the Cortex Web UI in the browser. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the frontend service.

• app.hostnames.backend : The DNS name where the Cortex REST API will be available. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the backend service.

• app.hostnames.protocol : Set to https if you are configuring TLS termination at the load balancer (recommended), or http if you will use Cortex over cleartext. Note that this field does not set up TLS - TLS is configured at the load balancer; this setting is used only for generating URLs.

• app.backend.enabled : Set to true to enable the backend containers to run

• app.worker.enabled : Set to true to enable the worker containers to run

Additional container configuration:

There are additional settings available to configure the backend and worker for your environment. The following paths can be configured under app.backend and app.worker:

• replicaCount (default: 2) This field determines how many containers to create. Depending on your resource constraints, you may need to change this to 1.

• containerConfiguration.resources: Depending on your resource constraints, you may need to adjust requests.memory and limits.memory. By default, each container will consume a minimum of 8GB and a limit of 16GB.

• containerConfiguration.livenessProbe.periodSeconds (default: 10) and containerConfiguration.livenessProbe.timeoutSeconds (default: 6): In environments with slow networking or database, probes may expire before the containers are fully started, causing them to continually be restarted. If this occurs, you may need to raise the values of these settings.

Example overrides.yaml

Below is an example overrides.yaml implementing one backend pod and one worker pod with minimum resources, and configuring TLS termination using network load balancers in AWS:

# Image pull secret configuration
image:
  secrets:
    - name: cortex-docker-registry-secret

app:
  # initial configuration - workspace name and default user email
  initialConfiguration:
    defaultEmail: [email protected]
    workspace: demo

  # Service configuration - how Cortex is exposed
  service:
    type: LoadBalancer  # or NodePort or ClusterIP, depending on your setup
    annotations:
      # Example AWS NLB configuration - adjust for your cloud provider
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
      # TLS termination at load balancer (replace with your certificate ARN)
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:region:account:certificate/cert-id"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
  
  # Ingress configuration
  ingress:
    type: nginx  # or your preferred ingress controller

  # Reference to the Kubernetes secret containing database credentials and license
  secret: cortex-secret

  # Hostnames for accessing Cortex - must match your DNS records
  hostnames:
    frontend: cortex-app.yourdomain.dom  # Web UI endpoint
    backend: cortex-api.yourdomain.dom   # API endpoint
    protocol: https                      # Use https when you have TLS configured

  # Backend API service (required)
  backend:
    enabled: true
    replicaCount: 1  # Increase for high availability
    containerConfiguration:
      resources:
        requests:       # Example: overriding container resource requirements
          cpu: "1"      # Minimum 1 CPU core
          memory: "4Gi" # Minimum 4GB RAM
        limits:         # Example: overriding container resource limits
          cpu: "2"      # Maximum 2 CPU cores
          memory: "8Gi" # Maximum 8GB RAM

  # Worker service (required for background jobs)
  worker:
    enabled: true
    replicaCount: 1  # Increase based on workload
    containerConfiguration:
      resources:
        requests:       # Example: overriding container resource requirements
          cpu: "1"      # Minimum 1 CPU core
          memory: "4Gi" # Minimum 4GB RAM
        limits:         # Example: overriding container resource limits
          cpu: "2"      # Maximum 2 CPU cores
          memory: "8Gi" # Maximum 8GB RAM

  # Frontend service is enabled by default, no configuration needed unless
  # you want to customize replica count or resources

  # Disable built-in PostgreSQL (using external database)
  postgres:
    builtIn:
      enable: false

Step 4.3: Install the helm chart

1. Identify the version of Cortex you want to install.

   • Consider consulting with your Cortex account team to identify the appropriate version. Alternatively, you can find the latest version by running helm search repo cortex/cortex. Cortex version numbers look like 0.0.411.

   • Make a note of the version you are installing, in case you want to change configuration values while remaining on the same version.

2. Before installation, use the following command to preview what will be deployed. Replace DESIRED_VERSION with the version number you identified in the previous step:

helm diff upgrade --install cortex cortex/cortex \
  --namespace cortex \
  --version DESIRED_VERSION \
  -f overrides.yaml

1. Use helm to install Cortex. Replace DESIRED_VERSION with the version number you identified in the first step:

helm install cortex cortex/cortex \
  --namespace cortex \
  --version DESIRED_VERSION \
  -f overrides.yaml

1. Monitor the deployment status:

# Watch all resources in the cortex namespace
kubectl get all -n cortex

# Once backend pods start, tail the logs
kubectl logs -f deployment/cortex-deployment-backend -n cortex

The backend is ready when you see log lines containing:

GET /actuator/health/readiness HTTP/1.1" 200 25

It may take a few minutes, depending on your infrastructure.

Step 5: Configure DNS

After installation, Kubernetes will create two load balancers: one for the frontend and one for the backend. Configure your DNS records:

Configure and verify DNS

1. Get the load balancer endpoints:

   kubectl get service -n cortex

   You'll see two services of type LoadBalancer. Identify which is frontend and which is backend by their names.

2. Update your DNS records:

   • AWS NLB: Create CNAME records pointing to the respective load balancer DNS names.

   • Azure: Create A records pointing to the respective load balancer IPs.

   • Point your frontend hostname (for example, cortex-app.yourdomain.dom) to the frontend load balancer.

   • Point your backend hostname (for example, cortex-api.yourdomain.dom) to the backend load balancer.

3. Verify DNS resolution:

   dig cortex-app.yourdomain.dom
   dig cortex-api.yourdomain.dom

4. Test access to Cortex:

   • Open your frontend URL (for example, https://cortex-app.yourdomain.dom) in your web browser.

     • You should be logged in automatically as "Demo User."

     • After this, you can configure SSO for proper authentication.

Additional configuration for a cloud cluster installation​

If you're installing Cortex into a cloud cluster, for example EKS or GKE, you may need to install required plugins to your cluster. For example, kubectl apply the nginx plugin for AWS.

Backup and recovery

As a best practice, we recommend regularly backing up the following:

• Your Kubernetes secrets

  • To get copies of your Kubernetes secrets as YAML files that can be restored using kubectl apply, use commands like the following:

kubectl get secret cortex-docker-registry-secret -n cortex -o yaml > cortex-docker-registry-secret.yaml
kubectl get secret cortex-secret -n cortex -o yaml > cortex-secret.yaml

• Your overrides.yaml file

  • This contains all the configuration values for your Cortex installation. We recommend keeping this file in a revision control system.

• Your PostgresSQL database

  • This contains all of your Cortex data. We recommend that you implement a regular backup schedule and periodically test restore procedures.

Recovery process

1. Create namespace if it doesn't exist: kubectl create namespace cortex

2. Restore PostgreSQL database from backup

3. Apply Kubernetes secrets to the cortex namespace

4. Install Cortex using your saved overrides.yaml and the appropriate version

Backup and recovery best practices

• Don't fork or modify the Helm chart: If you fork or modify the Helm chart itself, it will be much more difficult to obtain timely support from Cortex. If it seems like the chart needs to be modified, please reach out to your Cortex team to see if there are value overrides available, or if Cortex can modify the official Helm chart to meet your needs.

• Version Control: Keep your overrides.yaml in version control

• Documentation: Document any custom configurations or integrations

• Testing: Always test upgrades in non-production first

• Monitoring: Set up monitoring for Cortex components

• Regular Updates: Stay current with Cortex releases for security and feature updates

• Backup Verification: Regularly test your backup and restore process​

Additional configuration options

Modifying the configuration after installing​

You can make changes to your configuration by editing overrides.yaml and applying them using helm upgrade. First, identify the Cortex version you are running - the Cortex version appears at the bottom of the left-hand nav in the settings page in the Cortex UI. Cortex version numbers look like 0.0.411.

Next, preview the changes:

helm diff upgrade cortex cortex/cortex \
  --namespace cortex \
  --version <your-Cortex-version> \
  -f overrides.yaml

Read the output carefully to be sure that no unexpected changes will be made. Finally, apply the updated configuration using helm upgrade:

helm upgrade cortex cortex/cortex \
  --namespace cortex \
  --version <your-Cortex-version> \
  -f overrides.yaml

Self-Managed SSL Certificates​

This configuration is used when Cortex needs to act as a client connecting to other services in your infrastructure that use self-signed or private CA certificates. Common use cases include:

• Connecting to a self-hosted GitLab instance with custom certificates

• Accessing internal APIs that use private certificate authorities

• Integrating with on-premises services using self-signed certificates

This is not for TLS termination of the Cortex API or UI; that should be handled by your load balancer or ingress controller.

Configuration

Before getting started, note the following prerequisites and considerations:

Certificate prerequisites

Prerequisites:

• Certificates must be in PEM format (Base64 encoded).

  • If you have certificates in other formats, convert them to PEM:

    • DER to PEM: openssl x509 -inform der -in cert.der -out cert.pem

    • P7B to PEM: openssl pkcs7 -print_certs -in cert.p7b -out cert.pem

    • PFX to PEM: openssl pkcs12 -in cert.pfx -out cert.pem -nodes

• Certificates must include the complete certificate chain if they are using intermediate CAs.

• Each certificate should begin with -----BEGIN CERTIFICATE----- and end with -----END CERTIFICATE-----

• Multiple certificates can be concatenated in a single file.

Considerations:

• After following these instructions, the certificate will be added to the Java trust store in both backend and worker pods.

• This configuration is only needed for outbound connections from Cortex

• Inbound TLS/SSL should be configured at the load balancer

Example of creating a Kubernetes secret with concatenated certificates:

kubectl create secret generic tls-custom-certs \
  --namespace cortex \
  --from-literal tls.crt="$(cat cert1.crt cert2.crt cert3.crt)"

Example of PEM formatted certificates:

-----BEGIN CERTIFICATE-----
ABCDEFGCAkWgAwIBAgIJAKl4jvFMb6jYMA0GCSqGSIb3DQEBCwUAMEU123456789
[... base64 encoded certificate data ...]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
ABCDEFGCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG123456789
[... base64 encoded intermediate CA certificate data ...]
-----END CERTIFICATE-----

Follow these steps to add a certificate to the trusted keystores:

1. Create a kubernetes secret in the same namespace as your Cortex helm installation called tls.cert, containing the contents of the certificate file:

kubectl create secret generic tls-secret \
    --namespace cortex \
    --from-literal tls.crt="$(cat my_cert_file.txt)"

1. In your overrides.yaml file, enable selfSignedCerts and ensure the secret name is the same as the one created in the last step. In the example below, the secret is called tls-secret:

app:
  ...
  backend:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret
  worker:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret

1. Repeat the previous step for each service in your overrides.yaml where you want to use the provided certificate.

2. Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.

Troubleshoot certificate issues

Troubleshoot SSL/TLS certificate issues

# Check if certificates are loaded
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  keytool -list -keystore /etc/ssl/certs/java/cacerts -storepass changeit | grep -i "your-cert"

# View certificate details
kubectl get secret tls-custom-certs -n cortex -o jsonpath='{.data.tls\.crt}' | base64 -d

Enabling Redis​

Cortex provides optional Redis support by provisioning Redis in Kubernetes as part of the Helm chart. Enable Redis to gain support for outbound rate limiting of integration API calls. Without enabling Redis, Cortex will still function, but may not have rate limiting capabilities for external API calls.

Redis is enabled via a third-party Helm chart; a full reference to all of the possible settings can be found in this repository on GitHub.

Enable basic Redis

1. Set redis.enabled to true in your overrides.yaml:

redis:
  enabled: true

1. Run helm to apply the changes as outlined in .

Note: It will only be accessible from within the Kubernetes cluster, and authentication will not be enabled by default.

Set authentication for Redis

In production environments, it is recommended that you enable authentication.

Configure the Redis subchart to use your secret

If you enable authentication for Redis:

1. Create a secret for the Redis password:

kubectl create secret generic cortex-helm-redis \
--namespace cortex \
--from-literal redis-password=$(openssl rand -base64 32 | tr '/@:' _)

1. Open your overrides.yaml. Configure the Redis subchart to use that secret:

redis:
  auth:
    enabled: true
    existingSecret: cortex-helm-redis
    existingSecretPasswordKey: redis-password

1. Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.

Embedded Redis upgrades

The vendor Cortex uses to provide an embedded Redis solution, Bitnami, is sunsetting their Helm chart as of August 28, 2025. If you are using embedded Redis (you have redis.enabled = true in your Helm chart values), you must take action to ensure service continuity. Expand the tile below for instructions.

If you are not using Redis, or you are using an externally managed Redis such as AWS Elasticache, no action is required.

Embedded Redis upgrade instructions

If you are using embedded Redis, service interruption may occur if you upgrade after August 28, 2025.

To avoid service interruption, you must follow these instructions when you upgrade:

• If you have set redis.global.imagePullSecrets, ensure that the list of secrets includes the secret that enables you to pull the Cortex images, usually cortex-docker-registry-secret.

  • If you have not set redis.global.imagePullSecrets, no action is required.

• New Redis images are hosted by Cortex in ghcr.io/cortexapps/helm-chart/. If you have set any image URI options, or mirrored the old official images locally, you will need to update your values to use the new images.

  • Image options include:

    • redis.global.imageRegistry

    • redis.image.registry, redis.image.repository, or redis.image.tag

    • redis.metrics.image.registry, redis.metrics.image.repository, or redis.metrics.image.tag

  • If none of the above are set, no action is required.

• Ensure that all of the following settings are false:

  • redis.sentinel.enabled

  • redis.volumePermissions.enabled

  • redis.sysctl.enabled

  • redis.kubectl.enabled

These are all false by default, so if you have not set them, no action is required.

If you have questions about these steps or about the upgrade, please reach out to Cortex support.

Monitor Redis usage

You can monitor Redis usage to ensure it's properly sized:

Monitor Redis

# Connect to Redis pod (if using built-in)
kubectl exec -it service/cortex-redis-master -n cortex -- redis-cli

# Check memory usage
INFO memory

# Check connected clients
INFO clients

# Monitor commands being executed
MONITOR

Enabling metrics​

Prometheus​

Prometheus metrics provide visibility into Cortex's performance and health, enabling:

• Performance monitoring: Track response times, request rates, and error rates

• Resource utilization: Monitor memory usage, CPU consumption, and thread pools

• Integration health: Track success or failure rates for external API calls

• Alerting: Set up alerts for degraded performance or failures

• Capacity planning: Understand usage patterns to plan scaling

Expand the tile below to learn how to publish Prometheus metrics and configure your Prometheus instance to scrape pods directly.

Publish metrics and configure Prometheus

To publish Prometheus metrics:

1. Set app.shared.prometheusMetrics.enabled to true in your overrides.yaml:

app:
  shared:
    prometheusMetrics:
      enabled: true

1. Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.

When enabled, Prometheus metrics will be published on port 8181 at path /manage/prometheus (http://localhost:8181/manage/prometheus) for both backend and worker pods.

Prometheus configuration

To scrape metrics from Cortex, configure your Prometheus instance to scrape pods directly, for example:

# prometheus-config.yaml
scrape_configs:
  - job_name: 'cortex-backend'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - cortex
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: cortex-backend
        action: keep
      - target_label: __address__
        replacement: ${1}:8181
      - target_label: __metrics_path__
        replacement: /manage/prometheus
  
  - job_name: 'cortex-worker'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - cortex
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: cortex-worker
        action: keep
      - target_label: __address__
        replacement: ${1}:8181
      - target_label: __metrics_path__
        replacement: /manage/prometheus

This configuration uses Kubernetes service discovery to find Cortex pods by their labels and scrapes metrics directly from port 8181.

Access Prometheus metrics

Expand the tile below to see some of the available metrics and how to access them.

Available Prometheus metrics

Application Metrics:

• http_server_requests_seconds - HTTP request latencies

• spring_data_repository_invocations_seconds - Database query durations

• comparatron_operation_seconds - External API call durations

• scorecard_latest_cache_access_total - Scorecard cache hit/miss rates

• executor_active_threads - Background job thread activity

• executor_queued_tasks - Queued background tasks

JVM Metrics:

• jvm_memory_used_bytes - Memory usage by area

• jvm_gc_pause_seconds - Garbage collection pause times

• jvm_threads_live_threads - Active thread count

• jvm_classes_loaded_classes - Loaded class count

Database Metrics:

• hikaricp_connections_active - Active database connections

• hikaricp_connections_pending - Pending connection requests

• hikaricp_connections_idle - Idle connections in pool

• hikaricp_connections_timeout_total - Connection timeout count

To check published Prometheus metrics:

# From within a backend pod
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# From within a worker pod
kubectl exec -it deployment/cortex-deployment-worker -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# Using port-forward from your local machine (backend)
kubectl port-forward -n cortex deployment/cortex-deployment-backend 8181:8181
# Then in another terminal: curl localhost:8181/manage/prometheus

# Using port-forward from your local machine (worker)
kubectl port-forward -n cortex deployment/cortex-deployment-worker 8182:8181
# Then in another terminal: curl localhost:8182/manage/prometheus

Add Prometheus annotations

Enable automatic discovery by Prometheus

1. Add annotations in overrides.yaml:

app:
  backend:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8181"
      prometheus.io/path: "/manage/prometheus"
  
  worker:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8181"
      prometheus.io/path: "/manage/prometheus"

1. Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.

Example alerting rules

See an example of Prometheus alerting rules for Cortex:

Prometheus alerting rules

groups:
  - name: cortex_alerts
    rules:
      - alert: CortexHighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{job=~".*cortex.*",status=~"5.."}[5m])) 
          / 
          sum(rate(http_server_requests_seconds_count{job=~".*cortex.*"}[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate in Cortex backend"
          description: "Error rate is {{ $value | humanizePercentage }}"
      
      - alert: CortexHighMemoryUsage
        expr: |
          sum(jvm_memory_used_bytes{job=~".*cortex.*",area="heap"}) 
          / 
          sum(jvm_memory_max_bytes{job=~".*cortex.*",area="heap"}) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High heap memory usage in Cortex"
          description: "Heap usage is {{ $value | humanizePercentage }}"
      
      - alert: CortexDatabaseConnectionPoolExhausted
        expr: hikaricp_connections_pending{job=~".*cortex.*"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Database connection pool has pending requests"
          description: "{{ $value }} threads waiting for database connections"
      
      - alert: CortexHighDatabaseConnectionUsage
        expr: |
          hikaricp_connections_active{job=~".*cortex.*"} 
          / 
          hikaricp_connections_max{job=~".*cortex.*"} > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High database connection usage"
          description: "Using {{ $value | humanizePercentage }} of available connections"
      
      - alert: CortexSlowAPIResponses
        expr: |
          histogram_quantile(0.95, 
            sum(rate(http_server_requests_seconds_bucket{job=~".*cortex.*"}[5m])) 
            by (uri, le)
          ) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow API responses detected"
          description: "95th percentile response time for {{ $labels.uri }} is {{ $value }}s"
      
      - alert: CortexBackgroundJobBacklog
        expr: executor_queued_tasks{job=~".*cortex.*"} > 50
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Background job queue building up"
          description: "{{ $value }} tasks queued in {{ $labels.name }}"

Create a Cortex-specific Grafana dashboard

Grafana dashboard

You can create a Grafana dashboard to display the metrics from Prometheus.

Note: The dashboard below is a basic dashboard to get started. You can expand it by adding more panels for:

• Background job executors (executor_active_threads)

• Scorecard cache performance (scorecard_latest_cache_access_total)

• Integration API calls (comparatron_operation_seconds_count)

• Database query performance (spring_data_repository_invocations_seconds)

Import JSON into Grafana

Save the following as cortex-dashboard.json and import it into Grafana:

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{job=~\".*cortex.*\"}[5m])) by (uri)",
          "refId": "A"
        }
      ],
      "title": "HTTP Request Rate by Endpoint",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "id": 2,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{job=~\".*cortex.*\",status=~\"4..|5..\"}[5m])) by (status)",
          "refId": "A"
        }
      ],
      "title": "HTTP Error Rate (4xx/5xx)",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 1
              },
              {
                "color": "red",
                "value": 5
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "id": 3,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": [
            "lastNotNull"
          ],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "text": {}
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket{job=~\".*cortex.*\"}[5m])) by (uri, le))",
          "refId": "A"
        }
      ],
      "title": "API Response Times (p95)",
      "type": "gauge"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "id": 4,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "hikaricp_connections_active{job=~\".*cortex.*\"}",
          "legendFormat": "Active",
          "refId": "A"
        },
        {
          "expr": "hikaricp_connections_idle{job=~\".*cortex.*\"}",
          "legendFormat": "Idle",
          "refId": "B"
        },
        {
          "expr": "hikaricp_connections_pending{job=~\".*cortex.*\"}",
          "legendFormat": "Pending",
          "refId": "C"
        }
      ],
      "title": "Database Connection Pool",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 1,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 0.7
              },
              {
                "color": "red",
                "value": 0.9
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "id": 5,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": [
            "lastNotNull"
          ],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "text": {}
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(jvm_memory_used_bytes{job=~\".*cortex.*\",area=\"heap\"}) / sum(jvm_memory_max_bytes{job=~\".*cortex.*\",area=\"heap\"})",
          "refId": "A"
        }
      ],
      "title": "JVM Heap Usage",
      "type": "gauge"
    }
  ],
  "refresh": "30s",
  "schemaVersion": 27,
  "style": "dark",
  "tags": ["cortex"],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "Prometheus",
          "value": "Prometheus"
        },
        "hide": 0,
        "includeAll": false,
        "label": "Data Source",
        "multi": false,
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Cortex Monitoring",
  "uid": "cortex-monitoring",
  "version": 0
}

Troubleshoot Prometheus metrics issues

Troubleshooting Prometheus metrics

# Verify metrics are accessible on backend
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# Verify metrics are accessible on worker
kubectl exec -it deployment/cortex-deployment-worker -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

Enabling feature flags

Occasionally, Cortex may ask you to enable a feature flag. To do so, add settings to the app.shared.featureFlags map in your overrides.yaml:

app:
  shared:
    featureFlags:
      example.flag: true

Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.

Getting Help

When working with Cortex Customer Support, you will be asked to collect diagnostic data using a tool we distribute called brain-freeze. To install brain-freeze, follow this link to the latest release, and download the .tar.gz file that matches the operating system and architecture where you are running kubectl. Extract the file on the machine where you are running kubectl, and put the enclosed brain-freeze binary somewhere in your path. Then you can run:

brain-freeze k8s logs --namespace cortex --timeInMinutes 1440
brain-freeze k8s dump --namespace cortex --helm-deployment cortex

This will create a data subdirectory in your current directory, containing the Kubernetes logs from your deployment from the last 24 hours (1440 minutes) and the details of your Kubernetes deployment configuration.

The brain-freeze CLI does not dump sensitive information. The values of secrets are masked.

Product analytics​

Cortex collects basic, anonymized data from self-managed customers. If you would like to opt out, please reach out to your Cortex Customer Success Manager.