# Self-managed Cortex

For organizations with strict security and compliance requirements, Cortex can be deployed on-premises into your own Kubernetes cluster.

The Cortex installation is provided as a Helm chart.

{% hint style="info" %}
Due to the frequency of our releases and rapid changes to our product, we only offer dedicated support for releases up to two months prior to the current release version. For versions older than this we will provide best-effort support. If you are currently using a release older than two months, we strongly recommend updating to the latest release. If you encounter an issue while using a version of Cortex older than two months, you may be asked to upgrade as part of our ongoing support efforts.
{% endhint %}

### Additional self-managed resources <a href="#getting-started-with-a-cortex-self-managed-account" id="getting-started-with-a-cortex-self-managed-account"></a>

After following the instructions on this page to get started with your self-managed Cortex account, see these additional guides to enable other features and integrations:

* **User management**: [Enabling SSO](/self-managed/features/users/sso.md), [connecting users via GitHub OAuth](/self-managed/features/users/github-oauth.md), [configuring email notifications](/self-managed/features/users/notifications.md)
* **Configuring integrations**: [Atlassian Connect app](/self-managed/features/integrations/atlassian-connect.md), [AWS](/self-managed/features/integrations/aws.md), [GitHub app](/self-managed/features/integrations/github.md), [GitHub OAuth](/self-managed/features/users/github-oauth.md), [Microsoft Teams](/self-managed/features/integrations/ms-teams.md), [Slack](/self-managed/features/integrations/slack.md)
* **Upgrades and rollbacks**: [Upgrading or rolling back Self-Managed Cortex](/self-managed/upgrade.md)
* **Enable AI Docs Assistant:** [Access Cortex's public docs from within the app](#enabling-ai-docs-assistant)

## Getting started with a Cortex self-managed account[​](https://docs.cortex.io/docs/self-managed#getting-started-with-a-cortex-self-managed-account) <a href="#getting-started-with-a-cortex-self-managed-account" id="getting-started-with-a-cortex-self-managed-account"></a>

To deploy Cortex on-premises, you will:

* Make sure you have the required environment and infrastructure set up
* Create a PostgreSQL database where Cortex data will be stored
* Create Kubernetes secrets for pulling software images and connecting to the database
* Create a YAML file containing your configuration values for the Cortex Helm chart
* Preview the Kubernetes deployment
* Deploy using Helm
* Configure DNS records to allow users to access Cortex

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

See the prerequisites below:

<details>

<summary><strong>Prerequisites</strong></summary>

Before getting started, you will need:

* A running Kubernetes cluster, configured with permission to make HTTPS requests to the public internet (our Docker images are hosted on GitHub) - you can use AWS/EKS, Azure/AKS, GCP/GKE, or your own locally installed Kubernetes distribution.
* Permission to add services, deployments, secrets and namespaces to the cluster
* [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) installed and connected to your Kubernetes cluster
* [Helm Package Manager](https://helm.sh/) installed
* A PostgresSQL database (15+) that is accessible from your k8s cluster, and has a dedicated database and user for Cortex.
  * Learn more under [Step 2: Set up a persistent database](#database-prerequisites).
* A GitHub personal access token (PAT) and a JWT license key, both provided to you by your Cortex account team

**Required resources in Kubernetes**

Below is a list of the containers that make up a minimum Cortex deployment, with their resource requirements:

* 2 instances of backend container, with 8GB memory per instance (2 cores recommended)
* 1 instance of worker container, with 8GB memory (2 cores recommended)
* 1 instance of frontend container
  * 500MB memory is generally sufficient, but you can adjust based on your available resources.

Ensure that your cluster has enough resources available to run the containers you plan to deploy.

**DNS and TLS requirements**

Before installing Cortex, you need to prepare:

* **DNS Names**: Create two DNS records in your domain:
  * One for the frontend (web UI): e.g., `cortex-app.yourdomain.dom`
  * One for the backend (API): e.g., `cortex-api.yourdomain.dom`
* **TLS Certificate**: Obtain a TLS certificate that covers both DNS names. You can use:
  * A wildcard certificate (e.g., `*.yourdomain.dom`)
  * A certificate with Subject Alternative Names (SANs) for both DNS names
  * For AWS: AWS Certificate Manager (ACM)
  * For Azure: Azure Key Vault certificates
* **DNS Configuration**: After installation, you'll need to point DNS records for the frontend and backend donain names to the two load balancers created by Kubernetes. The exact configuration depends on your environment:
  * For AWS: Point to each NLB's DNS name
  * For Azure: Point to each Load Balancer's public IP
  * For self-managed load balancers: Point to your load balancers' IPs or hostnames

</details>

<details>

<summary>Verify Kubernetes connection</summary>

**Check kubectl installation and cluster access:**

```sh
# Check kubectl version
kubectl version --client

# Verify cluster access
kubectl cluster-info

# List nodes to confirm connectivity
kubectl get nodes
```

**Check Helm installation:**

```sh
# Check Helm version (must be v3.x)
helm version

# List current Helm releases
helm list --all-namespaces
```

</details>

### Step 1: Set up a persistent database <a href="#step-2-set-up-a-persistent-database" id="step-2-set-up-a-persistent-database"></a>

Cortex stores all persistent data in a PostgreSQL database. You will need to set up a database in order to install and use Cortex.[​](https://docs.cortex.io/docs/self-managed#step-2-set-up-a-persistent-database)

<details>

<summary>Step 1.1: <strong>Database prerequisites</strong></summary>

Starting May 5th, 2026, we'll only be supporting versions 18+ for PostgreSQL. This will enable us to ensure we're able to take advantage of the latest features of Postgres. Please let us know if you're on any version that will be unsupported and need guidance for the upgrade. You have several options, depending on your environment:

* **Self-managed PostgreSQL**: Install PostgreSQL on your own infrastructure
* **AWS RDS**: Use Amazon RDS for PostgreSQL if running on AWS
* **Azure Database for PostgreSQL**: Use Azure's managed PostgreSQL service if running on Azure

You will need the following resources:

* 15GB minimum available storage, 40 GB recommended
* Maximum number of connections ≥100. The maximum number of connections should be ≥2x the backend connection pool size, which is 25 per instance, for a total of 50 when two backend containers are running.

</details>

<details>

<summary>Step 1.2: <strong>Create a Postgres database</strong></summary>

1. On your PostgreSQL server, create a database with UTF-8 encoding that will be accessible from your instance.
2. Create a user with the following permissions on the `public.schema` within the database you created:
   * Add `ALL` on the public schema to create/modify tables
   * Add `INSERT`, `SELECT`, `UPDATE`, `DELETE` on all tables in the public schema
3. Make a note of the following database connection details, as you will need them in the next steps:
   * The hostname and port number for connecting to the database server
   * The name of the database you created (this guide assumes the database is named `cortex`)
   * The username and password to access the database

Here is an example of SQL that you might run to create the database and the user with sufficient permissions:

```sql
-- Create the cortex database with UTF-8 encoding
CREATE DATABASE cortex ENCODING 'UTF8';

-- Create the cortex user with a password
CREATE USER cortex_user WITH PASSWORD 'strongpassword';

-- Two options for permissions below: granular or admin
-- Permissions Option A: Granular permissions
\c cortex
GRANT USAGE, CREATE ON SCHEMA public TO cortex_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO cortex_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO cortex_user;

-- OR --
-- Permissions Option B: Database-level admin
-- This gives full control over the cortex database but not others.
GRANT ALL PRIVILEGES ON DATABASE cortex TO cortex_user;
\c cortex
GRANT ALL PRIVILEGES ON SCHEMA public TO cortex_user;

```

</details>

<details>

<summary>Step 1.3: <strong>Verify database connection</strong></summary>

Verify that containers in your Kubernetes cluster will be able to connect to your Postgres database using the connection details you noted above:

```sh
# Set your database connection details
export DB_HOST="<your-postgres-host>"
export DB_PORT="5432"
export DB_NAME="<your-cortex-database>"
export DB_USERNAME="<your-database-user>"
export DB_PASSWORD="<your-database-password>"

# Test database connectivity and permissions
kubectl run pgtools-test --rm -i --restart=Never \
  --namespace cortex \
  --image=postgres:17 \
  --env="PGPASSWORD=${DB_PASSWORD}" \
  --command -- psql \
  -h "${DB_HOST}" \
  -p "${DB_PORT}" \
  -U "${DB_USERNAME}" \
  -d "${DB_NAME}" \
  -c "CREATE TABLE IF NOT EXISTS connection_test (id serial PRIMARY KEY, test_time timestamp DEFAULT NOW()); DROP TABLE connection_test;"
```

This command will:

* Connect to your database from within the cluster
* Create and drop a test table to verify write permissions
* Exit with an error if the database doesn't exist or the user lacks permissions

If it is successful, you will see:

```
CREATE TABLE
DROP TABLE
```

</details>

### Step 2: Configure your Kubernetes cluster <a href="#step-1-configure-your-kubernetes-cluster" id="step-1-configure-your-kubernetes-cluster"></a>

<details>

<summary><strong>Configure Kubernetes namespace and secrets</strong></summary>

1. Create a Kubernetes namespace. A namespace is optional, but it's recommended to install Cortex in its own namespace for better organization and isolation:

```sh
kubectl create namespace cortex
```

2. Run the following command to add a Docker registry secret for pulling Cortex images from the GitHub Container Registry:

```
kubectl create secret docker-registry cortex-docker-registry-secret \
  --namespace cortex \
  --docker-server=ghcr.io \
  --docker-username=cortex-image-bot \
  --docker-password=<enter the GitHub PAT provided by Cortex> \
  --docker-email=<yourname@example.com>
```

* Replace `<enter the GitHub PAT provided by Cortex>` with the value of the GitHub PAT that was provided to you by your Cortex account team, and replace `<yourname@example.com>` with an email address.\\

3. Run the following command to create a secret for your Cortex license and your database connection details:

```sh
kubectl create secret generic cortex-secret \
    --from-literal DB_HOST=${DB_HOST} \
    --from-literal DB_PORT=${DB_PORT} \
    --from-literal DB_USERNAME=${DB_USERNAME} \
    --from-literal DB_NAME=${DB_NAME} \
    --from-literal DB_PASSWORD=${DB_PASSWORD} \
    --from-literal ENTITLEMENTS_JWT='your_Cortex_license_JWT'
```

* Substitute the placeholders in this command with the actual values of your port, username, password, database, and Cortex license JWT.

See the Kubernetes documentation for more information on [managing secrets using `kubectl`](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/).

</details>

### [**​**](https://docs.cortex.io/docs/self-managed#create-a-postgres-database)Step 3: Add the Cortex repo to Helm

<details>

<summary><strong>Add the Cortex Helm repo</strong></summary>

1. In command line, run the following command to add the Cortex repo to Helm:

```sh
helm repo add cortex https://helm-charts.cortex.io
```

2. Install the [Helm diff plugin](https://github.com/databus23/helm-diff/blob/master/README.md) to preview changes before applying them:

```sh
helm plugin install https://github.com/databus23/helm-diff
```

</details>

### Step 4: Configure and install the helm chart

{% hint style="info" %}
**Chart defaults**

Note the following:

* The default values for resource allocation in the chart are provided as guidance, but you may need to adjust these depending on your resource consumption.
* User authentication is disabled by default. After installing Cortex, visiting the frontend URL in a browser will immediately log you in as a demo user. You will then be able to configure authentication using an OIDC provider.
* The chart does not ship with built in TLS or certificates. If you are deploying Cortex into a cluster behind your existing load balancer with TLS, change the `protocol` value in the chart.
  {% endhint %}

<details>

<summary><strong>Step 4.1: Configure the helm chart</strong></summary>

It's not recommended to modify the Helm chart or `values.yaml` directly. Instead, we recommend that you create a separate `overrides.yaml` file for your configuration. This approach ensures:

* Configurations are preserved across upgrades
* Easy version control of your settings
* Clear separation between defaults and customizations
* Timely support from Cortex in the event of an issue

Cortex requires at minimum one frontend pod, one backend pod, and one worker pod. See the example overrides.yaml below demonstrating this.

In this step, you will create and configure the `overrides.yaml` file. In the next step, you will install the chart and include the `overrides.yaml`.

\
**See a list of available settings**

To see a complete list of available settings, you can review the `values.yaml` that is bundled in the Cortex Helm chart by running the following command:

```sh
helm show values cortex/cortex
```

\
**Recommended configurations in overrides.yaml**

In your `overrides.yaml`, adjust the replica counts and resource allocations based on your expected load and availability requirements.

At a minimum, the following values should be set:

* `image.secrets.name` : The name of the Docker registry secret you set in step 2
* `app.service.type` : Set this field based on your infrastructure. If you're using EKS, AKS, or GKE, set the type as `LoadBalancer`. If you're using your own Kubernetes distribution and plan to do your own load balancing, configure `NodePort`.
* `app.ingress.type`: The type of ingress controller to use with your load balancing setup. For EKS and AKS, use `nginx` . For GKE, use `gcp`. If you are using your own Kubernetes distribution, this setting is not used.
* `app.secret` : The name of the secret you set up in step 2 containing your Cortex license key and database connection details
* `app.hostnames.frontend` : The DNS name where users will visit the Cortex Web UI in the browser. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the frontend service.
* `app.hostnames.backend` : The DNS name where the Cortex REST API will be available. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the backend service.
* `app.hostnames.protocol` : Set to `https` if you are configuring TLS termination at the load balancer (recommended), or `http` if you will use Cortex over cleartext. Note that this field does not set up TLS - TLS is configured at the load balancer; this setting is used only for generating URLs.
* `app.backend.enabled` : Set to `true` to enable the backend containers to run
* `app.worker.enabled` : Set to `true` to enable the worker containers to run

\
**Additional container configuration:**

There are additional settings available to configure the backend and worker for your environment. The following paths can be configured under `app.backend` and `app.worker`:

* `replicaCount` (default: 2) This field determines how many containers to create. Depending on your resource constraints, you may need to change this to `1`.
* `containerConfiguration.resources`: Depending on your resource constraints, you may need to adjust `requests.memory` and `limits.memory`. By default, each container will consume a minimum of 8GB and a limit of 16GB.
* `containerConfiguration.livenessProbe.periodSeconds` (default: 10) and `containerConfiguration.livenessProbe.timeoutSeconds` (default: 6): In environments with slow networking or database, probes may expire before the containers are fully started, causing them to continually be restarted. If this occurs, you may need to raise the values of these settings.

**Example overrides.yaml**

Below is an example `overrides.yaml` implementing one backend pod and one worker pod with minimum resources, and configuring TLS termination using network load balancers in AWS:

```yaml
# Image pull secret configuration
image:
  secrets:
    - name: cortex-docker-registry-secret

app:
  # initial configuration - workspace name and default user email
  initialConfiguration:
    defaultEmail: demo@getcortexapp.com
    workspace: demo

  # Service configuration - how Cortex is exposed
  service:
    type: LoadBalancer  # or NodePort or ClusterIP, depending on your setup
    annotations:
      # Example AWS NLB configuration - adjust for your cloud provider
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
      # TLS termination at load balancer (replace with your certificate ARN)
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:region:account:certificate/cert-id"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
  
  # Ingress configuration
  ingress:
    type: nginx  # or your preferred ingress controller

  # Reference to the Kubernetes secret containing database credentials and license
  secret: cortex-secret

  # Hostnames for accessing Cortex - must match your DNS records
  hostnames:
    frontend: cortex-app.yourdomain.dom  # Web UI endpoint
    backend: cortex-api.yourdomain.dom   # API endpoint
    protocol: https                      # Use https when you have TLS configured

  # Backend API service (required)
  backend:
    enabled: true
    replicaCount: 1  # Increase for high availability
    containerConfiguration:
      resources:
        requests:       # Example: overriding container resource requirements
          cpu: "1"      # Minimum 1 CPU core
          memory: "4Gi" # Minimum 4GB RAM
        limits:         # Example: overriding container resource limits
          cpu: "2"      # Maximum 2 CPU cores
          memory: "8Gi" # Maximum 8GB RAM

  # Worker service (required for background jobs)
  worker:
    enabled: true
    replicaCount: 1  # Increase based on workload
    containerConfiguration:
      resources:
        requests:       # Example: overriding container resource requirements
          cpu: "1"      # Minimum 1 CPU core
          memory: "4Gi" # Minimum 4GB RAM
        limits:         # Example: overriding container resource limits
          cpu: "2"      # Maximum 2 CPU cores
          memory: "8Gi" # Maximum 8GB RAM

  # Frontend service is enabled by default, no configuration needed unless
  # you want to customize replica count or resources

  # Disable built-in PostgreSQL (using external database)
  postgres:
    builtIn:
      enable: false
```

</details>

<details>

<summary><strong>Step 4.3: Install the helm chart</strong></summary>

1. Identify the version of Cortex you want to install.
   * Consider consulting with your Cortex account team to identify the appropriate version. Alternatively, you can find the latest version by running `helm search repo cortex/cortex`. Cortex version numbers look like `0.0.411`.
   * Make a note of the version you are installing, in case you want to change configuration values while remaining on the same version.
2. Before installation, use the following command to preview what will be deployed. Replace `DESIRED_VERSION` with the version number you identified in the previous step:

```sh
helm diff upgrade --install cortex cortex/cortex \
  --namespace cortex \
  --version DESIRED_VERSION \
  -f overrides.yaml
```

2. Use helm to install Cortex. Replace `DESIRED_VERSION` with the version number you identified in the first step:

```sh
helm install cortex cortex/cortex \
  --namespace cortex \
  --version DESIRED_VERSION \
  -f overrides.yaml
```

3. Monitor the deployment status:

```sh
# Watch all resources in the cortex namespace
kubectl get all -n cortex

# Once backend pods start, tail the logs
kubectl logs -f deployment/cortex-deployment-backend -n cortex
```

The backend is ready when you see log lines containing:

```
GET /actuator/health/readiness HTTP/1.1" 200 25
```

It may take a few minutes, depending on your infrastructure.

</details>

### Step 5: Configure DNS <a href="#additional-configuration-for-a-cloud-cluster-installation" id="additional-configuration-for-a-cloud-cluster-installation"></a>

After installation, Kubernetes will create two load balancers: one for the frontend and one for the backend. Configure your DNS records:

<details>

<summary><strong>Configure and verify DNS</strong></summary>

1. Get the load balancer endpoints:

   ```sh
   kubectl get service -n cortex
   ```

   You'll see two services of type `LoadBalancer`. Identify which is frontend and which is backend by their names.
2. Update your DNS records:
   * **AWS NLB**: Create CNAME records pointing to the respective load balancer DNS names.
   * **Azure**: Create A records pointing to the respective load balancer IPs.
   * Point your frontend hostname (for example, `cortex-app.yourdomain.dom`) to the frontend load balancer.
   * Point your backend hostname (for example, `cortex-api.yourdomain.dom`) to the backend load balancer.
3. Verify DNS resolution:

   ```
   dig cortex-app.yourdomain.dom
   dig cortex-api.yourdomain.dom
   ```
4. Test access to Cortex:
   * Open your frontend URL (for example, `https://cortex-app.yourdomain.dom`) in your web browser.
     * You should be logged in automatically as "Demo User."
     * After this, you can [configure SSO](/self-managed/features/users/sso.md) for proper authentication.

</details>

### Additional configuration for a cloud cluster installation[​](https://docs.cortex.io/docs/self-managed#additional-configuration-for-a-cloud-cluster-installation) <a href="#additional-configuration-for-a-cloud-cluster-installation" id="additional-configuration-for-a-cloud-cluster-installation"></a>

If you're installing Cortex into a cloud cluster, for example EKS or GKE, you may need to install required plugins to your cluster. For example, `kubectl apply` the [nginx plugin for AWS](https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.34.1/deploy/static/provider/aws/deploy.yaml).

## Backup and recovery

As a best practice, we recommend regularly backing up the following:

* **Your Kubernetes secrets**
  * To get copies of your Kubernetes secrets as YAML files that can be restored using `kubectl apply`, use commands like the following:

```sh
kubectl get secret cortex-docker-registry-secret -n cortex -o yaml > cortex-docker-registry-secret.yaml
kubectl get secret cortex-secret -n cortex -o yaml > cortex-secret.yaml
```

* **Your overrides.yaml file**
  * This contains all the configuration values for your Cortex installation. We recommend keeping this file in a revision control system.
* **Your PostgresSQL database**
  * This contains all of your Cortex data. We recommend that you implement a regular backup schedule and periodically test restore procedures.

### Recovery process

1. Create namespace if it doesn't exist: `kubectl create namespace cortex`
2. Restore PostgreSQL database from backup
3. Apply Kubernetes secrets to the cortex namespace
4. Install Cortex using your saved `overrides.yaml` and the appropriate version

### Backup and recovery best practices

* **Don't fork or modify the Helm chart**: If you fork or modify the Helm chart itself, it will be much more difficult to obtain timely support from Cortex. If it seems like the chart needs to be modified, please reach out to your Cortex team to see if there are value overrides available, or if Cortex can modify the official Helm chart to meet your needs.
* **Version Control**: Keep your `overrides.yaml` in version control
* **Documentation**: Document any custom configurations or integrations
* **Testing**: Always test upgrades in non-production first
* **Monitoring**: Set up monitoring for Cortex components
* **Regular Updates**: Stay current with Cortex releases for security and feature updates
* **Backup Verification**: Regularly test your backup and restore process[​](https://docs.cortex.io/docs/self-managed#additional-configuration-options)

## Additional configuration options <a href="#additional-configuration-options" id="additional-configuration-options"></a>

### Modifying the configuration after installing[​](https://docs.cortex.io/docs/self-managed#modifying-the-configuration-after-installing) <a href="#modifying-the-configuration-after-installing" id="modifying-the-configuration-after-installing"></a>

You can make changes to your configuration by editing `overrides.yaml` and applying them using `helm upgrade`. First, identify the Cortex version you are running - the Cortex version appears at the bottom of the left-hand nav in the settings page in the Cortex UI. Cortex version numbers look like `0.0.411`.

Next, preview the changes:

```sh
helm diff upgrade cortex cortex/cortex \
  --namespace cortex \
  --version <your-Cortex-version> \
  -f overrides.yaml
```

Read the output carefully to be sure that no unexpected changes will be made. Finally, apply the updated configuration using `helm upgrade`:

```sh
helm upgrade cortex cortex/cortex \
  --namespace cortex \
  --version <your-Cortex-version> \
  -f overrides.yaml
```

### Self-Managed SSL Certificates[​](https://docs.cortex.io/docs/self-managed#self-managed-ssl-certificates) <a href="#self-managed-ssl-certificates" id="self-managed-ssl-certificates"></a>

This configuration is used when Cortex needs to act as a client connecting to other services in your infrastructure that use self-signed or private CA certificates. Common use cases include:

* Connecting to a self-hosted GitLab instance with custom certificates
* Accessing internal APIs that use private certificate authorities
* Integrating with on-premises services using self-signed certificates

{% hint style="info" %}
This is not for TLS termination of the Cortex API or UI; that should be handled by your load balancer or ingress controller.
{% endhint %}

#### Configuration

Before getting started, note the following prerequisites and considerations:

<details>

<summary>Certificate prerequisites</summary>

**Prerequisites**:

* Certificates must be in PEM format (Base64 encoded).
  * If you have certificates in other formats, convert them to PEM:
    * DER to PEM: `openssl x509 -inform der -in cert.der -out cert.pem`
    * P7B to PEM: `openssl pkcs7 -print_certs -in cert.p7b -out cert.pem`
    * PFX to PEM: `openssl pkcs12 -in cert.pfx -out cert.pem -nodes`
* Certificates must include the complete certificate chain if they are using intermediate CAs.
* Each certificate should begin with `-----BEGIN CERTIFICATE-----` and end with `-----END CERTIFICATE-----`
* Multiple certificates can be concatenated in a single file.

**Considerations**:

* After following these instructions, the certificate will be added to the Java trust store in both backend and worker pods.
* This configuration is only needed for outbound connections from Cortex
* Be sure the full certificate chain is included, since importing only the leaf certificate can still result in PKIX path building errors when Java services attempt internal HTTPS calls.
* Inbound TLS/SSL should be configured at the load balancer

**Example of creating a Kubernetes secret with concatenated certificates**:

```sh
kubectl create secret generic tls-custom-certs \
  --namespace cortex \
  --from-literal tls.crt="$(cat cert1.crt cert2.crt cert3.crt)"
```

**Example of PEM formatted certificates**:

```
-----BEGIN CERTIFICATE-----
ABCDEFGCAkWgAwIBAgIJAKl4jvFMb6jYMA0GCSqGSIb3DQEBCwUAMEU123456789
[... base64 encoded certificate data ...]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
ABCDEFGCAimgAwIBAgITBmyfz5m/jAo54vB4ikPmljZbyjANBgkqhkiG123456789
[... base64 encoded intermediate CA certificate data ...]
-----END CERTIFICATE-----
```

</details>

Follow these steps to add a certificate to the trusted keystores:

1. Create a kubernetes secret in the same namespace as your Cortex helm installation called `tls.cert`, containing the contents of the certificate file:

```sh
kubectl create secret generic tls-secret \
    --namespace cortex \
    --from-literal tls.crt="$(cat my_cert_file.txt)"
```

2. In your `overrides.yaml` file, enable `selfSignedCerts` and ensure the secret name is the same as the one created in the last step. In the example below, the secret is called `tls-secret`:

```yaml
app:
  ...
  backend:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret
  worker:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret
```

3. Repeat the previous step for each service in your `overrides.yaml` where you want to use the provided certificate.
4. Preview and apply the updated configuration by following the steps in **Modifying the configuration after installing** above.

**Troubleshoot certificate issues**

<details>

<summary>Troubleshoot SSL/TLS certificate issues</summary>

```sh
# Check if certificates are loaded
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  keytool -list -keystore /etc/ssl/certs/java/cacerts -storepass changeit | grep -i "your-cert"

# View certificate details
kubectl get secret tls-custom-certs -n cortex -o jsonpath='{.data.tls\.crt}' | base64 -d
```

</details>

### Using Envoy Gateway (alternative to NGINX Ingress) <a href="#using-envoy-gateway" id="using-envoy-gateway"></a>

As an alternative to the default NGINX Ingress controller, the Cortex Helm chart supports [Envoy Gateway](https://gateway.envoyproxy.io/) using the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/). This is the recommended path forward, as NGINX Ingress is being deprecated.

{% hint style="info" %}
Envoy Gateway requires the Gateway API CRDs to be installed in your cluster. If you haven't already, install them before proceeding:

```sh
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/latest/download/standard-install.yaml
```

You will also need the [Envoy Gateway controller](https://gateway.envoyproxy.io/docs/install/) installed in your cluster.
{% endhint %}

<details>

<summary><strong>Basic Envoy Gateway setup</strong></summary>

To switch from NGINX Ingress to Envoy Gateway, update your `overrides.yaml` to disable Ingress and enable the Gateway API:

```yaml
app:
  # Disable NGINX Ingress
  ingress:
    create: false

  # Enable Gateway API with Envoy Gateway
  gateway:
    enabled: true
    create: true
    className: envoy-gateway

    # TLS configuration (if terminating TLS at the gateway)
    tls:
      enabled: true
      secretName: cortex-tls-secret  # Kubernetes TLS secret with your certificate

    # Envoy proxy service configuration
    envoyProxy:
      create: true
      service:
        type: LoadBalancer  # or NodePort, depending on your setup
        annotations: {}     # Add cloud-provider-specific annotations here
```

This configuration will create:

* A `Gateway` resource using the `envoy-gateway` GatewayClass
* `HTTPRoute` resources for the frontend and backend services
* An `EnvoyProxy` resource that configures the Envoy proxy service

</details>

<details>

<summary><strong>Referencing an existing Gateway</strong></summary>

If you already have a Gateway resource managed outside of the Cortex Helm chart (for example, provisioned by your platform team or by Terraform), you can attach Cortex's HTTPRoutes to it instead of creating a new one:

```yaml
app:
  ingress:
    create: false

  gateway:
    enabled: true
    create: false  # Do not create a Gateway resource

    # Reference the existing Gateway
    ref:
      name: my-existing-gateway
      namespace: gateway-system

    envoyProxy:
      create: false
```

</details>

<details>

<summary><strong>Cloud provider annotations</strong></summary>

You can add annotations to the Envoy proxy service for cloud-provider-specific configuration, such as requesting an internal load balancer:

**AWS (internal NLB):**

```yaml
app:
  gateway:
    envoyProxy:
      create: true
      service:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
          service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
```

**Azure (internal load balancer):**

```yaml
app:
  gateway:
    envoyProxy:
      create: true
      service:
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
```

</details>

After applying your changes, verify the Gateway and HTTPRoutes are created and accepted:

```sh
kubectl get gateway -n cortex
kubectl get httproute -n cortex
```

Both resources should show a status of `Accepted`. If the Gateway shows `Programmed`, the Envoy proxy is ready to receive traffic. Update your DNS records to point to the new load balancer endpoint as described in [Step 5: Configure DNS](#additional-configuration-for-a-cloud-cluster-installation).

#### Troubleshooting Envoy Gateway

If you're experiencing issues with Envoy Gateway, the most useful debugging step is checking the status of the Gateway and HTTPRoute resources. These resources report detailed conditions that indicate where things are getting stuck in state transitions.

<details>

<summary><strong>Check Gateway status</strong></summary>

Inspect the Gateway resource status to verify it has been accepted, programmed, and assigned an address:

```sh
kubectl get gateway -n cortex cortex-helm-gateway -o json | jq '.status'
```

A healthy Gateway will show `Accepted` and `Programmed` conditions with `status: "True"`, and an address assigned:

```json
{
  "addresses": [
    {
      "type": "IPAddress",
      "value": "34.94.230.49"
    }
  ],
  "conditions": [
    {
      "message": "The Gateway has been scheduled by Envoy Gateway",
      "reason": "Accepted",
      "status": "True",
      "type": "Accepted"
    },
    {
      "message": "Address assigned to the Gateway, 1/1 envoy replicas available",
      "reason": "Programmed",
      "status": "True",
      "type": "Programmed"
    }
  ],
  "listeners": [
    {
      "attachedRoutes": 2,
      "conditions": [
        {
          "reason": "Programmed",
          "status": "True",
          "type": "Programmed"
        },
        {
          "reason": "Accepted",
          "status": "True",
          "type": "Accepted"
        },
        {
          "reason": "ResolvedRefs",
          "status": "True",
          "type": "ResolvedRefs"
        }
      ],
      "name": "http"
    }
  ]
}
```

Things to look for:

* **`Accepted` is `False`**: The GatewayClass may not exist, or the Envoy Gateway controller isn't running. Check `kubectl get gatewayclass` and `kubectl get pods -n envoy-gateway-system`.
* **`Programmed` is `False`**: The Envoy proxy deployment may be failing. Check for resource constraints or image pull errors.
* **No address assigned**: The LoadBalancer service may be pending. Check `kubectl get svc -n envoy-gateway-system`.
* **`attachedRoutes` is 0**: No HTTPRoutes have successfully attached to this listener.

</details>

<details>

<summary><strong>Check HTTPRoute status</strong></summary>

Inspect the HTTPRoute status to verify routes are accepted and their backend references are resolved:

```sh
kubectl get httproute -n cortex cortex-helm-backend-http-route -o json | jq '.status'
```

A healthy HTTPRoute will show `Accepted` and `ResolvedRefs` conditions with `status: "True"`:

```json
{
  "parents": [
    {
      "conditions": [
        {
          "message": "Route is accepted",
          "reason": "Accepted",
          "status": "True",
          "type": "Accepted"
        },
        {
          "message": "Resolved all the Object references for the Route",
          "reason": "ResolvedRefs",
          "status": "True",
          "type": "ResolvedRefs"
        }
      ],
      "controllerName": "gateway.envoyproxy.io/gatewayclass-controller",
      "parentRef": {
        "group": "gateway.networking.k8s.io",
        "kind": "Gateway",
        "name": "cortex-helm-gateway",
        "namespace": "cortex"
      }
    }
  ]
}
```

Things to look for:

* **`Accepted` is `False`**: The route's `parentRef` may not match an existing Gateway, or the Gateway's `allowedRoutes` may restrict the namespace.
* **`ResolvedRefs` is `False`**: A backend service referenced in the route doesn't exist or has the wrong port. Verify with `kubectl get svc -n cortex`.
* **No `parents` entries**: The route hasn't been picked up by any controller. Check that the Envoy Gateway controller is running.

</details>

### Enabling Redis[​](https://docs.cortex.io/docs/self-managed#enabling-redis) <a href="#enabling-redis" id="enabling-redis"></a>

Cortex provides optional Redis support by provisioning Redis in Kubernetes as part of the Helm chart. Enable Redis to gain support for outbound rate limiting of integration API calls. Without enabling Redis, Cortex will still function, but may not have rate limiting capabilities for external API calls.

Redis is enabled via a third-party Helm chart; a full reference to all of the possible settings can be found in [this repository on GitHub](https://github.com/bitnami/charts/blob/redis/20.6.0/bitnami/redis/values.yaml).

<details>

<summary><strong>Enable basic Redis</strong></summary>

1. Set `redis.enabled` to `true` in your `overrides.yaml`:

```yaml
redis:
  enabled: true
```

2. Run helm to apply the changes as outlined in .

Note: It will only be accessible from within the Kubernetes cluster, and authentication will not be enabled by default.

</details>

#### Set authentication for Redis

In production environments, it is recommended that you enable authentication.

<details>

<summary><strong>Configure the Redis subchart to use your secret</strong></summary>

If you enable authentication for Redis:

1. Create a secret for the Redis password:

```sh
kubectl create secret generic cortex-helm-redis \
--namespace cortex \
--from-literal redis-password=$(openssl rand -base64 32 | tr '/@:' _)
```

2. Open your `overrides.yaml`. Configure the Redis subchart to use that secret:

```yaml
redis:
  auth:
    enabled: true
    existingSecret: cortex-helm-redis
    existingSecretPasswordKey: redis-password
```

3. Preview and apply the updated configuration by following the steps in **Modifying the configuration after installing** above.

</details>

#### Embedded Redis upgrades

{% hint style="warning" %}
The vendor Cortex uses to provide an embedded Redis solution, Bitnami, is [sunsetting their Helm chart](https://github.com/bitnami/charts/issues/35164) as of August 28, 2025. If you are using embedded Redis (you have `redis.enabled = true` in your Helm chart values), you must take action to ensure service continuity. Expand the tile below for instructions.

If you are not using Redis, or you are using an externally managed Redis such as AWS Elasticache, no action is required.
{% endhint %}

<details>

<summary>Embedded Redis upgrade instructions</summary>

If you are using embedded Redis, service interruption may occur if you upgrade after August 28, 2025.

To avoid service interruption, you must follow these instructions when you upgrade:

* If you have set `redis.global.imagePullSecrets`, ensure that the list of secrets includes the secret that enables you to pull the Cortex images, usually `cortex-docker-registry-secret`.
  * If you have not set `redis.global.imagePullSecrets`, no action is required.
* New Redis images are hosted by Cortex in ghcr.io/cortexapps/helm-chart/. If you have set any image URI options, or mirrored the old official images locally, you will need to update your values to use the new images.
  * Image options include:
    * `redis.global.imageRegistry`
    * `redis.image.registry`, `redis.image.repository`, or `redis.image.tag`
    * `redis.metrics.image.registry`, `redis.metrics.image.repository`, or `redis.metrics.image.tag`
  * If none of the above are set, no action is required.
* Ensure that all of the following settings are false:
  * `redis.sentinel.enabled`
  * `redis.volumePermissions.enabled`
  * `redis.sysctl.enabled`
  * `redis.kubectl.enabled`

These are all false by default, so if you have not set them, no action is required.

If you have questions about these steps or about the upgrade, please reach out to Cortex support.

</details>

#### Monitor Redis usage

You can monitor Redis usage to ensure it's properly sized:

<details>

<summary><strong>Monitor Redis</strong></summary>

```sh
# Connect to Redis pod (if using built-in)
kubectl exec -it service/cortex-redis-master -n cortex -- redis-cli

# Check memory usage
INFO memory

# Check connected clients
INFO clients

# Monitor commands being executed
MONITOR
```

</details>

### Enabling metrics[​](https://docs.cortex.io/docs/self-managed#enabling-metrics)

#### **Prometheus**[**​**](https://docs.cortex.io/docs/self-managed#prometheus)

Prometheus metrics provide visibility into Cortex's performance and health, enabling:

* **Performance monitoring**: Track response times, request rates, and error rates
* **Resource utilization**: Monitor memory usage, CPU consumption, and thread pools
* **Integration health**: Track success or failure rates for external API calls
* **Alerting**: Set up alerts for degraded performance or failures
* **Capacity planning**: Understand usage patterns to plan scaling

Expand the tile below to learn how to publish Prometheus metrics and configure your Prometheus instance to scrape pods directly.

<details>

<summary>Publish metrics and configure Prometheus</summary>

**To publish Prometheus metrics**:

1. Set `app.shared.prometheusMetrics.enabled` to `true` in your `overrides.yaml`:

```yaml
app:
  shared:
    prometheusMetrics:
      enabled: true
```

2. Preview and apply the updated configuration by following the steps in **Modifying the configuration after installing** above.

When enabled, Prometheus metrics will be published on port 8181 at path `/manage/prometheus` (`http://localhost:8181/manage/prometheus`) for both backend and worker pods.

**Prometheus configuration**

To scrape metrics from Cortex, configure your Prometheus instance to scrape pods directly, for example:

```yaml
# prometheus-config.yaml
scrape_configs:
  - job_name: 'cortex-backend'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - cortex
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: cortex-backend
        action: keep
      - target_label: __address__
        replacement: ${1}:8181
      - target_label: __metrics_path__
        replacement: /manage/prometheus
  
  - job_name: 'cortex-worker'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - cortex
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        regex: cortex-worker
        action: keep
      - target_label: __address__
        replacement: ${1}:8181
      - target_label: __metrics_path__
        replacement: /manage/prometheus
```

This configuration uses Kubernetes service discovery to find Cortex pods by their labels and scrapes metrics directly from port 8181.

</details>

**Access Prometheus metrics**

Expand the tile below to see some of the available metrics and how to access them.

<details>

<summary>Available Prometheus metrics</summary>

**Application Metrics:**

* `http_server_requests_seconds` - HTTP request latencies
* `spring_data_repository_invocations_seconds` - Database query durations
* `comparatron_operation_seconds` - External API call durations
* `scorecard_latest_cache_access_total` - Scorecard cache hit/miss rates
* `executor_active_threads` - Background job thread activity
* `executor_queued_tasks` - Queued background tasks

**JVM Metrics:**

* `jvm_memory_used_bytes` - Memory usage by area
* `jvm_gc_pause_seconds` - Garbage collection pause times
* `jvm_threads_live_threads` - Active thread count
* `jvm_classes_loaded_classes` - Loaded class count

**Database Metrics:**

* `hikaricp_connections_active` - Active database connections
* `hikaricp_connections_pending` - Pending connection requests
* `hikaricp_connections_idle` - Idle connections in pool
* `hikaricp_connections_timeout_total` - Connection timeout count

To check published Prometheus metrics:

```sh
# From within a backend pod
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# From within a worker pod
kubectl exec -it deployment/cortex-deployment-worker -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# Using port-forward from your local machine (backend)
kubectl port-forward -n cortex deployment/cortex-deployment-backend 8181:8181
# Then in another terminal: curl localhost:8181/manage/prometheus

# Using port-forward from your local machine (worker)
kubectl port-forward -n cortex deployment/cortex-deployment-worker 8182:8181
# Then in another terminal: curl localhost:8182/manage/prometheus
```

</details>

**Add Prometheus annotations**

<details>

<summary>Enable automatic discovery by Prometheus</summary>

1. Add annotations in `overrides.yaml`:

```yaml
app:
  backend:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8181"
      prometheus.io/path: "/manage/prometheus"
  
  worker:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "8181"
      prometheus.io/path: "/manage/prometheus"
```

2. Preview and apply the updated configuration by following the steps in **Modifying the configuration after installing** above.

</details>

**Example alerting rules**

See an example of Prometheus alerting rules for Cortex:

<details>

<summary>Prometheus alerting rules</summary>

```yaml
groups:
  - name: cortex_alerts
    rules:
      - alert: CortexHighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{job=~".*cortex.*",status=~"5.."}[5m])) 
          / 
          sum(rate(http_server_requests_seconds_count{job=~".*cortex.*"}[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate in Cortex backend"
          description: "Error rate is {{ $value | humanizePercentage }}"
      
      - alert: CortexHighMemoryUsage
        expr: |
          sum(jvm_memory_used_bytes{job=~".*cortex.*",area="heap"}) 
          / 
          sum(jvm_memory_max_bytes{job=~".*cortex.*",area="heap"}) > 0.9
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High heap memory usage in Cortex"
          description: "Heap usage is {{ $value | humanizePercentage }}"
      
      - alert: CortexDatabaseConnectionPoolExhausted
        expr: hikaricp_connections_pending{job=~".*cortex.*"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Database connection pool has pending requests"
          description: "{{ $value }} threads waiting for database connections"
      
      - alert: CortexHighDatabaseConnectionUsage
        expr: |
          hikaricp_connections_active{job=~".*cortex.*"} 
          / 
          hikaricp_connections_max{job=~".*cortex.*"} > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High database connection usage"
          description: "Using {{ $value | humanizePercentage }} of available connections"
      
      - alert: CortexSlowAPIResponses
        expr: |
          histogram_quantile(0.95, 
            sum(rate(http_server_requests_seconds_bucket{job=~".*cortex.*"}[5m])) 
            by (uri, le)
          ) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Slow API responses detected"
          description: "95th percentile response time for {{ $labels.uri }} is {{ $value }}s"
      
      - alert: CortexBackgroundJobBacklog
        expr: executor_queued_tasks{job=~".*cortex.*"} > 50
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Background job queue building up"
          description: "{{ $value }} tasks queued in {{ $labels.name }}"
```

</details>

**Create a Cortex-specific Grafana dashboard**

<details>

<summary>Grafana dashboard</summary>

You can create a Grafana dashboard to display the metrics from Prometheus.

Note: The dashboard below is a basic dashboard to get started. You can expand it by adding more panels for:

* Background job executors (`executor_active_threads`)
* Scorecard cache performance (`scorecard_latest_cache_access_total`)
* Integration API calls (`comparatron_operation_seconds_count`)
* Database query performance (`spring_data_repository_invocations_seconds`)

**Import JSON into Grafana**

Save the following as `cortex-dashboard.json` and import it into Grafana:

```json
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "panels": [
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{job=~\".*cortex.*\"}[5m])) by (uri)",
          "refId": "A"
        }
      ],
      "title": "HTTP Request Rate by Endpoint",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "reqps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "id": 2,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(rate(http_server_requests_seconds_count{job=~\".*cortex.*\",status=~\"4..|5..\"}[5m])) by (status)",
          "refId": "A"
        }
      ],
      "title": "HTTP Error Rate (4xx/5xx)",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 1
              },
              {
                "color": "red",
                "value": 5
              }
            ]
          },
          "unit": "s"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 8
      },
      "id": 3,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": [
            "lastNotNull"
          ],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "text": {}
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket{job=~\".*cortex.*\"}[5m])) by (uri, le))",
          "refId": "A"
        }
      ],
      "title": "API Response Times (p95)",
      "type": "gauge"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {
              "tooltip": false,
              "viz": false,
              "legend": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "never",
            "spanNulls": true
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 8
      },
      "id": 4,
      "options": {
        "tooltip": {
          "mode": "single"
        },
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom"
        }
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "hikaricp_connections_active{job=~\".*cortex.*\"}",
          "legendFormat": "Active",
          "refId": "A"
        },
        {
          "expr": "hikaricp_connections_idle{job=~\".*cortex.*\"}",
          "legendFormat": "Idle",
          "refId": "B"
        },
        {
          "expr": "hikaricp_connections_pending{job=~\".*cortex.*\"}",
          "legendFormat": "Pending",
          "refId": "C"
        }
      ],
      "title": "Database Connection Pool",
      "type": "timeseries"
    },
    {
      "datasource": "${datasource}",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 1,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "yellow",
                "value": 0.7
              },
              {
                "color": "red",
                "value": 0.9
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 16
      },
      "id": 5,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": [
            "lastNotNull"
          ],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
        "text": {}
      },
      "pluginVersion": "8.0.0",
      "targets": [
        {
          "expr": "sum(jvm_memory_used_bytes{job=~\".*cortex.*\",area=\"heap\"}) / sum(jvm_memory_max_bytes{job=~\".*cortex.*\",area=\"heap\"})",
          "refId": "A"
        }
      ],
      "title": "JVM Heap Usage",
      "type": "gauge"
    }
  ],
  "refresh": "30s",
  "schemaVersion": 27,
  "style": "dark",
  "tags": ["cortex"],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "Prometheus",
          "value": "Prometheus"
        },
        "hide": 0,
        "includeAll": false,
        "label": "Data Source",
        "multi": false,
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Cortex Monitoring",
  "uid": "cortex-monitoring",
  "version": 0
}
```

</details>

**Troubleshoot Prometheus metrics issues**

<details>

<summary>Troubleshooting Prometheus metrics</summary>

```sh
# Verify metrics are accessible on backend
kubectl exec -it deployment/cortex-deployment-backend -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20

# Verify metrics are accessible on worker
kubectl exec -it deployment/cortex-deployment-worker -n cortex -- \
  curl -s http://localhost:8181/manage/prometheus | head -20
```

</details>

### Enabling feature flags

Occasionally, Cortex may ask you to enable a feature flag. To do so, add settings to the `app.shared.featureFlags` map in your `overrides.yaml`:

```yaml
app:
  shared:
    featureFlags:
      example.flag: true
```

Preview and apply the updated configuration by following the steps in **Modifying the configuration after installing** above.

### Enabling AI Docs Assistant

1. Reach out to Cortex requesting Docs Assistant enablement. We willy supply you with a unique integration ID.
2. Once obtained, add `REACT_APP_DOCS_AI_INTEGRATION_ID` to the window variables configuration in your Cortex helm chart's `values.yaml` file

```
app:
	frontend:
		windowVars:
			REACT_APP_DOCS_AI_INTEGRATION_ID: "<integration ID copied from step 2d>"
```

By enabling the AI Docs Assistant feature, you must agree to the following:

1. Allowing Cortex to collect analytics through the AI Docs Assistant chat. This includes tracking questions that were asked in the chat interface and by who. Feedback (downvoting, upvoting, and comments) is also collected through the chat interface.
2. Allowing reCAPTCHA usage. The Docs Assistant feature requires bot protection.

## Getting Help

When working with [Cortex Customer Support](https://support.getcortexapp.com/hc), you will be asked to collect diagnostic data using a tool we distribute called `brain-freeze`. To install `brain-freeze`, follow [this link](https://github.com/cortexapps/brain-freeze/releases/tag/latest) to the latest release, and download the .tar.gz file that matches the operating system and architecture where you are running kubectl. Extract the file on the machine where you are running kubectl, and put the enclosed brain-freeze binary somewhere in your path. Then you can run:

```
brain-freeze k8s logs --namespace cortex --timeInMinutes 1440
brain-freeze k8s dump --namespace cortex --helm-deployment cortex
```

This will create a `data` subdirectory in your current directory, containing the Kubernetes logs from your deployment from the last 24 hours (1440 minutes) and the details of your Kubernetes deployment configuration.

The brain-freeze CLI does not dump sensitive information. The values of secrets are masked.

## Product analytics[​](https://docs.cortex.io/docs/self-managed#product-analytics) <a href="#product-analytics" id="product-analytics"></a>

Cortex collects basic, anonymized data from self-managed customers. If you would like to opt out, please reach out to your Cortex Customer Success Manager.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cortex.io/self-managed.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
