Skip to main content

Cortex Self-Managed

For organizations with strict security and compliance requirements, Cortex can be deployed on-premises into your own Kubernetes (k8s) cluster.

The Cortex installation is provided as a Helm chart.

Getting started with a Cortex self-managed account​

To deploy Cortex on-premises, you will download and extract the Cortex Helm chart, edit its configuration, then install it from the local copy.

Prerequisites​

Before getting started, you will need:

  • A running k8s cluster, configured with permission to talk to the public internet (our Docker images are hosted on GitHub)
  • Permission to add image pull secrets to the cluster
  • kubectl installed and connected to your k8s cluster
  • Helm Package Manager installed
  • A GitHub personal access token (PAT) and a JWT license key, both provided to you by the Cortex team

Step 1: Configure your Kubernetes cluster​

  1. Run the following command to configure your cluster to talk to Cortex:
Create a Docker image pull secret
kubectl create secret docker-registry cortex-docker-registry-secret \
  --docker-server=ghcr.io \
  --docker-username=cortex-image-bot \
  --docker-password=<enter the GitHub PAT provided by Cortex> \
  --docker-email=<yourname@example.com>
  • Make sure to replace <enter the GitHub PAT provided by Cortex> with the value of your PAT, and replace <yourname@example.com> with an email address.
  1. Run the following command to create a secret for your Cortex license:
Create a secret for your Cortex License
kubectl create secret generic cortex-secret \
   --from-literal ENTITLEMENTS_JWT=your_JWT
  • Make sure to replace your_JWT with the value of your JWT license key.

Step 2: Set up a persistent database​

Cortex requires a Postgres Database (version 15), and does not support other datastores. We recommend using your standard database provider, for example a hosted RDS or Cloud SQL instance.

Prerequisites​

You will need the following resources:

  • 2 instances of backend container, with 8GB memory per instance (2 cores recommended)
  • 1 instance of frontend container
    • < 500MB memory is generally sufficient, but you can adjust based on your available resources.
  • Postgres DB v15 with 15GB storage and 4GB memory, with a maximum number of connections ≥100.
    • The maximum number of connections should be ≥2x the backend's connection pool size, which is 50 by default (25 per instance).

Create a Postgres Database​

Note: You must create (or utilize an existing) an instance of Postgres and the database within.

  1. Create a Postgres Database (version 15) with UTF8 encoding that will be accessible from your instance. Create the following and note their values, as you will need them in the next steps:
    • The database instance and database with: instance URI, database name, and instance port
    • The username and password to access the database
  2. Set up database user permissions on the default public.schema:
    • Add ALL on the public schema to create/modify tables
    • Add INSERT, SELECT, UPDATE, DELETE on all tables in the public schema

Step 3: Fetch the helm chart​

  • In command line, run the following commands to download and unzip the latest version of the Cortex chart into a cortex directory:
helm repo add cortex https://helm-charts.cortex.io
helm pull cortex/cortex

tar -xvzf cortex-*.tgz

Step 4: Configure and install the helm chart​

Chart defaults

Note the following:

  • The default values for resource allocation in the chart are provided as guidance, but you may need to adjust these depending on your resource consumption.
  • Auth is disabled by default. Any users who want to access and test Cortex will have access to the dashboard through a “demo” user.
  • The chart does not ship with built in TLS or certificates. If deployed into a cluster behind your existing load balancer with TLS, change the protocol value in the chart.

Step 4.1: Create a k8s secret for your database and license​

  1. Run the following command to create a k8s secret that will hold your database and license credentials:

    kubectl create secret generic cortex-secret \
        --from-literal DB_HOST="hostname (without jdbc:postgresql:// or the port)" \
        --from-literal DB_PORT=5432 \
        --from-literal DB_USERNAME=postgres \
        --from-literal DB_NAME=postgres \
        --from-literal DB_PASSWORD=mysecretpassword \
        --from-literal ENTITLEMENTS_JWT='your_JWT_value'
    • Make sure to substitute the placeholders in this command with the actual values of your port, username, password, database, and JWT.
    • See the Kubernetes documentation for more information on managing secrets using kubectl.

  2. Continue to the next steps to configure the chart.

Step 4.2: Configure the helm chart​

  1. In command line, cd to the cortex directory, where you downloaded the helm chart in the previous steps.
  2. Using a text editor such as vi, open the values.yaml file.
  3. You may need to make modifications, such as the following:
    • app.service.type: Configure this field based on your infrastructure. For example, if you're using EKS, AKS, or GKE, configure the type as LoadBalancer. If you're configuring this locally on your system, configure NodePort.
    • app.ingress: Configure these fields if you are using a load balancer.
    • Under the Hosted DATABASE section:
      • Locate the line # secret: cortex-secret. Uncomment this line by deleting the #.
        • Make sure to remove the additional space before secret to maintain the correct YAML formatting.
          Locate the line containing the cortex secret and uncomment it
      • app.hostnames: Configure the frontend and backend hostnames that the servers will be expecting to be reached on.
        • In a load balancer setup, the frontend hostname should be pointed at the external IP of the load balancer.
      • app.protocol: If your cloud provider or load balancer automatically uses TLS or SSL, change app.protocol to https.
        • Note that this field does not set up TLS; it is used for URL generation.
    • Under the Cortex API (Backend) Configuration section:
      • backend.replicaCount: This field determines how many containers to create. Depending on your resource constraints, you may need to change this to 1.
      • backend.containerConfiguration.resources: Depending on your resource constraints, you may need to adjust requests.memory and limits.memory. By default, each container will consume a minimum of 8GB and a limit of 16GB.
      • backend.containerConfiguration.livenessProbe.periodSeconds and backend.containerConfiguration.livenessProbe.timeoutSeconds: You may need to extend these settings if your containers aren't starting after you install the helm chart.
    • For any containers that you do not require for your use case, you can disabled them by setting their enabled field to false.
tip

The README contains tips on configuring Cortex when running locally in a Minikube instance.

Step 4.3: Install the helm chart​

  1. To install the helm chart, run helm install cortex . at the root of the chart.
  2. Run kubectl get all to see the status of the containers.
    • If the backend container makes multiple attempts to start and continues to fail, you may need to extend the backend timeout settings in values.yaml.
    • The containers may also fail to start if there are issues with outbound HTTP access or your GitHub PAT.

Additional configuration for a cloud cluster installation​

If you're installing Cortex into a cloud cluster, for example EKS or GKE, you may need to install required plugins to your cluster. For example, kubectl apply the nginx plugin for AWS

Additional configuration options​

Modifying the configuration after installing​

After making additional modifications to values.yaml, apply the changes by running helm upgrade.

Self-Managed SSL Certificates​

In some cases, you may want to enable your Cortex deployment to access managed services within your infrastructure (such as a self-hosted GitLab instance) that have their own SSL Certificates. Follow these steps to add a certificate to the trusted keystores:

  1. Create a kubernetes secret in the same namespace as your Cortex helm installation called tls.cert, containing the contents of the certificate file:
kubectl create secret generic tls-secret 
    --from-literal tls.crt="Replace with contents of cert file"
  1. Open your values.yaml file. Enable selfSignedCerts and ensure the secret name is the same as the one created in the last step. In the example below, the secret is called tls-secret:
app:
  ...
  backend:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret
  worker:
    enabled: true
    ...
    selfSignedCerts:
      enable: true
      secret: tls-secret
  1. Repeat the previous step for each service in your values.yaml where you want to use the provided certificate.
  2. Run helm upgrade.

Enabling Redis​

Cortex provides optional Redis support by provisioning Redis in Kubernetes as part of the Helm chart. Enable Redis to gain support for outbound rate limiting of integration API calls.

Redis is enabled via a third-party Helm chart; a full reference to all of the possible settings can be found in this repository on GitHub.

To enable Redis, set redis.enabled to true in your values.yaml:

redis:
  enabled: true

It will only be accessible from within the Kubernetes cluster, and authentication will not be enabled by default.

Set authentication for Redis​

If you enable auth for Redis:

  1. Create a secret:
kubectl create secret generic cortex-helm-redis \
--from-literal redis-password=$(openssl rand -base64 12 | tr '/@:' _)
  1. Configure the Redis subchart to use that secret:
redis:
  auth:
    enabled: true
    existingSecret: cortex-helm-redis
    existingSecretPasswordKey: redis-password

Enabling metrics​

Prometheus​

Prometheus metrics can be published from the backend and worker pods by setting app.shared.prometheusMetrics.enabled to true in your values.yaml:

app:
  shared:
    prometheusMetrics:
      enabled: true

With this enabled, Prometheus metrics will be published on port 8181 at the /actuator/prometheus endpoint.

Debugging​

Cortex provides a CLI to debug the on-prem installation. All commands create a data folder that has the data needed for the Cortex team to debug the issue.

Helm / K8s dump​

To dump all the information about the installation, run the following command:

brain-freeze k8s dump [options]

Note: The CLI only dumps non-sensitive information. For the secrets, the values are masked.

K8s logs​

To fetch the logs for all Cortex related deployments, run the following command:

brain-freeze k8s logs [options]

Cortex-specific info​

After a successful installation, the Cortex team might ask for settings or configurations on your setup. To get this information, run the following command:

/brain-freeze backend info --token="[CORTEX PAT]" --url="[BACKEND HOST]"

You can create a Cortex PAT in your Cortex workspace under Settings > Personal access tokens.