Cortex Self-Managed

Cortex can be deployed on-prem into your own Kubernetes cluster, if you're dealing with sensitive data and don't want information to leave your premises.

We provide the Cortex installation as a Helm chart, making it easy to get started and manage.

Requirements

You'll need a few things to get started with Cortex Self-Managed:

1. A running k8s cluster with permission to talk to the public internet (our Docker images are hosted on GitHub)
2. Permission to add image pull secrets to the cluster
3. Helm Package Manager ready to use
4. A GitHub PAT to access our docker images. If you haven't been granted access, reach out to our team for a demo of the platform.
5. A JWT license key provided by you Cortex account team
6. Our public chart itself

Fetching the chart

Our chart is hosted on a standard helm repository, so you can fetch it with the following command:

helm repo add cortex https://helm-charts.cortex.io
helm pull cortex/cortex

tar -xvzf cortex-*.tgz

This will download and unzip the latest version of the Cortex chart into your current directory.

Installing Cortex

Basics

You'll need to configure your cluster to talk to Cortex.

Create a Docker image pull secret

kubectl create secret docker-registry cortex-docker-registry-secret \
  --docker-server=ghcr.io \
  --docker-username=gsdatta \
  --docker-password=<Token provided to you by the Cortex team> \
  --docker-email=<doesn't matter>

Create a secret for your Cortex License

kubectl create secret generic cortex-secret \
   --from-literal ENTITLEMENTS_JWT='your_JWT_value'

Once you've created the secret, cd into the helm chart, and run helm install with your desired configuration (and uncomment secret: cortex-secret). For example helm install cortex .

tip

The README contains some additional tips on configuring Cortex when running locally in a Minikube cluster.

Cluster installation

If you're installing Cortex into a cloud cluster, for example EKS/GKE, you'll need some additional configuration.

1. Ensure that your values.yaml file in the Helm chart has been modified as desired.
2. Install any required plugins to your cluster, e.g. kubectl apply the nginx plugin for AWS
3. Make sure the Docker secret in the previous section has been added to the cluster
4. Create env var ENTITLEMENTS_JWT='your_JWT_value'
5. Install the Helm chart normally

Once you've run the Helm install, you'll want to get the hostnames wired up. Get the accessible DNS hostnames for the backend and frontend services that the Helm chart exposes (instructions differ based on your cloud provider).

Modify the app.host field in your values.yaml file to point to these respective hostnames. If your cloud provider/LB/ingress automatically sets up TLS, make sure to change app.protocol to https.

Run a helm upgrade to apply the changes.

Chart defaults

There are a few things to note when setting up the chart!

1. The Helm chart spins up an ephemeral datastore with no persistent volumes, see Setting up a persistent DB
2. Auth disabled by default. This means anyone who wants to play with Cortex will have access to the dashboard through a “demo” user.
3. The chart does not ship with built in TLS/certs, but if deployed into a cluster behind your existing load balancer with TLS, you just need to change the protocol value in the chart.

Productionalization

Once you have Cortex running, you may want to productionalize it by setting up a persistent DB or SSO.

Persistent database & license

Cortex requires a Postgres Database (version 15), and does not support other datastores. We recommend using your standard DB provider, for example a hosted RDS or Cloud SQL instance.

Prerequisite Setup

Note You must create (or utilize an existing) instance of Postgres and the database within!

1. Create a Postgres Database (version 15) with UTF8 encoding that will be accessible from your instance. Be sure to create and note:
   1. The database instance and database with: instance URI, database name, and instance port
   2. The username and password to access the database
   3. Set up database user permissions
      1. Add ALL on the public schema to create/modify tables
      2. Add INSERT, SELECT, UPDATE, DELETE on all tables in the public schema
      3. This is tied to the default public.schema

Using Kubectl

1. Create a k8s secret which will hold your DB & license credentials (if you created the cortex-secret above, run kubectl delete secret to delete)

   kubectl create secret generic cortex-secret \
   --from-literal DB_HOST="hostname (without jdbc:postgresql:// or the port)" \
       --from-literal DB_PORT=5432 \
       --from-literal DB_USERNAME=postgres \
       --from-literal DB_NAME=postgres \
       --from-literal DB_PASSWORD=mysecretpassword \
       --from-literal ENTITLEMENTS_JWT='your_JWT_value'
2. (Alternative to Step 1 – creating secret manually) Add your DB credentials to a new secret (keeping in mind k8s secrets are base64 encoded):
   1. DB_HOST
   2. DB_NAME
   3. DB_USERNAME
   4. DB_PASSWORD
   5. DB_PORT
3. Uncomment the secret field in values.yaml, and add the name of the secret you just created
4. Remove the DB_* fields from templates/backend/configmap.yaml & templates/worker/configmap.yaml in the Helm chart
5. Run helm upgrade

Resources

1. 2 instances of backend container, with 8GB memory per instance (2 cores recommended).
2. 1 instance of frontend container, low memory requirement (< 500mb is enough, it's a static nginx proxy).
3. Postgres DB v15 with 15GB storage and 4GB memory with max number of connections ≥100. The max number of connections should be ≥2x the backend's connection pool size, which is 50 by default (25 per instance).

Self-Managed SSL Certificates

In some cases, you may want to enable your Cortex deployment to access managed services within your infrastructure (such as a self-hosted GitLab instance) that have their own SSL Certificates. In these cases you can add a given certificate to the trusted keystores by performing the following:

1. Create a kubernetes secret in the same namespace as your Cortex helm installation called tls.cert with the contents of the certificate file. kubectl create secret generic tls-secret --from-literal tls.crt="Replace with contents of cert file"
2. Update your values.yaml to enable selfSignedCerts and ensure the secret name is the same as the one created in step #2.

app:
   ...
   backend:
       enabled: true
      ...
       selfSignedCerts:
         enable: true
         secret: tls-secret
   worker:
       enabled: true
      ...
       selfSignedCerts:
         enable: true
         secret: tls-secret

Note: These are service by service configurations, and as such must be set for each service in the values.yaml that you want to use the provided certificates.

3. Run helm upgrade

Metrics

Prometheus

Cortex exposes Prometheus metrics on the /actuator/prometheus endpoint for the backend & worker pods. Exposed on the port 8181 Enable this by setting PROMETHEUS_METRICS_ENABLED=true in your backend & worker configmaps.

Debugging

Cortex provides a CLI to debug the on-prem installation. All commands create a data folder that has the data needed for the Cortex team to debug the issue.

Helm / K8s dump

You can use brain-freeze k8s dump [options] command to dump all the information about the installation. Note The CLI only dumps non-sensitive information. For the secrets, the values are masked.

K8s logs

You can use brain-freeze k8s logs [options] to fetch the logs for all Cortex related deployments.

Cortex-specific info

After a successful installation, the Cortex team might ask for settings/configurations on your setup. You can use /brain-freeze backend info --token="[CORTEX PAT]" --url="[BACKEND HOST]" to get this information.

You can create a Cortex PAT from Settings -> Personal access tokens in the app.