Additional self-managed resources
After following the instructions on this page to get started with your self-managed Cortex account, see these additional guides to enable other features and integrations:- User management: Enabling SSO, connecting users via GitHub OAuth, configuring email notifications
- Configuring integrations: Atlassian Connect app, AWS, GitHub app, GitHub OAuth, Microsoft Teams, Slack
- Upgrades and rollbacks: Upgrading or rolling back Self-Managed Cortex
- Enable AI Docs Assistant: Access Cortex’s public docs from within the app
Getting started with a Cortex self-managed account↗
To deploy Cortex on-premises, you will:- Make sure you have the required environment and infrastructure set up
- Create a PostgreSQL database where Cortex data will be stored
- Create Kubernetes secrets for pulling software images and connecting to the database
- Create a YAML file containing your configuration values for the Cortex Helm chart
- Preview the Kubernetes deployment
- Deploy using Helm
- Configure DNS records to allow users to access Cortex
Prerequisites
See the prerequisites below:Prerequisites
Prerequisites
- A running Kubernetes cluster, configured with permission to make HTTPS requests to the public internet (our Docker images are hosted on GitHub) - you can use AWS/EKS, Azure/AKS, GCP/GKE, or your own locally installed Kubernetes distribution.
- Permission to add services, deployments, secrets and namespaces to the cluster
- kubectl installed and connected to your Kubernetes cluster
- Helm Package Manager installed
- A PostgresSQL database (15+) that is accessible from your k8s cluster, and has a dedicated database and user for Cortex.
- Learn more under Step 2: Set up a persistent database.
- A GitHub personal access token (PAT) and a JWT license key, both provided to you by your Cortex account team
- 2 instances of backend container, with 8GB memory per instance (2 cores recommended)
- 1 instance of worker container, with 8GB memory (2 cores recommended)
- 1 instance of frontend container
- 500MB memory is generally sufficient, but you can adjust based on your available resources.
- DNS Names: Create two DNS records in your domain:
- One for the frontend (web UI): e.g.,
cortex-app.yourdomain.dom - One for the backend (API): e.g.,
cortex-api.yourdomain.dom
- One for the frontend (web UI): e.g.,
- TLS Certificate: Obtain a TLS certificate that covers both DNS names. You can use:
- A wildcard certificate (e.g.,
*.yourdomain.dom) - A certificate with Subject Alternative Names (SANs) for both DNS names
- For AWS: AWS Certificate Manager (ACM)
- For Azure: Azure Key Vault certificates
- A wildcard certificate (e.g.,
- DNS Configuration: After installation, you’ll need to point DNS records for the frontend and backend donain names to the two load balancers created by Kubernetes. The exact configuration depends on your environment:
- For AWS: Point to each NLB’s DNS name
- For Azure: Point to each Load Balancer’s public IP
- For self-managed load balancers: Point to your load balancers’ IPs or hostnames
Verify Kubernetes connection
Verify Kubernetes connection
Step 1: Set up a persistent database
Cortex stores all persistent data in a PostgreSQL database. You will need to set up a database in order to install and use Cortex.↗Step 1.1: Database prerequisites
Step 1.1: Database prerequisites
- Self-managed PostgreSQL: Install PostgreSQL on your own infrastructure
- AWS RDS: Use Amazon RDS for PostgreSQL if running on AWS
- Azure Database for PostgreSQL: Use Azure’s managed PostgreSQL service if running on Azure
- 15GB minimum available storage, 40 GB recommended
- Maximum number of connections ≥100. The maximum number of connections should be ≥2x the backend connection pool size, which is 25 per instance, for a total of 50 when two backend containers are running.
Step 1.2: Create a Postgres database
Step 1.2: Create a Postgres database
- On your PostgreSQL server, create a database with UTF-8 encoding that will be accessible from your instance.
- Create a user with the following permissions on the
public.schemawithin the database you created:- Add
ALLon the public schema to create/modify tables - Add
INSERT,SELECT,UPDATE,DELETEon all tables in the public schema
- Add
- Make a note of the following database connection details, as you will need them in the next steps:
- The hostname and port number for connecting to the database server
- The name of the database you created (this guide assumes the database is named
cortex) - The username and password to access the database
Step 1.3: Verify database connection
Step 1.3: Verify database connection
- Connect to your database from within the cluster
- Create and drop a test table to verify write permissions
- Exit with an error if the database doesn’t exist or the user lacks permissions
Step 2: Configure your Kubernetes cluster
Configure Kubernetes namespace and secrets
Configure Kubernetes namespace and secrets
- Create a Kubernetes namespace. A namespace is optional, but it’s recommended to install Cortex in its own namespace for better organization and isolation:
- Run the following command to add a Docker registry secret for pulling Cortex images from the GitHub Container Registry:
- Replace
<enter the GitHub PAT provided by Cortex>with the value of the GitHub PAT that was provided to you by your Cortex account team, and replace<yourname@example.com>with an email address.
- Run the following command to create a secret for your Cortex license and your database connection details:
- Substitute the placeholders in this command with the actual values of your port, username, password, database, and Cortex license JWT.
kubectl.↗Step 3: Add the Cortex repo to Helm
Add the Cortex Helm repo
Add the Cortex Helm repo
- In command line, run the following command to add the Cortex repo to Helm:
- Install the Helm diff plugin to preview changes before applying them:
Step 4: Configure and install the helm chart
- The default values for resource allocation in the chart are provided as guidance, but you may need to adjust these depending on your resource consumption.
- User authentication is disabled by default. After installing Cortex, visiting the frontend URL in a browser will immediately log you in as a demo user. You will then be able to configure authentication using an OIDC provider.
- The chart does not ship with built in TLS or certificates. If you are deploying Cortex into a cluster behind your existing load balancer with TLS, change the
protocolvalue in the chart.
Step 4.1: Configure the helm chart
Step 4.1: Configure the helm chart
values.yaml directly. Instead, we recommend that you create a separate overrides.yaml file for your configuration. This approach ensures:- Configurations are preserved across upgrades
- Easy version control of your settings
- Clear separation between defaults and customizations
- Timely support from Cortex in the event of an issue
overrides.yaml file. In the next step, you will install the chart and include the overrides.yaml.See a list of available settingsTo see a complete list of available settings, you can review the values.yaml that is bundled in the Cortex Helm chart by running the following command:overrides.yaml, adjust the replica counts and resource allocations based on your expected load and availability requirements.At a minimum, the following values should be set:image.secrets.name: The name of the Docker registry secret you set in step 2app.service.type: Set this field based on your infrastructure. If you’re using EKS, AKS, or GKE, set the type asLoadBalancer. If you’re using your own Kubernetes distribution and plan to do your own load balancing, configureNodePort.app.ingress.type: The type of ingress controller to use with your load balancing setup. For EKS and AKS, usenginx. For GKE, usegcp. If you are using your own Kubernetes distribution, this setting is not used.app.secret: The name of the secret you set up in step 2 containing your Cortex license key and database connection detailsapp.hostnames.frontend: The DNS name where users will visit the Cortex Web UI in the browser. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the frontend service.app.hostnames.backend: The DNS name where the Cortex REST API will be available. This should match the common name or a subject alternative name on the TLS certificate you use with your load balancer for the backend service.app.hostnames.protocol: Set tohttpsif you are configuring TLS termination at the load balancer (recommended), orhttpif you will use Cortex over cleartext. Note that this field does not set up TLS - TLS is configured at the load balancer; this setting is used only for generating URLs.app.backend.enabled: Set totrueto enable the backend containers to runapp.worker.enabled: Set totrueto enable the worker containers to run
app.backend and app.worker:replicaCount(default: 2) This field determines how many containers to create. Depending on your resource constraints, you may need to change this to1.containerConfiguration.resources: Depending on your resource constraints, you may need to adjustrequests.memoryandlimits.memory. By default, each container will consume a minimum of 8GB and a limit of 16GB.containerConfiguration.livenessProbe.periodSeconds(default: 10) andcontainerConfiguration.livenessProbe.timeoutSeconds(default: 6): In environments with slow networking or database, probes may expire before the containers are fully started, causing them to continually be restarted. If this occurs, you may need to raise the values of these settings.
overrides.yaml implementing one backend pod and one worker pod with minimum resources, and configuring TLS termination using network load balancers in AWS:Step 4.3: Install the helm chart
Step 4.3: Install the helm chart
- Identify the version of Cortex you want to install.
- Consider consulting with your Cortex account team to identify the appropriate version. Alternatively, you can find the latest version by running
helm search repo cortex/cortex. Cortex version numbers look like0.0.411. - Make a note of the version you are installing, in case you want to change configuration values while remaining on the same version.
- Consider consulting with your Cortex account team to identify the appropriate version. Alternatively, you can find the latest version by running
- Before installation, use the following command to preview what will be deployed. Replace
DESIRED_VERSIONwith the version number you identified in the previous step:
- Use helm to install Cortex. Replace
DESIRED_VERSIONwith the version number you identified in the first step:
- Monitor the deployment status:
Step 5: Configure DNS
After installation, Kubernetes will create two load balancers: one for the frontend and one for the backend. Configure your DNS records:Configure and verify DNS
Configure and verify DNS
-
Get the load balancer endpoints:
You’ll see two services of type
LoadBalancer. Identify which is frontend and which is backend by their names. -
Update your DNS records:
- AWS NLB: Create CNAME records pointing to the respective load balancer DNS names.
- Azure: Create A records pointing to the respective load balancer IPs.
- Point your frontend hostname (for example,
cortex-app.yourdomain.dom) to the frontend load balancer. - Point your backend hostname (for example,
cortex-api.yourdomain.dom) to the backend load balancer.
-
Verify DNS resolution:
-
Test access to Cortex:
- Open your frontend URL (for example,
https://cortex-app.yourdomain.dom) in your web browser.- You should be logged in automatically as “Demo User.”
- After this, you can configure SSO for proper authentication.
- Open your frontend URL (for example,
Additional configuration for a cloud cluster installation↗
If you’re installing Cortex into a cloud cluster, for example EKS or GKE, you may need to install required plugins to your cluster. For example,kubectl apply the nginx plugin for AWS.
Backup and recovery
As a best practice, we recommend regularly backing up the following:- Your Kubernetes secrets
- To get copies of your Kubernetes secrets as YAML files that can be restored using
kubectl apply, use commands like the following:
- To get copies of your Kubernetes secrets as YAML files that can be restored using
- Your overrides.yaml file
- This contains all the configuration values for your Cortex installation. We recommend keeping this file in a revision control system.
- Your PostgresSQL database
- This contains all of your Cortex data. We recommend that you implement a regular backup schedule and periodically test restore procedures.
Recovery process
- Create namespace if it doesn’t exist:
kubectl create namespace cortex - Restore PostgreSQL database from backup
- Apply Kubernetes secrets to the cortex namespace
- Install Cortex using your saved
overrides.yamland the appropriate version
Backup and recovery best practices
- Don’t fork or modify the Helm chart: If you fork or modify the Helm chart itself, it will be much more difficult to obtain timely support from Cortex. If it seems like the chart needs to be modified, please reach out to your Cortex team to see if there are value overrides available, or if Cortex can modify the official Helm chart to meet your needs.
- Version Control: Keep your
overrides.yamlin version control - Documentation: Document any custom configurations or integrations
- Testing: Always test upgrades in non-production first
- Monitoring: Set up monitoring for Cortex components
- Regular Updates: Stay current with Cortex releases for security and feature updates
- Backup Verification: Regularly test your backup and restore process↗
Additional configuration options
Modifying the configuration after installing↗
You can make changes to your configuration by editingoverrides.yaml and applying them using helm upgrade. First, identify the Cortex version you are running - the Cortex version appears at the bottom of the left-hand nav in the settings page in the Cortex UI. Cortex version numbers look like 0.0.411.
Next, preview the changes:
helm upgrade:
Self-Managed SSL Certificates↗
This configuration is used when Cortex needs to act as a client connecting to other services in your infrastructure that use self-signed or private CA certificates. Common use cases include:- Connecting to a self-hosted GitLab instance with custom certificates
- Accessing internal APIs that use private certificate authorities
- Integrating with on-premises services using self-signed certificates
Configuration
Before getting started, note the following prerequisites and considerations:Certificate prerequisites
Certificate prerequisites
- Certificates must be in PEM format (Base64 encoded).
- If you have certificates in other formats, convert them to PEM:
- DER to PEM:
openssl x509 -inform der -in cert.der -out cert.pem - P7B to PEM:
openssl pkcs7 -print_certs -in cert.p7b -out cert.pem - PFX to PEM:
openssl pkcs12 -in cert.pfx -out cert.pem -nodes
- DER to PEM:
- If you have certificates in other formats, convert them to PEM:
- Certificates must include the complete certificate chain if they are using intermediate CAs.
- Each certificate should begin with
-----BEGIN CERTIFICATE-----and end with-----END CERTIFICATE----- - Multiple certificates can be concatenated in a single file.
- After following these instructions, the certificate will be added to the Java trust store in both backend and worker pods.
- This configuration is only needed for outbound connections from Cortex
- Be sure the full certificate chain is included, since importing only the leaf certificate can still result in PKIX path building errors when Java services attempt internal HTTPS calls.
- Inbound TLS/SSL should be configured at the load balancer
- Create a kubernetes secret in the same namespace as your Cortex helm installation called
tls.cert, containing the contents of the certificate file:
- In your
overrides.yamlfile, enableselfSignedCertsand ensure the secret name is the same as the one created in the last step. In the example below, the secret is calledtls-secret:
- Repeat the previous step for each service in your
overrides.yamlwhere you want to use the provided certificate. - Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.
Troubleshoot SSL/TLS certificate issues
Troubleshoot SSL/TLS certificate issues
Using Envoy Gateway (alternative to NGINX Ingress)
As an alternative to the default NGINX Ingress controller, the Cortex Helm chart supports Envoy Gateway using the Kubernetes Gateway API. This is the recommended path forward, as NGINX Ingress is being deprecated.Basic Envoy Gateway setup
Basic Envoy Gateway setup
overrides.yaml to disable Ingress and enable the Gateway API:- A
Gatewayresource using theenvoy-gatewayGatewayClass HTTPRouteresources for the frontend and backend services- An
EnvoyProxyresource that configures the Envoy proxy service
Referencing an existing Gateway
Referencing an existing Gateway
Cloud provider annotations
Cloud provider annotations
Accepted. If the Gateway shows Programmed, the Envoy proxy is ready to receive traffic. Update your DNS records to point to the new load balancer endpoint as described in Step 5: Configure DNS.
Troubleshooting Envoy Gateway
If you’re experiencing issues with Envoy Gateway, the most useful debugging step is checking the status of the Gateway and HTTPRoute resources. These resources report detailed conditions that indicate where things are getting stuck in state transitions.Check Gateway status
Check Gateway status
Accepted and Programmed conditions with status: "True", and an address assigned:AcceptedisFalse: The GatewayClass may not exist, or the Envoy Gateway controller isn’t running. Checkkubectl get gatewayclassandkubectl get pods -n envoy-gateway-system.ProgrammedisFalse: The Envoy proxy deployment may be failing. Check for resource constraints or image pull errors.- No address assigned: The LoadBalancer service may be pending. Check
kubectl get svc -n envoy-gateway-system. attachedRoutesis 0: No HTTPRoutes have successfully attached to this listener.
Check HTTPRoute status
Check HTTPRoute status
Accepted and ResolvedRefs conditions with status: "True":AcceptedisFalse: The route’sparentRefmay not match an existing Gateway, or the Gateway’sallowedRoutesmay restrict the namespace.ResolvedRefsisFalse: A backend service referenced in the route doesn’t exist or has the wrong port. Verify withkubectl get svc -n cortex.- No
parentsentries: The route hasn’t been picked up by any controller. Check that the Envoy Gateway controller is running.
Enabling Redis↗
Cortex provides optional Redis support by provisioning Redis in Kubernetes as part of the Helm chart. Enable Redis to gain support for outbound rate limiting of integration API calls. Without enabling Redis, Cortex will still function, but may not have rate limiting capabilities for external API calls. Redis is enabled via a third-party Helm chart; a full reference to all of the possible settings can be found in this repository on GitHub.Enable basic Redis
Enable basic Redis
- Set
redis.enabledtotruein youroverrides.yaml:
- Run helm to apply the changes as outlined in .
Set authentication for Redis
In production environments, it is recommended that you enable authentication.Configure the Redis subchart to use your secret
Configure the Redis subchart to use your secret
- Create a secret for the Redis password:
- Open your
overrides.yaml. Configure the Redis subchart to use that secret:
- Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.
Embedded Redis upgrades
Embedded Redis upgrade instructions
Embedded Redis upgrade instructions
- If you have set
redis.global.imagePullSecrets, ensure that the list of secrets includes the secret that enables you to pull the Cortex images, usuallycortex-docker-registry-secret.- If you have not set
redis.global.imagePullSecrets, no action is required.
- If you have not set
- New Redis images are hosted by Cortex in ghcr.io/cortexapps/helm-chart/. If you have set any image URI options, or mirrored the old official images locally, you will need to update your values to use the new images.
- Image options include:
redis.global.imageRegistryredis.image.registry,redis.image.repository, orredis.image.tagredis.metrics.image.registry,redis.metrics.image.repository, orredis.metrics.image.tag
- If none of the above are set, no action is required.
- Image options include:
- Ensure that all of the following settings are false:
redis.sentinel.enabledredis.volumePermissions.enabledredis.sysctl.enabledredis.kubectl.enabled
Monitor Redis usage
You can monitor Redis usage to ensure it’s properly sized:Monitor Redis
Monitor Redis
Enabling metrics↗
Prometheus↗
Prometheus metrics provide visibility into Cortex’s performance and health, enabling:- Performance monitoring: Track response times, request rates, and error rates
- Resource utilization: Monitor memory usage, CPU consumption, and thread pools
- Integration health: Track success or failure rates for external API calls
- Alerting: Set up alerts for degraded performance or failures
- Capacity planning: Understand usage patterns to plan scaling
Publish metrics and configure Prometheus
Publish metrics and configure Prometheus
- Set
app.shared.prometheusMetrics.enabledtotruein youroverrides.yaml:
- Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.
/manage/prometheus (http://localhost:8181/manage/prometheus) for both backend and worker pods.Prometheus configurationTo scrape metrics from Cortex, configure your Prometheus instance to scrape pods directly, for example:Available Prometheus metrics
Available Prometheus metrics
http_server_requests_seconds- HTTP request latenciesspring_data_repository_invocations_seconds- Database query durationscomparatron_operation_seconds- External API call durationsscorecard_latest_cache_access_total- Scorecard cache hit/miss ratesexecutor_active_threads- Background job thread activityexecutor_queued_tasks- Queued background tasks
jvm_memory_used_bytes- Memory usage by areajvm_gc_pause_seconds- Garbage collection pause timesjvm_threads_live_threads- Active thread countjvm_classes_loaded_classes- Loaded class count
hikaricp_connections_active- Active database connectionshikaricp_connections_pending- Pending connection requestshikaricp_connections_idle- Idle connections in poolhikaricp_connections_timeout_total- Connection timeout count
Enable automatic discovery by Prometheus
Enable automatic discovery by Prometheus
- Add annotations in
overrides.yaml:
- Preview and apply the updated configuration by following the steps in Modifying the configuration after installing above.
Prometheus alerting rules
Prometheus alerting rules
Grafana dashboard
Grafana dashboard
- Background job executors (
executor_active_threads) - Scorecard cache performance (
scorecard_latest_cache_access_total) - Integration API calls (
comparatron_operation_seconds_count) - Database query performance (
spring_data_repository_invocations_seconds)
cortex-dashboard.json and import it into Grafana:Troubleshooting Prometheus metrics
Troubleshooting Prometheus metrics
Enabling feature flags
Occasionally, Cortex may ask you to enable a feature flag. To do so, add settings to theapp.shared.featureFlags map in your overrides.yaml:
Enabling AI Docs Assistant
- Reach out to Cortex requesting Docs Assistant enablement. We willy supply you with a unique integration ID.
- Once obtained, add
REACT_APP_DOCS_AI_INTEGRATION_IDto the window variables configuration in your Cortex helm chart’svalues.yamlfile
- Allowing Cortex to collect analytics through the AI Docs Assistant chat. This includes tracking questions that were asked in the chat interface and by who. Feedback (downvoting, upvoting, and comments) is also collected through the chat interface.
- Allowing reCAPTCHA usage. The Docs Assistant feature requires bot protection.
Getting Help
When working with Cortex Customer Support, you will be asked to collect diagnostic data using a tool we distribute calledbrain-freeze. To install brain-freeze, follow this link to the latest release, and download the .tar.gz file that matches the operating system and architecture where you are running kubectl. Extract the file on the machine where you are running kubectl, and put the enclosed brain-freeze binary somewhere in your path. Then you can run:
data subdirectory in your current directory, containing the Kubernetes logs from your deployment from the last 24 hours (1440 minutes) and the details of your Kubernetes deployment configuration.
The brain-freeze CLI does not dump sensitive information. The values of secrets are masked.