Skip to main content

GitOps

Cortex makes it easy to update catalogs directly through the UI, but also supports a GitOps approach for greater control and velocity. Using a GitOps model - which entails using descriptor files to manage entities in Cortex through your version control system - provides a few benefits:

  • Metadata are version-controlled.
  • The repository where your code lives is also the source of truth for information.
  • You always own the data.

The GitOps model is powered by the entity descriptor file - also called the Cortex YAML - which describes each entity in your catalogs. Cortex checks for cortex.yaml or cortex.yml anywhere in the default branch (for GitHub and GitLab) and processes any changes pushed to the default branch.

All of Cortex's official Git integrations support automatic parsing of the entity descriptor file, enabling users to switch to GitOps in under five minutes.

When should I switch to GitOps?

It's best to switch to a GitOps model after Cortex has been broadly rolled out to your organization. Ideally, developers will have the opportunity to experiment with Cortex through the UI, and leaders will set a threshold for a GitOps cutover.

Basic configuration​

Getting started​

Before you can move to a GitOps approach, Cortex must be integrated with GitHub, GitLab, Azure DevOps, or Bitbucket.

Next, confirm that the Cortex UI editor is disabled for each entity type you want to use a GitOps approach for. From the GitOps page in Entities settings, you can enable or disable UI editing/importing for services, domains, teams, and other entity types.

Instead of using the UI editor, you'll make changes to your catalog data through an entity descriptor file and sync the changes to this file using either Cortex's official Git integrations or programmatically using the Cortex API.

This is to ensure consistency - if the UI editor is enabled, any changes made to the Cortex YAML in the repo will overwrite changes made in the UI.

Mixed settings​

It is possible to enable UI editing and disable UI importing for any entity. While this would allow users to create new entities via GitOps, they must make changes to the entity through the UI - any changes made via GitOps would not register in Cortex.

Entity GitOps repository allowlist​

The Entity GitOps repository allowlist settings allow admins to restrict the repositories from which entities YAML can be imported.

By default, Cortex will check all repositories for services, domains, teams, and other entity types. When one or more repositories are selected for a given entity type, Cortex will only check those repos for cortex.yaml changes.

caution

If you make changes in a repo that is not designated on the allowlist, Cortex will not process those changes.

Built-in integrations​

While GitHub, GitLab, Azure DevOps, and Bitbucket have different steps for setting up, there are several similarities:

  • Cortex will only check for files in the default branch (e.g. default-branch) unless otherwise specified. Cortex defaults to main if there is no default branch defined.
  • Cortex does not delete Scorecards if a corresponding scorecard YAML is deleted. You can enable automatic archival of entities through GitOps by toggling on "Enable auto archiving of services" in Settings → Entities → General.
  • The recommended placement for entity descriptor files is in the root of the repository or in the appropriate .cortex/catalog folder.
    • If you're using GitHub or GitLab, the descriptor can be located anywhere in the repository as long as the file is named cortex.yaml or cortex.yml.
    • For Bitbucket, Bitbucket Server, or Azure DevOps, you MUST place descriptor files in the appropriate .cortex/ subdirectory. It is possible to work around this for unique cases.
  • Domain, team, and Scorecard definitions must be in the .cortex/domains, .cortex/teams, or .cortex/scorecards folders, respectively.
  • You can define any number of entities within the same repository. Here is an example structure:
.
└── .cortex
    ├── catalog
    │   ├── database.yml
    │   ├── s3-bucket.yml
    │   ├── auth-service.yml
    │   └── billing-service.yml
    ├── domains
    │   ├── billing-domain.yml
    │   └── health-domain.yml
    ├── teams
    │   ├── eng-team.yml
    │   └── sre-team.yml
    └── scorecards
    │   ├── service-readiness.yml
    │   └── security.yml
Disable UI editing to enable GitOps

None of the integrations outlined below will work if in-app UI editing is enabled. Make sure to disable UI editing in Settings → Entities → GitOps.

GitHub​

Cortex has an official GitHub app that comes pre-configured for using the GitOps approach in Cortex. You can read more about the GitHub app in our GitHub integration docs.

Creating an entity​

To create a new entity from a new repository, first create the repository in the Cortex app (e.g. example-repo). Create a new cortex.yaml or cortex.yml file with the entity's properties in the repo you just created.

title: Example Entity
x-cortex-git:
  github:
    repository: example-repo
x-cortex-tag: example-entity
x-cortex-type: service
x-cortex-owners:
- name: docs-team
  type: GROUP
x-cortex-custom-metadata:
  test: 123
tip

The GitHub app has a built-in linter, so if an entity descriptor file is invalid, the GitHub app will comment on the pull request with outstanding issues.

There are two ways to verify that the new entity was created:

  • GitOps logs: GitOps logs (available in settings) displays all changes made in your Cortex workspace. If your entity creation was successful, you'll see it appear at the top of the list.
  • All entities: You can search the All entities page for the new entity's name or tag. If your creation was successful, it will appear in the search results.
caution

You'll need to refresh Cortex after adding it in the repo to see changes on the All entities page or in the GitOps logs.

Editing an entity​

Any changes you commit to an entity will appear under Recent activity on the entity page overview. When you define custom data in the x-cortex-custom-metadata block, it will also appear on that entity's Custom data page, which you can access from the side panel.

For example, to add custom data to the entity defined above, we would add a field in the x-cortex-custom-metadata block.

x-cortex-custom-metadata:
  test: 123
  test2: 456

Once the entity details page is refreshed, the new data will appear under Recent activity. In this example, both test and test2 will also appear on the Custom data page, with 123 and 456 listed in their respective rows in the Value column. The change will also appear in GitOps logs.

caution

If UI editing is enabled for an entity type, any changes you commit to that entity will not be reflected in Cortex.

If UI editing is enabled and UI importing is disabled, changes will display in GitOps logs and under an entity's recent activity, but those changes will not process. The Entities column in GitOps logs will show 0 entities if this is the case. When you open the commit, the panel will show No changes processed, and the cortex.yaml file will appear under Omitted files.

Multi-account configuration​

In addition to Cortex's GitHub app, you can use personal access tokens and custom GitHub apps to configure additional GitHub accounts.

If you're creating or editing a cortex.yaml in the non-default configuration, you must reference the alias. For example, if you've added a second GitHub configuration called non-default-example, you'd define the following block in the entity descriptor.

x-cortex-git:
  repository: organization/non-default-example-repo
  alias: non-default-example
caution

If you do not define the alias, Cortex will use the default configuration when processing changes made to the cortex.yaml file. If a repository is not included in the default configuration, changes will not be processed via GitOps.

GitHub webhook​

If you do not wish to use the official app, or prefer to integrate with GitHub using a personal access token, you can manually add a webhook to your GitHub organization.

  1. Generate a personal access token in GitHub.
  2. Go to GitHub settings in Cortex and click Add GitHub configuration. Select Personal access token from the dropdown in the corner to add the token and an alias for the configuration.
  3. Once the configuration is saved, add a secret passphrase in the Secret field under Webhook Secret. Save the passphrase somewhere secure - GitHub will use this to prove the webhook event is valid.
  4. Save the passphrase to generate a unique webhook URL. The URL will end with {alias} - make sure to replace this with the alias for the configuration you just created.
  5. Follow the instructions from GitHub on creating a webhook. Cortex recommends adding an organization webhook, but you can also define the webhook for a repository in the org. Set the content type to application/json. GitHub will request:
    • The secret passphrase from step 3.
    • The webhook URL from step 4.

Once GitOps is enabled, Cortex will detect push events in your repositories. For the first webhook for a given repository, Cortex looks for the cortex.yaml files and processes them. For subsequent webhooks, Cortex only processes files with a change in the webhook event. A maximum of 3,000 changed files will be reported per commit.

caution

The maximum number of files that can be reported via GitHub webhooks in each commit is 3,000.

GitLab​

Due to the lack of native support for applications, enabling GitOps via GitLab requires setting up a webhook so Cortex is notified when an entity descriptor is modified.

If you haven't already set up the GitLab integration, make sure to do that first. Once the integration is established, follow these steps:

  1. Create a token for your webhook from GitLab settings in Cortex.
  2. In GitLab, create a system hook(recommended) or a project hook.
  3. Copy the webhook URL - https://api.getcortexapp.com/api/v1/gitlab/webhook/{alias} - and replace {alias} with the alias for your GitLab configuration.
  4. Enable Push events for the webhook.
  5. Add the secret token from step 1 to the webhook configuration. This token allows Cortex to verify webhook events.

Azure DevOps​

To enable GitOps via Azure DevOps, you need to manually add a webhook. If you haven't already set up the Azure DevOps integration, make sure to do that first.

  1. Navigate to Settings → Azure DevOps and validate your Azure DevOps integration.
  2. Click Create a new webhook and copy the unique webhook URL.
  3. Follow the instructions from Azure on adding a webhook. Set the event type to Code pushed and use the URL from step 2.

Bitbucket​

The Bitbucket cloud integration will facilitate all configuration needed to enable GitOps. You can add a configuration from Bitbucket settings in Cortex.

info

You'll need to enable development mode to use the Cortex app. You can find this setting at https://bitbucket.org/{your_workspace}/workspace/settings/addon-management/ where {your_workspace} is the name of your workspace.

Bitbucket Server​

To enable GitOps via Bitbucket Server, you need to manually add a webhook. If you haven't already set up the Bitbucket Server integration, make sure to do that first.

  1. Navigate to Settings → Bitbucket and validate your configuration.
  2. Enter a secret token for your Bitbucket Server webhook.
  3. After saving the secret, click Create a new webhook and copy the unique webhook URL.
    • For a non-default configuration, the suffix /{alias} should be added to the generated URL.
  4. Follow the Bitbucket Server instructions for adding a project-level or repository-level webhook. Set the event type to repository push. Use the secret from step 2 and the URL from step 3.

API upload​

For custom integrations, you can use the Cortex API to upload descriptor files. You might use the API when you either cannot or do not want to use GitOps to manage entities.

  1. Create an API key from API keys settings in Cortex.
  2. Use the key to authenticate when uploading entity descriptors via the Create or update entity endpoint with the API key set as a header: Authorization: Bearer <key>.

Example​

curl \
  -v \
  --data-binary @cortex.yaml \
  -X POST \
  -H "Content-Type: application/openapi" \
  -H "Authorization: Bearer <token>" \
  "https://api.getcortexapp.com/api/v1/open-api"
tip

Upload this file as part of your CI/CD process so a given entity never goes stale.

Dry run​

This API also accepts two optional parameters:

  • dryRun (boolean): Whether or not to persist the changes being uploaded. Defaults to false.
    • This API can be used to lint your YAML file or check for breaking API changes as part of a CI process.
  • githubPullRequest (pull request ID): In conjunction with the dryRun flag, enables Cortex to comment on a pull request with any detected breaking changes.

Example​

curl \
  -v \
  --data-binary @cortex.yaml \
  -X POST \
  -H "Content-Type: application/openapi" \
  -H "Authorization: Bearer <token>" \
  "https://api.getcortexapp.com/api/v1/open-api?dryRun=true&githubPullRequest=17"

Monorepos​

When using GitOps with a monorepo - one Git repository with multiple entities - you can use one cortex.yaml or cortex.yml file per subdirectory to represent each entity.

Details on the relevant configuration used are covered on the respective integration pages for GitHub, GitLab, and Azure DevOps. Bitbucket requires special considerations when using monorepos.

Advanced configuration​

Cortex's out-of-the-box GitOps configuration suits most common use cases:

  • Single or many projects per repo
  • Only one branch needs to be processed for cortex.yaml
  • The cortex.yaml file is in the default or main branch

However, there may be scenarios that require special setups:

  • Monorepos in Bitbucket: Multiple projects in a single repo, split into subfolders.
  • Branches: Non-main/default branches, or different projects in multiple branches.

To account for these use cases, you can "configure" Cortex by telling us how to process events from your repo.

Cortex properties YAML file​

The configuration for a repository lives in a cortex-properties.yaml file:

  • Inclusion of this file in your repo is fully optional.
  • The file is automatically processed, just like the entity descriptor.
  • It should live in the default branch for the repo, regardless of which branches it states Cortex should use to find entity descriptor files.
info

Don't use this file unless you need advanced configuration - this file is optional and should only be used if you have a custom workflow, like using monorepos in Bitbucket or using a non-default branch as the home for your entity descriptor files.

You do not need to set up cortex-properties.yaml if you're using a monorepo and integrating with GitHub or GitLab, as Cortex supports this out of the box.

Branches​

You can configure Cortex to automatically process cortex.yaml files in non-standard branches, multiple branches, or both. Consider the scenario:

  1. You have a project where main is protected and is the default branch.
  2. You want to include cortex.yaml in the develop branch.
  3. You also have a separate project version in a staging branch with its own cortex.yaml file.

To represent this, you would add a cortex-properties.yaml file in the default branch of your repo, and define the branches field with a list of branches to process. The default branch must be explicitly defined if you're using advanced configuration and you want Cortex to search for a cortex.yaml file in the default branch.

branches:
  - main
  - develop
  - staging

If your cortex-properties.yaml file does not contain a branches field, Cortex will continue to process the default branch.

caution

When using the branches field in your cortex-properties.yaml file, make sure to include the default branch if you want Cortex to continue looking for a cortex.yaml file in the default branch.

Source directories​

You can configure Cortex to look for cortex.yaml files in multiple subdirectories. Consider the scenario:

  1. You have a monorepo structure where all projects live in a single repository.
  2. Each project lives in a subdirectory in the main repository (project1/, project2/, etc.).
  3. Each project has its own cortex.yaml file.

To represent this, you would add a src-dirs field in a cortex-properties.yaml file at the root of the repository, containing a list of directories to process.

src-dirs:
  - project1
  - project2

Cortex will still process any cortex.yaml file found in the root of the repository.

FAQs and Troubleshooting​

Conflicts between UI editing, GitOps, and the Cortex API​

If GitOps has previously been enabled, but UI editing is temporarily turned on, any changes made in Cortex to applicable entities will not be reflected in Git. When the file is next changed through your Git provider, it will override changes made in the UI.

The last received change in a cortex.yaml file will override previous changes, whether it originated from the create/update entity API or a push from your Git provider. Changes are not appended and the last submitted entire file takes precedence, so fields omitted in cortex.yaml will be removed.

Will my cortex.yaml file be picked up immediately?​

If you already have a cortex.yaml file when you set up GitOps, Cortex will automatically process it. However, the file will not be processed until UI editing is disabled.

The entity I created appears in GitOps logs, but displays 0 entities 0 scorecards in the Entities column.​

First, use the YAML linter to validate your cortex.yaml file. Then, confirm GitOps settings are configured correctly:

  • Make sure UI editing is disabled for the entity type that you're trying to create.
  • Check repositories in the GitOps repository allowlist(#entity-gitops-repository-allowlist). If there are repositories selected for the entity type you're working with, confirm that you're working from an allowed repo.