Skip to main content

Using GitOps for Cortex

As an alternative to working through the Cortex UI, Cortex supports following a GitOps approach. Using a GitOps model - which entails using descriptor files to manage entities in Cortex through your version control system - provides some benefits:

  • Metadata are version-controlled
  • The repository where your code lives is also the source of truth for information
  • You always own the data
  • You can easily monitor and track GitOps changes via GitOps logs
    • Users must have the View GitOps logs permission.

How GitOps works in Cortex​

When following the GitOps model in Cortex, you manage entities via entity descriptor files that live in your git repository. An entity descriptor, also referred to as a Cortex YAML, describes an entity in your catalogs. You may store all of your entity definitions in a single repository or you can store the YAML in the repository of the corresponding entity.

Cortex checks for cortex.yaml or cortex.yml anywhere in the default branch (for Bitbucket, GitHub, and GitLab) and processes any changes pushed to the default branch. For Azure, Cortex checks the .cortex directory.

If an entity's file is deleted from the repository and you have enabled GitOps-based auto-archival, then the corresponding entity in Cortex will be archived. This feature checks for deleted files, so if the file is moved, it will not be archived from Cortex. See Auto archiving entities for more information.

GitOps example repository structure​

The following example shows a single repository structure. Entity descriptor files are listed under catalog subdirectories for catalog, domains, and teams:

.
└── .cortex
    ├── catalog
    │   ├── database.yml
    │   ├── s3-bucket.yml
    │   ├── auth-service.yml
    │   └── billing-service.yml
    ├── domains
    │   ├── billing-domain.yml
    │   └── health-domain.yml
    ├── teams
    │   ├── eng-team.yml
    │   └── sre-team.yml

Note the following:

Switching from UI-based workflows to GitOps​

Cortex's git integrations support automatic parsing of the entity descriptor file, enabling users to switch to GitOps in under five minutes.

If you plan to use a GitOps workflow, we recommend switching to the GitOps model after Cortex has been broadly rolled out to your organization. Ideally, developers will have the opportunity to experiment with Cortex through the UI, and leaders will set a threshold for a GitOps cutover.

Since you can enable GitOps per entity type, you can choose to use a GitOps workflow for only some entity types.

Using both UI and GitOps​

It is possible to enable UI editing and disable UI importing for any entity. While this would allow users to create new entities via GitOps, they must make changes to the entity through the UI. Any changes made via GitOps would not register in Cortex.

Getting started with GitOps​

GitOps considerations​

Before getting started, note the following:

  • Cortex will only check for files in the repository's default branch, unless otherwise specified. Cortex defaults to main if there is no default branch defined.
  • Cortex does not delete Scorecards if a corresponding Scorecard YAML is deleted. You can enable automatic archival of entities through GitOps by toggling on "Enable auto archiving of services" in the Entities settings page. Read more about auto-archival in the docs.
  • Domain, team, and Scorecard definitions must be in the .cortex/domains, .cortex/teams, or .cortex/scorecards folders, respectively.
  • You can define any number of entities within the same repository.
  • The recommended placement for entity descriptor files is in the root of the repository or in the appropriate .cortex/catalog folder.
    • For GitHub or GitLab, the descriptor can be located anywhere in the repository as long as the file is named cortex.yaml or cortex.yml.
    • For Bitbucket, Bitbucket Server, or Azure DevOps, you MUST place descriptor files in the appropriate .cortex/ subdirectory. It is possible to work around this for unique cases.

Step 1: Disable UI editing​

When following a GitOps approach, you make changes to entities via their entity descriptor file and sync the changes using a Cortex git integration or programmatically using the Cortex API. You must disable UI editing to ensure consistency. If the UI editor is enabled, then changes made via git will not be processed in Cortex.

Confirm that the Cortex UI editor is disabled for each entity type you want to use a GitOps approach for:

  1. Navigate to the GitOps page in Entities settings.
  2. Disable the toggles for UI editing next to services, domains, teams, and other entity types.

Step 2: Configure a git integration​

Before you can move to a GitOps approach, Cortex must be integrated with GitHub, GitLab, Azure DevOps, or Bitbucket.

See the tabs below for instructions on each provider.

Azure DevOps​

  1. Follow the instructions to integrate Cortex with Azure DevOps.
  2. Add a webhook for Azure DevOps:
    1. Navigate to Settings > Azure DevOps and validate your Azure DevOps integration.
    2. Click Create a new webhook and copy the unique webhook URL.
    3. Follow the instructions from Azure on adding a webhook.
      • Set the event type to Code pushed and use the URL from the previous step.

Once GitOps is enabled, Cortex will detect push events in your repositories. For the first webhook for a given repository, Cortex looks for the cortex.yaml files and processes them. For subsequent webhooks, Cortex only processes files with a change in the webhook event. A maximum of 3,000 changed files will be reported per commit.

Additional configuration​

Cortex's out-of-the-box GitOps configuration suits most common use cases, but you may have a scenario that requires additional configuration, such as a monorepo in Bitbucket, a repository with different projects in multiple branches, a need to restrict which repositories to import from, and more. See Additional configuration options for GitOps below for more information.

Multi-account configuration​

It is possible to configure multiple account integrations with Cortex for each of the git integrations. If you're creating or editing a cortex.yaml in the non-default configuration, you must reference the alias you used for that integration when you configured it.

For example, if you added a second GitHub configuration called non-default-example, you would define the following block in the entity descriptor:

x-cortex-git:
  repository: organization/non-default-example-repo
  alias: non-default-example
caution

If you do not define the alias, Cortex will use the default configuration when processing changes made to the cortex.yaml file. If a repository is not included in the default configuration, changes will not be processed via GitOps.

Step 3: Get started with managing entities​

After completing your GitOps configuration, you can start managing entities via GitOps.

Before following these steps:

  • We recommend reviewing the Entities documentation to understand the basics about working with entities in Cortex.

Create an entity​

To create a new entity from a new repository:

  1. Create a repository in your git account.
  2. Create a new cortex.yaml or cortex.yml file in the repo you just created, including the entity's details.
  3. Commit and push your changes.

The entity will now exist in your repository and in the Cortex UI.

For example:

title: Example Entity
x-cortex-git:
  github:
    repository: example-repo
x-cortex-tag: example-entity
x-cortex-type: service
x-cortex-owners:
- name: docs-team
  type: GROUP
x-cortex-custom-metadata:
  test: 123
tip

The GitHub app has a built-in linter, so if an entity descriptor file is invalid, the GitHub app will comment on the pull request with outstanding issues.

Verify that your entity was created​

To verify that the new entity was created:

  • View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity creation was successful, you should expect to see it at the top of the list.
    • You must have the View GitOps logs permission to view this page.
  • Search the Catalogs > All entities page for the new entity's name or tag. If your creation was successful, it will appear in the search results.
caution

If you do not see your changes, refresh the browser window where you are logged in to Cortex.

Edit an entity​

Any changes you commit to an entity will appear under Recent activity on the entity page overview.

  1. Navigate to the YAML file for an entity.
  2. Add data to the entity
  3. Commit and push your changes.

The edits now exist in your repository and on the entity in the Cortex UI. On the entity details page, the update will appear under Recent activity.

caution

If UI editing is enabled for an entity type, any changes you commit to that entity will not be reflected in Cortex.

Verify that an entity was edited​

To verify that an entity was edited:

  • View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity edit was successful, you should expect to see it at the top of the list.
    • You must have the View GitOps logs permission to view this page.
  • Search the Catalogs > All entities page for the entity's name or tag. If your edit was successful, it will appear in the search results.

If UI editing is enabled and UI importing is disabled, changes will display in GitOps logs and under an entity's recent activity, but those changes will not process. The Entities column in GitOps logs will show 0 entities if this is the case. When you open the commit, the panel will show No changes processed, and the cortex.yaml file will appear under Omitted files.

Additional configuration options for GitOps​

Cortex's standard GitOps configuration suits most common use cases:

  • Single or many projects per repo
  • Only one branch needs to be processed for cortex.yaml
  • The cortex.yaml file is in the default or main branch

However, there may be scenarios that require special setups, such as the following examples:

  • Monorepos in Bitbucket: Multiple projects in a single repo, split into subfolders.
  • Branches: Non-main or non-default branches, or different projects in multiple branches.

See the sections below for more information about special configurations.

Restrict which repositories to import from​

By default, Cortex will check all repositories for services, domains, teams, and other entity types. It is possible to restrict which repositories entities are imported from:

  1. Navigate to the Entities settings page and click the GitOps tab.
  2. Under Options by entity type, find the dropdown labeled Entity GitOps repository allowlist for new entity types.
  3. Select the repositories you want to import from.

When one or more repositories are selected for a given entity type, Cortex will only check those repos for changes to the cortex.yaml file.

caution

If you make changes in a repo that is not designated on the allowlist, Cortex will not process those changes.

Using GitOps for a monorepo in Bitbucket​

Using GitOps with a monorepo - one Git repository with multiple entities - is supported out-of-the-box for Azure DevOps, GitHub, and GitLab.

When using a monorepo with Bitbucket, you must use the cortex-properties.yaml file:

  • The file is automatically processed, just like the entity descriptor.
  • It should live in the default branch for the repo, regardless of which branches it states Cortex should use to find entity descriptor files.

Note that this file should only be used for custom workflows, such as using monorepos in Bitbucket or using a non-default branch as the home for your entity descriptor files.

Using GitOps in a non-default branch​

You can configure Cortex to automatically process cortex.yaml files in non-standard branches, multiple branches, or both.

Consider the following example scenario:

  • You have a project where main is protected and is the default branch.
  • You want to include cortex.yaml in the develop branch.
  • You also have a separate project version in a staging branch with its own cortex.yaml file.

To represent this:

  • Add a cortex-properties.yaml file in the default branch of your repo
  • Define the branches field with a list of branches to process.
    • The default branch must be explicitly defined if you are using an advanced configuration and you want Cortex to search for a cortex.yaml file in the default branch.
branches:
  - main
  - develop
  - staging

If your cortex-properties.yaml file does not contain a branches field, Cortex will continue to process the default branch.

caution

When using the branches field in your cortex-properties.yaml file, make sure to include the default branch if you want Cortex to continue looking for a cortex.yaml file in the default branch.

Using multiple source directories​

When using Azure DevOps for GitOps, you can configure Cortex to look for cortex.yaml files in multiple subdirectories.

Consider the following example scenario:

  • You have a monorepo structure where all projects live in a single repository.
  • Each project lives in a subdirectory in the main repository (project1/, project2/, etc.).
  • Each project has its own cortex.yaml file.

To represent this:

  • Add a src-dirs field in a cortex-properties.yaml file at the root of the repository, containing a list of directories to process.
src-dirs:
  - project1
  - project2

Cortex will still process any cortex.yaml file found in the root of the repository.

FAQs and troubleshooting​

Conflicts between UI editing, GitOps, and the Cortex API​

If GitOps has previously been enabled, but UI editing is temporarily turned on, any changes made in Cortex to applicable entities will not be reflected in Git. When the file is next changed through your Git provider, it will override changes made in the UI.

The last received change in a cortex.yaml file will override previous changes, whether it originated from the create/update entity API or a push from your Git provider. Changes are not appended and the last submitted entire file takes precedence, so fields omitted in cortex.yaml will be removed.

Will my cortex.yaml file be picked up immediately?​

If you already have a cortex.yaml file when you set up GitOps, Cortex will automatically process it. However, the file will not be processed until UI editing is disabled.

The entity I created appears in GitOps logs, but displays 0 entities 0 scorecards in the Entities column.​

First, use the YAML linter to validate your cortex.yaml file. Then, confirm GitOps settings are configured correctly:

  • Make sure UI editing is disabled for the entity type that you're trying to create.
  • Check repositories in the GitOps repository allowlist. If there are repositories selected for the entity type you're working with, confirm that you're working from an allowed repo.