Using GitOps for Cortex

As an alternative to working through the Cortex UI, Cortex supports following a GitOps approach. Using a GitOps model - which entails using descriptor files to manage entities, Scorecards, and Workflows in Cortex through your version control system - provides some benefits:

• Metadata are version-controlled

• The repository where your code lives is also the source of truth for information

• You always own the data

• You can easily monitor and track GitOps changes via GitOps logs

  • Users must have the View GitOps logs permission.

How GitOps works in Cortex

When following the GitOps model in Cortex, you manage entities via entity descriptor files that live in your git repository. An entity descriptor, also referred to as a Cortex YAML, describes an entity in your catalogs. You may store all of your entity definitions in a single repository or you can store the YAML in the repository of the corresponding entity.

Cortex checks for cortex.yaml or cortex.yml anywhere in the default branch (for Bitbucket, GitHub, and GitLab) and processes any changes pushed to the default branch. For Azure, Cortex checks the .cortex directory.

Cortex recommends storing the cortex.yaml in the repository root or basePath.

If an entity's file is deleted from the repository and you have enabled GitOps-based auto-archival, then the corresponding entity in Cortex will be archived. This feature checks for deleted files, so if the file is moved, it will not be archived from Cortex. See Auto archiving entities for more information.

GitOps example repository structure

The following example shows a single repository structure. Entity descriptor files are listed under catalog subdirectories for catalog, domains, and teams:

.
└── .cortex
    ├── catalog
    │   ├── database.yml
    │   ├── s3-bucket.yml
    │   ├── auth-service.yml
    │   └── billing-service.yml
    ├── domains
    │   ├── billing-domain.yml
    │   └── health-domain.yml
    ├── teams
    │   ├── eng-team.yml
    │   └── sre-team.yml

Note the following:

• You cannot create catalogs via GitOps; catalogs can only be defined in the Cortex UI.

• The YAML files cannot be in sub-directories under catalog, domains and teams.

Switching from UI-based workflows to GitOps

Cortex's git integrations support automatic parsing of the entity descriptor file, enabling users to switch to GitOps in under five minutes.

If you plan to use a GitOps workflow, we recommend switching to the GitOps model after Cortex has been broadly rolled out to your organization. Ideally, developers will have the opportunity to experiment with Cortex through the UI, and leaders will set a threshold for a GitOps cutover.

Since you can enable GitOps per entity type, you can choose to use a GitOps workflow for only some entity types.

Using both UI and GitOps

It is possible to enable UI editing and disable UI importing for any entity. While this would allow users to create new entities via GitOps, they must make changes to the entity through the UI. Any changes made via GitOps would not register in Cortex.

Getting started with GitOps

GitOps considerations

Before getting started, note the following:

• Cortex will only check for files in the repository's default branch, unless otherwise specified. Cortex defaults to main if there is no default branch defined.
• Cortex does not delete Scorecards if a corresponding Scorecard YAML is deleted. You can enable automatic archival of entities through GitOps by toggling on "Enable auto archiving of services" in the Entities settings page. Read more in the auto-archival docs.
• Domain, team, and Scorecard definitions must be in the .cortex/domains, .cortex/teams, or .cortex/scorecards folders, respectively.

• The hierarchy of entities in Cortex is based on that hierarchy being defined in the entity's YAML file; Cortex does not set hierarchies or entity relationships based on a YAML file's location in your repository.

• You can define any number of entities within the same repository.

• The recommended placement for entity descriptor files is in the root of the repository, or in the appropriate .cortex/catalog folder.

  • For GitHub or GitLab, the descriptor can be located anywhere in the repository as long as the file is named cortex.yaml or cortex.yml.

  • For Bitbucket, Bitbucket Server, or Azure DevOps, you MUST place descriptor files in the appropriate .cortex/ subdirectory. It is possible to work around this for unique cases.
  • When using a single repository structure, as described in the example earlier on this page, the .cortex subdirectory only respects catalog, scorecards, domains, and teams subdirectories. Do not place the cortex.yaml file in the .cortex directory unless it is in one of the supported subdirectories.

For additional information on using GitOps to manage Scorecards and Workflows, see Scorecards as code and Workflows as code.

Step 1: Disable UI editing

When following a GitOps approach, you make changes to entities via their entity descriptor file and sync the changes using a Cortex git integration or programmatically using the Cortex API. You must disable UI editing to ensure consistency. If the UI editor is enabled, then changes made via git will not be processed in Cortex.

Confirm that the Cortex UI editor is disabled for each entity type you want to use a GitOps approach for:

1. Navigate to the GitOps page in Settings.
2. Disable the toggles for UI editing next to services, domains, teams, and other entity types.

Step 2: Configure a git integration

Before you can move to a GitOps approach, Cortex must be integrated with GitHub, GitLab, Azure DevOps, or Bitbucket.

See the tabs below for instructions on each provider.

Azure DevOps

Azure DevOps

1. Follow the instructions to integrate Cortex with Azure DevOps.
2. Add a webhook for Azure DevOps:

   1. In Cortex, navigate to Settings > Azure DevOps and validate your Azure DevOps integration.
   2. Click Create a new webhook and copy the unique webhook URL.

   3. Follow the instructions from Azure on adding a webhook.

      • Set the event type to Code pushed and use the URL from the previous step.

Bitbucket

The Bitbucket integration is pre-configured with the ability to use the GitOps approach in Cortex. If you are using Bitbucket Server, you will also need to add a webhook.

1. In Bitbucket, enable development mode.

   • You can find this setting in your Bitbucket instance at https://bitbucket.org//workspace/settings/addon-management/.

2. Follow the instructions to integrate Cortex with Bitbucket.

   • If you are not using Bitbucket Server, then your integration is complete and you are ready to use GitOps.

3. If you are using Bitbucket Server, add a webhook for Bitbucket:

   1. Navigate to Settings > Bitbucket and validate your configuration.
   2. Enter a secret token for your Bitbucket Server webhook.

   3. Click Create a new webhook and copy the unique webhook URL.

   4. Follow the Bitbucket Server instructions for adding a project-level or repository-level webhook.

      • Set the event type to repository push. Use the secret and the URL from the previous steps.

If you're using a monorepo, see the section below: Using GitOps for a monorepo in Bitbucket.

GitHub

The Cortex app for GitHub is pre-configured with the ability to use the GitOps approach in Cortex. If you configure the integration using a personal access token, you will also need to add a webhook.

1. Follow the instructions to integrate Cortex with GitHub.

   • If you integrated using the Cortex app, then your integration is complete and you are ready to use GitOps.

2. If you configured your GitHub integration using a personal access token, create a webhook:

   1. Enter a secret passphrase for your GitHub webhook in the Secret field.

   2. After saving the passphrase, a unique webhook URL is displayed.

      • The URL will end with {alias} - make sure to replace this with the alias you used when configuring your GitHub integration.

   3. Follow the instructions from GitHub on creating a webhook. Cortex recommends adding an organization webhook, but you can also define the webhook for a repository in the org.

      • Set the content type to application/json. Use the secret passphrase and the URL from the previous steps.

GitLab

1. Follow the instructions to integrate Cortex with GitLab.
2. Create a webhook:

   1. In the GitLab settings in Cortex, click Create a new token.

      • The token and the webhook URL will be displayed.

   2. In GitLab, create a system hook(recommended) or a project hook.

      • Enable Push events for the webhook. Use the token and the webhook URL from the previous steps.

Once GitOps is enabled, Cortex will detect push events in your repositories. For the first webhook for a given repository, Cortex looks for the cortex.yaml files and processes them. For subsequent webhooks, Cortex only processes files with a change in the webhook event. A maximum of 3,000 changed files will be reported per commit.

Additional configuration

Cortex's out-of-the-box GitOps configuration suits most common use cases, but you may have a scenario that requires additional configuration, such as a monorepo in Bitbucket, a repository with different projects in multiple branches, a need to restrict which repositories to import from, and more. See Additional configuration options for GitOps below for more information.

Multi-account configuration

It is possible to configure multiple account integrations with Cortex for each of the git integrations. If you're creating or editing a cortex.yaml in the non-default configuration, you must reference the alias you used for that integration when you configured it.

For example, if you added a second GitHub configuration called non-default-example, you would define the following block in the entity descriptor:

x-cortex-git:
  repository: organization/non-default-example-repo
  alias: non-default-example

If you do not define the alias, Cortex will use the default configuration when processing changes made to the cortex.yaml file. If a repository is not included in the default configuration, changes will not be processed via GitOps.

Step 3: Get started with managing entities

After completing your GitOps configuration, you can start managing entities via GitOps.

Before following these steps:

• We recommend reviewing the Entities documentation to understand the basics about working with entities in Cortex.

Create an entity

Create an entity

To create a new entity from a new repository:

1. Create a repository in your git account.

2. Create a new cortex.yaml or cortex.yml file in the repo you just created, including the entity's details.

3. Commit and push your changes.

The entity will now exist in your repository and in the Cortex UI.

For example:

title: Example Entity
x-cortex-git:
  github:
    repository: example-repo
x-cortex-tag: example-entity
x-cortex-type: service
x-cortex-owners:
- name: docs-team
  type: GROUP
x-cortex-custom-metadata:
  test: 123

The GitHub app has a built-in linter, so if an entity descriptor file is invalid, the GitHub app will comment on the pull request with outstanding issues.

Verify that your entity was created

To verify that the new entity was created:

• View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity creation was successful, you should expect to see it at the top of the list.

  • You must have the View GitOps logs permission to view this page.

• Search the Catalogs > All entities page for the new entity's name or tag. If your creation was successful, it will appear in the search results.

If you do not see your changes, refresh the browser window where you are logged in to Cortex.

Edit an entity

Edit an entity

Any changes you commit to an entity will appear under Recent activity on the entity page overview.

1. Navigate to the YAML file for an entity.

2. Add data to the entity

3. Commit and push your changes.

The edits now exist in your repository and on the entity in the Cortex UI. On the entity details page, the update will appear under Recent activity.

If UI editing is enabled for an entity type, any changes you commit to that entity will not be reflected in Cortex.

Verify that an entity was edited

To verify that an entity was edited:

• View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity edit was successful, you should expect to see it at the top of the list.

  • You must have the View GitOps logs permission to view this page.

• Search the Catalogs > All entities page for the entity's name or tag. If your edit was successful, it will appear in the search results.

If UI editing is enabled and UI importing is disabled, changes will display in GitOps logs and under an entity's recent activity, but those changes will not process. The Entities column in GitOps logs will show 0 entities if this is the case. When you open the commit, the panel will show No changes processed, and the cortex.yaml file will appear under Omitted files.

In the Cortex UI on an entity details page, entities created or updated via GitOps will display the file path of the entity's YAML file and a preview of the last GitOps log. You must have the View GitOps logs permission.

Additional configuration options for GitOps

Cortex's standard GitOps configuration suits most common use cases:

• Single or many projects per repo

• Only one branch needs to be processed for cortex.yaml

• The cortex.yaml file is in the default or main branch

However, there may be scenarios that require special setups, such as the following examples:

• Monorepos in Bitbucket: Multiple projects in a single repo, split into subfolders.

• Branches: Non-main or non-default branches, or different projects in multiple branches.

See the sections below for more information about special configurations.

Restrict which repositories to import from

By default, Cortex will check all repositories for services, domains, teams, and other entity types. It is possible to restrict which repositories entities are imported from:

1. Navigate to the Settings > GitOps.
2. Under Options by entity type, find the dropdown labeled Entity GitOps repository allowlist for new entity types.

3. Select the repositories you want to import from.

When one or more repositories are selected for a given entity type, Cortex will only check those repos for changes to the cortex.yaml file.

If you make changes in a repo that is not designated on the allowlist, Cortex will not process those changes.

Using GitOps for a monorepo in Bitbucket

Using GitOps with a monorepo - one Git repository with multiple entities - is supported out-of-the-box for Azure DevOps, GitHub, and GitLab when you use the basepath field to specify the subdirectory in the entity descriptor (see an example of this in the docs for Azure DevOps, GitHub, and GitLab).

When using a monorepo with Bitbucket, you must use the cortex-properties.yaml file:

• The file is automatically processed, just like the entity descriptor.

• It should live in the default branch for the repo, regardless of which branches it states Cortex should use to find entity descriptor files.

• When using the cortex-properties.yaml file, the basePath may not function as expected if you have service code elsewhere in your repository.

Note that this file should only be used for custom workflows, such as using monorepos in Bitbucket or using a non-default branch as the home for your entity descriptor files.

Using GitOps in a non-default branch

You can configure Cortex to automatically process cortex.yaml files in non-standard branches, multiple branches, or both.

Consider the following example scenario:

• You have a project where main is protected and is the default branch.

• You want to include cortex.yaml in the develop branch.

• You also have a separate project version in a staging branch with its own cortex.yaml file.

To represent this:

• Add a cortex-properties.yaml file in the default branch of your repo

• Define the branches field with a list of branches to process.

  • The default branch must be explicitly defined if you are using an advanced configuration and you want Cortex to search for a cortex.yaml file in the default branch.

branches:
  - main
  - develop
  - staging

If your cortex-properties.yaml file does not contain a branches field, Cortex will continue to process the default branch.

When using the branches field in your cortex-properties.yaml file, make sure to include the default branch if you want Cortex to continue looking for a cortex.yaml file in the default branch.

Using multiple source directories

When using Azure DevOps for GitOps, you can configure Cortex to look for cortex.yaml files in multiple subdirectories.

Consider the following example scenario:

• You have a monorepo structure where all projects live in a single repository.

• Each project lives in a subdirectory in the main repository (project1/, project2/, etc.).

• Each project has its own cortex.yaml file.

To represent this:

• Add a src-dirs field in a cortex-properties.yaml file at the root of the repository, containing a list of directories to process.

src-dirs:
  - project1
  - project2

Cortex will still process any cortex.yaml file found in the root of the repository.

FAQs and troubleshooting

Conflicts between UI editing, GitOps, and the Cortex API

If GitOps has previously been enabled, but UI editing is temporarily turned on, any changes made in Cortex to applicable entities will not be reflected in Git. When the file is next changed through your Git provider, it will override changes made in the UI.

The last received change in a cortex.yaml file will override previous changes, whether it originated from the create/update entity API or a push from your Git provider. Changes are not appended and the last submitted entire file takes precedence, so fields omitted in cortex.yaml will be removed.

Will my cortex.yaml file be picked up immediately?

If you already have a cortex.yaml file when you set up GitOps, Cortex will automatically process it. However, the file will not be processed until UI editing is disabled.

The entity I created appears in GitOps logs, but displays 0 entities 0 scorecards in the Entities column.

First, use the YAML linter to validate your cortex.yaml file. Then, confirm GitOps settings are configured correctly:

• Make sure UI editing is disabled for the entity type that you're trying to create.
• Check repositories in the GitOps repository allowlist. If there are repositories selected for the entity type you're working with, confirm that you're working from an allowed repo.