Get a DemoCortex AcademyWebsiteStatusLogin

Using GitOps for Cortex

As an alternative to working through the Cortex UI, Cortex supports following a GitOps approach. Using a GitOps model - which entails using descriptor files to manage entities, Scorecards, and Workflows in Cortex through your version control system - provides some benefits:

  • Metadata are version-controlled

  • The repository where your code lives is also the source of truth for information

  • You always own the data

  • You can easily monitor and track GitOps changes via GitOps logs

    • Users must have the View GitOps logs permission.

Looking to dive deeper? Check out the Cortex Academy course on using GitOps, available to all Cortex customers and POVs.

How GitOps works in Cortex

When following the GitOps model in Cortex, you manage entities via entity descriptor files that live in your Git repository. An entity descriptor, also referred to as a Cortex YAML, describes an entity in your catalogs. You may store all of your entity definitions in a single repository or you can store the YAML in the repository of the corresponding entity.

Cortex checks for cortex.yaml or cortex.yml anywhere in the default branch (for Bitbucket, GitHub, and GitLab) and processes any changes pushed to the default branch.

If you use a single repository structure, Cortex checks for YAML files in the .cortex directory.

Cortex recommends storing the cortex.yaml in the repository root or basePath.

If an entity's file is deleted from the repository and you have enabled GitOps-based auto-archival, then the corresponding entity in Cortex will be archived. This feature checks for deleted files, so if the file is moved, it will not be archived from Cortex. See Auto archiving entities for more information.

GitOps repository structure

There are two options for structuring your entity descriptors in your Git repository:

  • Option 1: Each repository contains a cortex.yaml file, stored anywhere in the default branch.

    • This approach is commonly used for a one-to-one mapping where the repository represents its Cortex entity.

    • This configuration is supported for Bitbucket, GitHub, and GitLab.

    • For Azure DevOps, the cortex.yaml file must be stored at the root of the repository of the default branch. It is possible to work around this for unique cases.

  • Option 2: All entity descriptor YAMLs are stored in a single repository, under a .cortex directory.

    • Entity descriptors would be stored under the subdirectories catalog, domains, and teams.

    • This configuration is supported for all Git providers.

The following example shows a single repository structure. Entity descriptor files are listed under catalog subdirectories for catalog, domains, and teams:

.
└── .cortex
    ├── catalog
    │   ├── database.yml
    │   ├── s3-bucket.yml
    │   ├── auth-service.yml
    │   └── my-custom-entity.yml
    ├── domains
    │   ├── billing-domain.yml
    │   └── health-domain.yml
    ├── teams
    │   ├── eng-team.yml
    │   └── sre-team.yml

Note the following:

  • You cannot create catalogs via GitOps; catalogs can only be defined in the Cortex UI.

  • The catalog subdirectory contains the YAML files for services and custom entities, regardless of which catalogs the services and custom entities are organized into.

    • For example, you might have created catalogs for API and Library, and those catalogs might contain services or custom entities. In your Git repository, those entities will appear in the catalog subdirectory.

  • The YAML files cannot be in sub-directories nested under catalog, domains and teams.

Switching from UI-based workflows to GitOps

Cortex's Git integrations support automatic parsing of the entity descriptor file, enabling users to switch to GitOps in under five minutes.

If you plan to use a GitOps workflow, we recommend switching to the GitOps model after Cortex has been broadly rolled out to your organization. Ideally, developers will have the opportunity to experiment with Cortex through the UI, and leaders will set a threshold for a GitOps cutover.

Since you can enable GitOps per entity type, you can choose to use a GitOps workflow for only some entity types.

Using both UI and GitOps

It is possible to enable UI editing and disable UI importing for any entity. While this would allow users to create new entities via GitOps, they must make changes to the entity through the UI. Any changes made via GitOps would not register in Cortex.

Getting started with GitOps

GitOps considerations

Before getting started, note the following:

  • Cortex will only check for files in the repository's default branch, unless otherwise specified. Cortex defaults to main if there is no default branch defined.

  • Cortex does not delete Scorecards if a corresponding Scorecard YAML is deleted. You can enable automatic archival of entities through GitOps by toggling on "Enable auto archiving of services" in the Entities settings page. Read more in the auto-archival docs.

  • Domain, team, and Scorecard definitions must be in the .cortex/domains, .cortex/teams, and .cortex/scorecards folders, respectively.

  • The hierarchy of entities in Cortex is based on that hierarchy being defined in the entity's YAML file; Cortex does not set hierarchies or entity relationships based on a YAML file's location in your repository.

  • You can define any number of entities within the same repository.

  • GitOps is not designed for high-volume or bulk updates. We recommend keeping batch sizes under 1,000 updates per hour to avoid hitting your Git provider's rate limits. For large-scale entity updates, we recommend using the Cortex API.

  • The recommended placement for entity descriptor files is in the root of the repository, or in the appropriate .cortex/catalog folder.

    • For Bitbucket, Bitbucket Server, GitHub, or GitLab, the descriptor can be located anywhere in the repository as long as the file is named cortex.yaml or cortex.yml. You can also use a single repository and place descriptor files in the appropriate .cortex subdirectory.

    • For Azure DevOps, the cortex.yaml file must be stored at the root of the repository of the default branch. It is possible to work around this for unique cases. You can also use a single repository and place descriptor files in the appropriate .cortex subdirectory.

    • When using a single repository structure, as described in the example earlier on this page, the .cortex subdirectory only respects catalog, scorecards, domains, and teams subdirectories. Do not place an entity's YAML file in the .cortex directory unless it is in one of the supported subdirectories.

For additional information on using GitOps to manage Scorecards and Workflows, see Scorecards as code and Workflows as code.

Step 1: Disable UI editing

When following a GitOps approach, you make changes to entities via their entity descriptor file and sync the changes using a Cortex git integration or programmatically using the Cortex API. You must disable UI editing to ensure consistency. If the UI editor is enabled, then changes made via git will not be processed in Cortex.

Confirm that the Cortex UI editor is disabled for each entity type you want to use a GitOps approach for:

  1. Navigate to the GitOps page in Settings.

  2. Disable the toggles for UI editing next to services, domains, teams, and other entity types.

To learn about other GitOps settings, see the GitOps settings documentation.

Step 2: Configure a Git integration

Before you can move to a GitOps approach, Cortex must be integrated with GitHub, GitLab, Azure DevOps, or Bitbucket.

See the tabs below for instructions on each provider.

Azure DevOps

  1. Follow the instructions to integrate Cortex with Azure DevOps.

  2. Add a webhook for Azure DevOps:

    1. In Cortex, navigate to Settings > Azure DevOps and validate your Azure DevOps integration.

    2. Click Create a new webhook and copy the unique webhook URL.

    3. Follow the instructions from Azure on adding a webhook.

      • Set the event type to Code pushed and use the URL from the previous step.

Once GitOps is enabled, Cortex will detect push events in your repositories. For the first webhook for a given repository, Cortex looks for the cortex.yaml files and processes them. For subsequent webhooks, Cortex only processes files with a change in the webhook event. A maximum of 3,000 changed files will be reported per commit.

Additional configuration

Cortex's out-of-the-box GitOps configuration suits most common use cases, but you may have a scenario that requires additional configuration, such as a monorepo in Bitbucket, a repository with different projects in multiple branches, a need to restrict which repositories to import from, and more. See Additional configuration options for GitOps below for more information.

Multi-account configuration

It is possible to configure multiple account integrations with Cortex for each of the git integrations. If you're creating or editing a cortex.yaml in the non-default configuration, you must reference the alias you used for that integration when you configured it.

For example, if you added a second GitHub configuration called non-default-example, you would define the following block in the entity descriptor:

x-cortex-git:
  repository: organization/non-default-example-repo
  alias: non-default-example

If you do not define the alias, Cortex will use the default configuration when processing changes made to the cortex.yaml file. If a repository is not included in the default configuration, changes will not be processed via GitOps.

Step 3: Get started with managing entities

After completing your GitOps configuration, you can start managing entities via GitOps.

Before following these steps:

  • We recommend reviewing the Entities documentation to understand the basics about working with entities in Cortex.

Create an entity

Create an entity

To create a new entity from a new repository:

  1. Create a repository in your git account.

  2. Create a new cortex.yaml or cortex.yml file in the repo you just created, including the entity's details.

  3. Commit and push your changes.

The entity will now exist in your repository and in the Cortex UI.

For example:

title: Example Entity
x-cortex-git:
  github:
    repository: example-repo
x-cortex-tag: example-entity
x-cortex-type: service
x-cortex-owners:
- name: docs-team
  type: GROUP
x-cortex-custom-metadata:
  test: 123

The GitHub app has a built-in linter, so if an entity descriptor file is invalid, the GitHub app will comment on the pull request with outstanding issues.

Learn more about creating entity YAMLs in Defining entities via YAML file.

Verify that your entity was created

To verify that the new entity was created:

  • View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity creation was successful, you should expect to see it at the top of the list.

    • You must have the View GitOps logs permission to view this page.

  • Search the Catalogs > All entities page for the new entity's name or tag. If your creation was successful, it will appear in the search results.

If you do not see your changes, refresh the browser window where you are logged in to Cortex.

Edit an entity

Edit an entity

Any changes you commit to an entity will appear under Recent activity on the entity page overview.

  1. Navigate to the YAML file for an entity.

  2. Add data to the entity

  3. Commit and push your changes.

The edits now exist in your repository and on the entity in the Cortex UI. On the entity details page, the update will appear under Recent activity.

If UI editing is enabled for an entity type, any changes you commit to that entity will not be reflected in Cortex.

Verify that an entity was edited

To verify that an entity was edited:

  • View the GitOps logs page in Cortex, which displays all changes made in your Cortex workspace. If your entity edit was successful, you should expect to see it at the top of the list.

    • You must have the View GitOps logs permission to view this page.

  • Search the Catalogs > All entities page for the entity's name or tag. If your edit was successful, it will appear in the search results.

If UI editing is enabled and UI importing is disabled, changes will display in GitOps logs and under an entity's recent activity, but those changes will not process. The Entities column in GitOps logs will show 0 entities if this is the case. When you open the commit, the panel will show No changes processed, and the cortex.yaml file will appear under Omitted files.

In the Cortex UI on an entity details page, entities created or updated via GitOps will display the file path of the entity's YAML file and a preview of the last GitOps log. You must have the View GitOps logs permission.

Additional configuration options for GitOps

Cortex's standard GitOps configuration suits most common use cases:

  • Single or many projects per repo

  • Only one branch needs to be processed for cortex.yaml

  • The cortex.yaml file is in the default or main branch

However, there may be scenarios that require special setups, such as the following examples:

  • Monorepos in Bitbucket: Multiple projects in a single repo, split into subfolders.

  • Branches: Non-main or non-default branches, or different projects in multiple branches.

See the sections below for more information about special configurations.

Restrict which repositories to import from

By default, Cortex will check all repositories for services, domains, teams, and other entity types. It is possible to restrict which repositories entities are imported from:

  1. Navigate to the Settings > GitOps.

  2. Under Options by entity type, find the dropdown labeled Entity GitOps repository allowlist for new entity types.

  3. Select the repositories you want to import from.

When one or more repositories are selected for a given entity type, Cortex will only check those repos for changes to the cortex.yaml file.

If you make changes in a repo that is not designated on the allowlist, Cortex will not process those changes.

Using GitOps for a monorepo in Bitbucket

Using GitOps with a monorepo - one Git repository with multiple entities - is supported out-of-the-box for Azure DevOps, GitHub, and GitLab when you use the basepath field to specify the subdirectory in the entity descriptor (see an example of this in the docs for Azure DevOps, GitHub, and GitLab).

When using a monorepo with Bitbucket, you must use the cortex-properties.yaml file:

  • The file is automatically processed, just like the entity descriptor.

  • It should live in the default branch for the repo, regardless of which branches it states Cortex should use to find entity descriptor files.

  • When using the cortex-properties.yaml file, the basePath may not function as expected if you have service code elsewhere in your repository.

Note that this file should only be used for custom workflows, such as using monorepos in Bitbucket or using a non-default branch as the home for your entity descriptor files.

Using GitOps in a non-default branch

You can configure Cortex to automatically process cortex.yaml files in non-standard branches, multiple branches, or both.

Consider the following example scenario:

  • You have a project where main is protected and is the default branch.

  • You want to include cortex.yaml in the develop branch.

  • You also have a separate project version in a staging branch with its own cortex.yaml file.

To represent this:

  • Add a cortex-properties.yaml file in the default branch of your repo

  • Define the branches field with a list of branches to process.

    • The default branch must be explicitly defined if you are using an advanced configuration and you want Cortex to search for a cortex.yaml file in the default branch.

branches:
  - main
  - develop
  - staging

If your cortex-properties.yaml file does not contain a branches field, Cortex will continue to process the default branch.

Cortex will process the most recently modified cortex.yaml file across all branches listed in the cortex-properties.yaml file.

When using the branches field in your cortex-properties.yaml file, make sure to include the default branch if you want Cortex to continue looking for a cortex.yaml file in the default branch.

Using multiple source directories

When using Azure DevOps for GitOps, you can configure Cortex to look for cortex.yaml files in multiple subdirectories.

Consider the following example scenario:

  • You have a monorepo structure where all projects live in a single repository.

  • Each project lives in a subdirectory in the main repository (project1/, project2/, etc.).

  • Each project has its own cortex.yaml file.

To represent this:

  • Add a src-dirs field in a cortex-properties.yaml file at the root of the repository, containing a list of directories to process.

src-dirs:
  - project1
  - project2

Cortex will still process any cortex.yaml file found in the root of the repository.

FAQs and troubleshooting

Conflicts between UI editing, GitOps, and the Cortex API

If GitOps has previously been enabled, but UI editing is temporarily turned on, any changes made in Cortex to applicable entities will not be reflected in Git. When the file is next changed through your Git provider, it will override changes made in the UI.

The last received change in a cortex.yaml file will override previous changes, whether it originated from the create/update entity API or a push from your Git provider. Changes are not appended and the last submitted entire file takes precedence, so fields omitted in cortex.yaml will be removed.

Will my cortex.yaml file be picked up immediately?

If you already have a cortex.yaml file when you set up GitOps, Cortex will automatically process it. However, the file will not be processed until UI editing is disabled.

The entity I created appears in GitOps logs, but displays 0 entities 0 scorecards in the Entities column.

First, use the YAML linter to validate your cortex.yaml file. Then, confirm GitOps settings are configured correctly:

  • Make sure UI editing is disabled for the entity type that you're trying to create.

  • Check repositories in the GitOps repository allowlist. If there are repositories selected for the entity type you're working with, confirm that you're working from an allowed repo.

Last updated

Was this helpful?