CQL v2.0

CQL (Cortex Query Language) is at the core of virtually everything Cortex provides, from defining how Scorecards evaluate health and readiness to deciding which entities a plugin should show up on.

Since its initial adoption, Cortex has made improvements to CQL by introducing greater consistency and improved tooling.

In this guide, we'll look at those improvements and cover breaking changes for users who built Scorecards, reports, and queries with old versions of CQL. You can use the CQL explorer to see all supported expressions in CQL v2.0.

Improvements and new functionality

Archived entities

You can check for archived entities using entity.isArchived().

Cortex also supports isArchived for dependency, children, and parent objects so that you can check whether an entity has any archived dependencies, children, or parents.

Example: You can check whether an entity has archived dependencies using dependencies.out(includeArchived = true).any((dependency) => dependency.isArchived == true).

Counting elements

The .count method has been replaced with .length.

Lists that previously returned numbers now use the numOf prefix.

Lists that did not return objects but required .count now use the numOf prefix.

Examples:

• sentry.issues is now sentry.numOfIssues()
• openapi.paths.count is now openapi.numOfPaths()

Creation and update timestamps

You can get creation and last updated timestamps using entity.created(asTimestamp = true) and entity.lastUpdated(asTimestamp = true).

Without the asTimestamp parameter, the result will return a duration.

git.branchProtection

Cortex expanded git.branchProtection to support disableOverridingApproversPerMergeRequest, mergeRequestsAuthorApproval, requiredStatusChecks, annd other features in GitLab Ultimate.

Map support

k8s.metadata.has_label and its matching function have been replaced with k8s.metadata.labels, which supports a list of labels and Map<String, Any>.

Naming conventions

Transitioned from snake_case to camelCase for consistency and reliability.

Example: has_group is now hasGroup

Objects

The entity object has been introduced, consolidating attributes and methods like entity_created and entity_descriptor into a more coherent structure.

The ownership attributes have been refined into a streamlined owners object, providing a list of members along with the source of ownership.

The slosPassing attribute has been updated to slos with a new field added to indicate whether the SLOs are passing.

dashboards has been updated to embeds().

documentation and runbooks have been removed and should be accessed through links("documentation") and links("runbooks").

PagerDuty incidents

You can now measure information about incidents from Pagerduty.

Pull requests

There is now added support for pull requests from GitHub and merge requests from GitLab so you can check attributes about PRs/MRs in Scorecards.

Service calls

Service calls must be enclosed in parentheses.

Example: entity.tag() and entity.description()

SonarQube metrics

Metrics from SonarQube have been expanded to innclude security_hotspots, security_rating, and more.

Strings

Strings support .split().

Team entities

Team entities support entity.children() and entity.parents() so you can validate that your team hierarchy is set up as expected.

k8s.clusters()

The k8s integration supports k8s.clusters() so you can access all clusters and write expressions against them.

Breaking changes in v2.0

checkmarx.number_of_vulnerabilities

Update checkmarx.number_of_vulnerabilities to checkmarx.numOfVulnerabilities().

.count

Replace all instances of .count with .length.

dashboards

Update dashboards to embeds().

Documentation and runbooks

Replace documentation and runbooks in favor of links(”documentation”) and links(”runbooks”).

entity

Use the entity object instead of:

• entity_children
• entity_created
• entity_description
• entity_descriptor
• entity_last_updated
• entity_name
• entity_parents
• entity_type

Entity object

interface Entity {
    created: 'Duration',
    descriptor: 'Object',
    definition: 'Object',
    description: 'Text',
    isArchived: 'Boolean',
    lastUpdated: 'Duration',
    name: 'Text',
    type: 'Text',
    tag: 'Text',
    // only supported for domains & teams
    parents: 'List<HierarchyParentNode>',
    children: 'List<HierarchyChildNode>'
}

has_service_group

Replace has_service_group with has_group.

k8s.metadata.has_label and k8s.metadata.has_label_matching

Update k8s.metadata.has_label and k8s.metadata.has_label_matching to just k8s.metadata.labels, which supports a list of labels and Map<String, Any>.

Links

Remove title_matching and url_matching support for links in favor of map functions.

Example: links("documentation").title_matching("prod*").count > 0 is now links("logs").filter((link) => link.name.matches("prod*")).length > 0.

Lists with .length

Migrate false lists with one .length member to numOf.

matches and matchesIn

Use matches to check if the entire input string matches the regular expression.

Use matchesIn to find all occurrences of the regular expression pattern within the input string.

newrelic

Update newrelic to newrelic.applications.

While entity descriptors support multiple New Relic applications, New Relic only returns one application so CQL only returns the first app specified.

New Relic application object

interface NewRelicApplication {
    apdexScore: 'Number | Null',
    apdexTarget: 'Number | Null',
    concurrentstanceCount: 'Number | Null',
    errorRate: 'Number | Null',
    hostCount: 'Number | Null',
    instanceCount: 'Number | Null',
    name: 'Text',
    responseTime: 'Number | Null',
    throughput: 'Number | Null',
}

Ownership

Remove owner_groups, owners_is_set, and owner_descriptions in favor of a new ownership object that includes a list of members. Each member has a source.

Example: owner_groups.length > 0 is now owners.teams().length > 0.

Ownership object

interface Ownership {
    teams: List<Team> // Cortex team entities
    allOwners: List<Member>

    interface Team {
        tag: String
        name: String
        description: String | Null
        members: List<Member> // all members, including from idp groups
        identityProviderGroups: List<IDPGroup> // know if the team was sourced from idpgroup
        isFallback: boolean // how did this entity get this owner
        isInherited: boolean,
        isArchived: boolean
    }

    interface Member {
        name: Text,
        email: Text | Null,
        description: Text | Null,
        roles: List<TeamMemberRole>
        // if empty it was manually defined in the yaml
        identityProviderSources: List<IdentityProviderGroup>
        notificationsEnabled: boolean
        // does this user have a Cortex user mapping
        hasCortexUser: boolean
    }

    interface IdentityProviderGroup {
        groupIdentifier: Text // Example my-okta-team-thing
        source: OKTA | WORKDAY | etc;
    }
}

SemVer

Custom data expressions must be manually cast to semver.

Example: custom("key") > semver("1.0.0") is now semver(custom("key")) > semver("1.0.0").string.

Service calls

Use () when making a service call.

slosPassing

Updated slosPassing to slos with a new field for passing.

snake_case

Replace all instances of snake_case with camelCase.

Migrating to CQL v2.0

Automatic migration

To make the migration as easy for users as possible, Cortex will automatically migrate some queries to v2.0. This is only possible if the Cortex UI editor is enabled and GitOps is disabled.

Admins can also find a complete list of outdated expressions and their source on the Outdated CQL expressions page in settings.

caution

Cortex does not support auto-migration if the GitOps service configuration is enabled.

Migrating Scorecard rules

When you open a Scorecard with outdated rules, you'll see a banner at the top of the page that says "One or more rules in this Scorecard as well as the entity filter have outdated CQL expressions. Please upgrade expressions to CQL version 2."

There will be a yellow dot next to the Rules tab when the outdated expression is a rule (rather than a filter). Yellow dots will appear next to the specific rules that use outdated expressions.

caution

Only users who have permissions to edit Scorecards will have this view and the ability to migrate rules.

Select Edit at the top of the page to update the outdated expressions. The yellow dots will also appear within the edit flow so you can easily identify them.

Open the edit modal for an outdated rule and select CQL Editor. Below the expression you'll see This CQL expression is outdated and a button to Update to CQL v2.

When you click Update to CQL v2, Cortex will automatically convert the expression for you. The message and the button will then disappear.

If you have outdated filters, you'll see the same message and button under the Query field.

Once all rules and filters are updated, the banner at the top of the Scorecard's details page will disappear.

Migrating CQL reports and Query builder queries

The process for migrating CQL reports and Query builder queries is the same as with Scorecards.

CQL reports will automatically migrate outdated expressions during evaluation. When you open a CQL report with outdated expressions, you’ll see the same alert and yellow dots. Just like with Scorecards, the banner will also display if a CQL filter is using a 1.0 expression.

The yellow dots will appear next to Saved Queries in the Query builder that use outdated expressions.

Use the edit modal to update expressions in both CQL reports and the Query builder - open the CQL form editor and click Update to CQL v2.

The Refresh Query function will also be disabled for saved queries until the expression has been updated.

outdated expressions; you’ll need to update the expression for this feature to work.

GitOps workaround

Cortex users who have GitOps enabled can facilitate auto-migration with the following workaround:

1. Toggle on Enable Scorecard UI editor in Scorecards settings.
2. Re-evaluate Scorecards with outdated expressions by clicking Evaluate now on a given Scorecard's page.
3. Export the YAML by clicking ⋮ and selecting Export Scorecard.
4. Toggle off Enable Scorecard UI editor in Scorecards settings, thus re-enabling GitOps.
5. Push your YAML changes in the corresponding YAML file(s).

Using the Cortex command line interface to facilitate the migration

The Cortex command line interface can be help with this migration. If you use the CLI, you may want to follow the steps below to update your Scorecards.

The links in these lists point to recipes in the CLI documentation that you can follow to automate these tasks.

If you are using GitOps to manage Scorecards:

1. Disable GitOps for Scorecards.
2. Create copies of all Scorecards as drafts.
3. Convert any CQL rules that weren't auto-converted to CQL v2 following the recommendations in the Migrating Scorecard Rules section earlier in this document.
4. For each Scorecard, compare results of the existing Scorecard and draft Scorecard.
5. Re-enable GitOps for Scorecards.
6. Export draft Scorecards, change draft to false and commit in Git.
7. Delete draft Scorecards.

If you are NOT using GitOps to manage Scorecards:

1. Make a backup of all Scorecards.
2. Create copies of all Scorecards as drafts.
3. Convert any CQL rules that weren't auto-converted to CQL v2 following the recommendations in the section above.
4. For each Scorecard, compare results of the existing Scorecard and draft Scorecard.
5. Export draft Scorecards, change draft to false and replace existing Scorecards.

note

You do not need to follow all of these instructions.

For example, you might just want to make a backup of your Scorecards and edit the existing ones, using the backups just in case you don’t get the results you expect.

You may want to follow all of these steps if it’s critical for existing Scorecards to function without any interruptions.

Teting new CQL v2.0 expressions

There are three quick ways to ensure that your expressions have been properly migrated before publishing to all users:

1. Create a copy of an existing Scorecard in draft mode. Manually update the necessary expressions and verify that they match the results of the existing Scorecard.
2. Input the new expression in the Query builder and run it for an entity for which you know the expression is true. Ensure that it shows that entity in the results.
3. Create a CQL report filtered to a specific entity with the v2.0 expressions as columns and check that they provide the expected output.

Still need help?

The following are all the ways to get assistance from our customer engineering team. Please use the option that is best for your users:

• Email: help@cortex.io, or open a support ticket in the in app Resource Center
• Chat: Available in the Resource Center
• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your customer success manager.