LogoLogo
Login to CortexBook a DemoCortex Academycortex.io
  • Cortex Docs
  • Cortex Quick Start
  • Ingesting data into Cortex
    • Managing Entities
      • Adding entities
        • Add services
        • Add domains
        • Add teams
        • Add custom entity types
        • Defining dependencies
      • Entity details page
      • Defining ownership
      • Defining relationship types
      • Grouping entities
      • Adding external documentation
      • Adding Deploy data
      • Adding custom data
      • Viewing discovered entities
      • Archiving entities
      • Relationship graph
      • Using On-call Assistant for incidents
      • Managing Terraform infra in Cortex
    • Managing Catalogs
    • Integrations
      • Internally hosted integrations
      • ArgoCD
      • AWS
      • Azure DevOps
      • Azure Resources
      • BambooHR
      • Bitbucket
      • BugSnag
      • Buildkite
      • Checkmarx
      • CircleCI
      • ClickUp
      • Codecov
      • Coralogix
      • Custom webhook integrations
      • Datadog
      • Dynatrace
      • Entra ID (Azure AD)
      • FireHydrant
      • GitHub
      • GitLab
      • Google
      • Grafana
      • incident.io
      • Instana
      • Jenkins
      • Jira
      • Kubernetes
      • LaunchDarkly
      • Lightstep
      • Mend
      • Microsoft Teams
      • New Relic
      • Okta
      • Opsgenie
      • PagerDuty
      • Prometheus
      • Rollbar
      • Rootly
      • Sentry
      • ServiceNow
      • Slack
      • Snyk
      • SonarQube
      • Splunk Observability Cloud (SignalFx)
      • Splunk On-Call (VictorOps)
      • Sumo Logic
      • Veracode
      • Wiz
      • Workday
      • xMatters
  • Scorecards
    • Initiatives and Action items
      • Creating issues based on Initiatives
    • Scorecard rule exemptions
    • Scorecard rule filters
    • Scorecard examples
    • Scorecards as code
  • Reports
    • Executive report
    • All Scorecards report
    • Bird's eye report
    • Progress report
    • Report card
  • Eng Intelligence
    • Custom Metrics
    • Jira Metrics
    • Metrics Explorer (Beta)
  • Cortex Query Language (CQL)
    • Using CQL reports
    • Using JQ in Cortex
  • Workflows
    • Creating a Workflow
      • Workflows as code
    • Blocks
    • Running a Workflow
    • Registering a Scaffolder template
      • Scaffolder advanced usage
    • Using a Workflow to sync in ArgoCD
    • Kicking off a Jenkins pipeline in a Workflow
    • Calling internal service endpoints in a Workflow
  • Plugins
    • Creating a plugin
      • Creating a plugin proxy
    • Migrating Backstage plugins to Cortex
  • Engineering homepage
  • Workspace Settings
    • Using GitOps for Cortex
      • GitOps logs
    • Managing users
      • Roles and permissions
        • Custom roles
        • Team ownership entity editing
      • Configuring SSO
        • Microsoft Entra ID
        • Google
        • Other OIDC providers
        • Okta
          • Okta SCIM
      • Configuring identity mappings
      • Onboarding management
    • API keys, secrets, and tokens
      • Secrets
      • Personal tokens
    • Audit logs
    • Entity settings
      • Data verification
      • Auto archiving entities
    • IP allowlist
    • Notifications
      • Notification logs
    • Customizing your workspace
    • Using search in Cortex
  • Cortex API
    • REST API operations
      • API Keys
      • Audit Logs
      • Catalog Entities
      • Custom Data
        • Custom Data (Advanced)
      • Custom Events
      • Custom Metrics
      • Dependencies
      • Deploys
      • Discovery Audit
      • Docs
      • Eng Intel: User Labels
      • Entity Relationship Types (Beta)
      • Entity Relationships (Beta)
      • Entity Types
      • GitOps Logs
      • Groups
      • Initiatives
      • Integrations APIs
        • Azure Active Directory (Entra ID) API
        • Azure Resources API
        • AWS API
        • Azure DevOps API
        • CircleCI API
        • Coralogix API
        • Datadog API
        • GitHub API
        • GitLab API
        • incident.io API
        • LaunchDarkly API
        • New Relic API
        • PagerDuty API
        • Prometheus API
        • SonarQube API
      • IP Allowlist
      • Notification Logs
      • On call
      • Packages
      • Plugins
      • Queries
      • SCIM
      • Scorecards
      • Secrets
      • Team Hierarchies
      • Teams
      • Workflows
Powered by GitBook
On this page
  • Setting up Scorecards as code
  • Converting an existing Scorecard to code
  • GitOps Scorecard best practices
  • Example Scorecard YAML file
  • Scorecard YAML reference

Was this helpful?

Export as PDF
  1. Scorecards

Scorecards as code

Last updated 3 days ago

Was this helpful?

Once Scorecards are incorporated into your regular engineering workflows, they become production-grade tools. Scorecards set the benchmarks for how your organization defines “good,” which impacts everything from security standards to on-call rotations. Because Scorecards are crucial to achieving and maintaining standards, you likely want to employ some kind of controlled access and version controlling.

With Scorecards as code, you can treat your Scorecard like any other production asset. This feature lets you manage Scorecards through , just like you’d manage the entities in your catalogs. Every Scorecard has its own YAML file (you can learn more about Scorecard YAML formatting below under ). When GitOps is enabled, changes made to Scorecards will appear in , just like when changes are made to other entities.

Setting up Scorecards as code

To migrate to Scorecards as code:

  1. Navigate to then click the Scorecards tab.

  2. Toggle off the Enable Scorecard UI editor option.

When you disable the Scorecard UI editor, it only affects Scorecards that are backed by a YAML file. You will still be able to create Scorecards via the Cortex UI, and you can still edit a Scorecard via the UI if it is not being managed through GitOps.

Converting an existing Scorecard to code

To convert an existing Scorecard into code:

  1. Navigate to the Scorecard's details page.

  2. Click the vertical ellipsis (⋮), then click Export CSV.

This will download a YAML version of your Scorecard.

Managing Scorecards via GitOps is designed to reduce as much friction as possible, so you can continue to make the most of this flexible tool.

GitOps Scorecard best practices

In general, the best place to put Scorecards is in their own repository, separate from catalog entities, at the repository's root directory within .cortex/scorecards. Note that it is not recommended to put Scorecard definitions in a service repository as Scorecards are not meant to be 1:1 with catalog entitites. For example, a simple repository might have the structure:

.
├── .cortex
│    └── scorecards
│        ├── dora.yml
│        └── performance.yml
└── src
    └── index.js
    └── ...

Any file found within the .cortex/scorecards directory will be automatically picked up and parsed as a Scorecard.

Example Scorecard YAML file

The dora-metrics-scorecard.yaml descriptor file might look something like this:

name: DORA Metrics
tag: dora-metrics
description: >-
  [DORA metrics](https://www.cortex.io/post/understanding-dora-metrics) are used by DevOps teams to measure their performance.

  The 4 key metrics are Lead Time for Changes, Deployment Frequency, Mean Time to Recovery, and Change Failure Rate.
draft: false
notifications:
  enabled: true
  scoreDropNotificationsEnabled: false
exemptions:
  enabled: true
  autoApprove: false
ladder:
  levels:
    - name: Bronze
      rank: 1
      description: Pretty good
      color: "#c38b5f"
    - name: Silver
      rank: 2
      description: Very good
      color: "#8c9298"
    - name: Gold
      rank: 3
      description: Excellent
      color: "#cda400"
rules:
  - title: Ratio of rollbacks to deploys in the last 7 days
    expression: >+
      (deploys(lookback=duration("P7D"),types=["ROLLBACK"]).length /
      deploys(lookback=duration("P7D"),types=["DEPLOY"]).length) > 0.05
    identifier: 3c42fa96-b422-30a4-b75a-f8b1cc233408
    description: Change Failure Rate
    weight: 25
    failureMessage: Less than 95% of deployments in the last 7 days were successful
    level: Gold
  - title: Incident was ack'ed within 5 minutes
    expression: oncall.analysis(lookback = duration("P7D")).meanSecondsToFirstAck <= 300
    identifier: 8713f2c0-f161-3688-9f99-bcfaab476b63
    description: MTTA (Mean time to acknowledge)
    weight: 25
    failureMessage: Incidents in the last 7 days were not ack'ed
    level: Silver
  - title: Last commit was within 24 hours
    expression: git.lastCommit().freshness = 7
    identifier: a16b7eeb-545b-359e-81a7-3946baacdd4b
    description: Deployment Frequency
    weight: 25
    failureMessage: No deployments in the last 7 days
filter:
  kind: GENERIC
  types:
    include:
      - service
  groups:
    include:
      - production
evaluation:
  window: 4

Scorecard YAML reference

Objects

Scorecard: 

name

description

name

The human-readable name of the Scorecard

tag

A unique slug for the Scorecard consisting of only alphanumeric characters and dashes

description

A human-readable description of the Scorecard

draft

Whether or not the Scorecard is a draft

notifications

Notifications settings for the Scorecard

ladder

The ladder to apply to the rules

rules

A list of rules that are evaluated each time the Scorecard is evaluated

filter

Enables the ability to exclude entities from being evaluated by this Scorecard

evaluation

Enables the ability to change the evaluation window for this Scorecard

Notifications: 

name

description

enabled

Whether or not to include the Scorecard in notifications

scoreDropNotificationsEnabled

Whether to notify about entities' scores that dropped after each evaluation of this Scorecard

Exemptions: 

name

description

enabled

Whether or not rule exemptions are enabled for Scorecard

autoApprove

Whether or not rule exemptions are auto approved for Scorecard

Ladder: 

name

description

levels

The levels of the ladder

Level: 

name

description

name

The human-readable name of the level

rank

The rank of the Level within the ladder. Higher rank is better.

description

A human-readable description of the level

color

The hex color of the badge that is displayed with the level

Rule: 

name

description

title

The human-readable name of the Rule

expression

The CQL expression to evaluate; must evaluate to a boolean

identifier

Identifier of the rule, unique within Scorecard scope.

description

A human-readable description of the Rule

weight

The number of points this Rule provides when successful

failureMessage

A human-readable message that will be presented when the Rule is failing

level

The name of the level this rule is associated with; can be null even when a ladder is present

filter

Enables the ability to exclude entities from being evaluated for this rule

effectiveFrom

Date when the rule starts being evaluated (e.g. 2024-01-01T00:00:00Z)

Filter: 

Note: One of types, groups, query must be present for it to be considered a valid filter

name

description

kind

The kind of filter to create. Currently only supports "GENERIC"

types

Types filter (to include / exclude specific types)

groups

Groups filter (to include / exclude specific groups)

query

A CQL query; only entities matching this query will be evaluated by the Scorecard

TypesFilter: 

Note: Only one of include/exclude can be specified at a time

name

description

include

List of types to include in set of entities

exclude

List of types to exclude in the set of entities

GroupsFilter: 

name

description

include

List of groups to include in set of entities

exclude

List of groups to exclude in the set of entities

Evaluation: 

name

description

window

By default, Scorecards are evaluated every 4 hours. That default can be changed under Settings > Scorecards > Default evaluation window. If you would like to evaluate Scorecards less frequently, you can override the evaluation window. This can help if you're encountering problems with rate limits. Note that Scorecards cannot be evaluated more than once per minimum set in Settings > Scorecards > Minimum evaluation window.

GitOps
GitOps logs
Settings > GitOps
GitOps Scorecard best practices
In the GitOps settings, click the Scorecard tab.