LogoLogo
Login to CortexBook a DemoCortex Academycortex.io
  • Cortex Docs
  • Cortex Quick Start
  • Ingesting data into Cortex
    • Managing Entities
      • Adding entities
        • Add services
        • Add domains
        • Add teams
        • Add custom entity types
        • Defining dependencies
      • Entity details page
      • Defining ownership
      • Defining relationship types
      • Grouping entities
      • Adding external documentation
      • Adding Deploy data
      • Adding custom data
      • Viewing discovered entities
      • Archiving entities
      • Relationship graph
      • Using On-call Assistant for incidents
      • Managing Terraform infra in Cortex
    • Managing Catalogs
    • Integrations
      • Internally hosted integrations
      • ArgoCD
      • AWS
      • Azure DevOps
      • Azure Resources
      • BambooHR
      • Bitbucket
      • BugSnag
      • Buildkite
      • Checkmarx
      • CircleCI
      • ClickUp
      • Codecov
      • Coralogix
      • Custom webhook integrations
      • Datadog
      • Dynatrace
      • Entra ID (Azure AD)
      • FireHydrant
      • GitHub
      • GitLab
      • Google
      • Grafana
      • incident.io
      • Instana
      • Jenkins
      • Jira
      • Kubernetes
      • LaunchDarkly
      • Lightstep
      • Mend
      • Microsoft Teams
      • New Relic
      • Okta
      • Opsgenie
      • PagerDuty
      • Prometheus
      • Rollbar
      • Rootly
      • Sentry
      • ServiceNow
      • Slack
      • Snyk
      • SonarQube
      • Splunk Observability Cloud (SignalFx)
      • Splunk On-Call (VictorOps)
      • Sumo Logic
      • Veracode
      • Wiz
      • Workday
      • xMatters
  • Scorecards
    • Initiatives and Action items
      • Creating issues based on Initiatives
    • Scorecard rule exemptions
    • Scorecard rule filters
    • Scorecard examples
    • Scorecards as code
  • Reports
    • Executive report
    • All Scorecards report
    • Bird's eye report
    • Progress report
    • Report card
  • Eng Intelligence
    • Custom Metrics
    • Jira Metrics
    • Metrics Explorer (Beta)
  • Cortex Query Language (CQL)
    • Using CQL reports
    • Using JQ in Cortex
  • Workflows
    • Creating a Workflow
      • Workflows as code
    • Blocks
    • Running a Workflow
    • Registering a Scaffolder template
      • Scaffolder advanced usage
    • Using a Workflow to sync in ArgoCD
    • Kicking off a Jenkins pipeline in a Workflow
    • Calling internal service endpoints in a Workflow
  • Plugins
    • Creating a plugin
      • Creating a plugin proxy
    • Migrating Backstage plugins to Cortex
  • Engineering homepage
  • Workspace Settings
    • Using GitOps for Cortex
      • GitOps logs
    • Managing users
      • Roles and permissions
        • Custom roles
        • Team ownership entity editing
      • Configuring SSO
        • Microsoft Entra ID
        • Google
        • Other OIDC providers
        • Okta
          • Okta SCIM
      • Configuring identity mappings
      • Onboarding management
    • API keys, secrets, and tokens
      • Secrets
      • Personal tokens
    • Audit logs
    • Entity settings
      • Data verification
      • Auto archiving entities
    • IP allowlist
    • Notifications
      • Notification logs
    • Customizing your workspace
    • Using search in Cortex
  • Cortex API
    • REST API operations
      • API Keys
      • Audit Logs
      • Catalog Entities
      • Custom Data
        • Custom Data (Advanced)
      • Custom Events
      • Custom Metrics
      • Dependencies
      • Deploys
      • Discovery Audit
      • Docs
      • Eng Intel: User Labels
      • Entity Relationship Types (Beta)
      • Entity Relationships (Beta)
      • Entity Types
      • GitOps Logs
      • Groups
      • Initiatives
      • Integrations APIs
        • Azure Active Directory (Entra ID) API
        • Azure Resources API
        • AWS API
        • Azure DevOps API
        • CircleCI API
        • Coralogix API
        • Datadog API
        • GitHub API
        • GitLab API
        • incident.io API
        • LaunchDarkly API
        • New Relic API
        • PagerDuty API
        • Prometheus API
        • SonarQube API
      • IP Allowlist
      • Notification Logs
      • On call
      • Packages
      • Plugins
      • Queries
      • SCIM
      • Scorecards
      • Secrets
      • Team Hierarchies
      • Teams
      • Workflows
Powered by GitBook
On this page
  • Overview
  • How to create Scorecards
  • Prerequisites
  • Step 1: Configure the basic Scorecard fields
  • Step 2: Add levels
  • Step 3: Create a rule
  • Step 4: Save the Scorecard
  • Next steps
  • Create Initiative from Scorecard
  • Viewing and editing Scorecards
  • View Scorecards list
  • Edit a Scorecard
  • Review a Scorecard
  • Scorecard examples
  • Scorecard evaluation
  • Trigger an evaluation
  • Cancel an evaluation

Was this helpful?

Export as PDF

Scorecards

Last updated 3 days ago

Was this helpful?

Overview

Use Scorecards to establish best practices, track migration, promote accountability among teams, enforce standardization across entities, or define maturity standards.

In a Scorecard, entities are scored against rules. Rules can reference entities' metadata within Cortex, and can pull data from third-party integrations. Use levels and points in Scorecards to gamify the process and encourage developers to make progress toward goals.

Common Scorecard use cases

How to create Scorecards

Prerequisites

  • To create a Scorecard, you must have the admin or manager role, or you must have a custom role that includes the permission Edit Scorecards.

  • Make sure you have set up any integrations you want to write rules against.

Step 1: Configure the basic Scorecard fields

When you create a new Scorecard, you have the option to use a Scorecard template or Start from scratch.

Cortex offers the following templates:

Scorecard templates
  • Onboarding Scorecard

  • Production Readiness

  • Service Maturity

  • Incident Preparedness

  • Security

  • Vulnerability Management

  • Eng Intelligence metrics

  • DORA metrics

  • Code Quality

  • JavaScript Best Practices

  • Secrets Management

  • Minimum Postgres version for Aurora

  • Minimum version for ElastiCache for Redis

  • Minimum Lambda runtime versions

  • Empty Scorecard with Gold/Silver/Bronze levels pre-configured

See the tabs below for instructions on starting your Scorecard.

To create a Scorecard using a template:

  1. On the Scorecards page in Cortex, click Create Scorecard.

  2. Click Scorecard template.

  3. Click the template you want to use, then at the bottom of the page, click Continue.

    • You will be redirected to a page that shows the default configuration for your Scorecard, including the integrations used by the Scorecard and the template's levels and rules. You will be able to modify the Scorecard on the next page.

  4. Click Continue at the bottom of the Scorecard default configuration page.

  5. Configure the basic fields for your Scorecard:

    • Name: Enter a descriptive name for your Scorecard.

    • Identifier: Enter a unique identifier for your Scorecard.

    • Description: Enter a description for your Scorecard.

    • Evaluation window: By default, Scorecards are evaluated every 4 hours. If you would like to evaluate Scorecards less frequently, you can override the evaluation window and enter a new number.

      • If a Scorecard is evaluating such a large number of entities that it cannot complete the evaluation in a 4-hour window without invoking rate limits, then a longer evaluation window is recommended.

    • Enable notifications: Choose whether to enable notifications.

      • If notifications are enabled, users who own entities being evaluated by this Scorecard will receive alerts when there are relevant updates on the Scorecard and any corresponding initiatives.

      • Notify when scores drop: When enabled, you will be notified any time a score drops from the last time the Scorecard was evaluated.

    • Enable exemptions: Choose whether to enable Scorecard exemptions. If not set, it defaults to true.

    • Enable auto-approval exemptions: Choose whether to enable auto-approving of Scorecard rule exemptions. If not set, it defaults to false.

  6. Select whether to apply this Scorecard to specific entities.

    • You can choose to include or exclude specific entity types.

    • Click Advanced options to see additional options for refining entity selection. You can include or exclude groups, and you can include a CQL expression to make your entity type selection.

  1. On the Scorecards page in Cortex, click Start from scratch, then at the bottom of the page, click Continue.

  2. Configure the basic fields for your Scorecard:

    • Name: Enter a descriptive name for your Scorecard.

    • Identifier: Enter a unique identifier for your Scorecard.

    • Description: Enter a description for your Scorecard.

    • Evaluation window: By default, Scorecards are evaluated every 4 hours. If you would like to evaluate Scorecards less frequently, you can override the evaluation window and enter a new number.

      • If a Scorecard is evaluating such a large number of entities that it cannot complete the evaluation in a 4-hour window without invoking rate limits, then a longer evaluation window is recommended.

    • Enable notifications: Choose whether to enable notifications.

      • If notifications are enabled, users who own entities being evaluated by this Scorecard will receive alerts when there are relevant updates on the Scorecard and any corresponding initiatives.

      • Notify when scores drop: When enabled, you will be notified any time a score drops from the last time the Scorecard was evaluated.

    • Enable exemptions: Choose whether allow users to request rule exemptions for evaluated entities. If not set, it defaults to enabled.

    • Enable auto-approval exemptions: If exemptions are allowed, choose whether to enable auto-approval of Scorecard rule exemptions. If not set, it defaults to disabled.

  3. Next to "Apply to specific entities," select whether to apply this Scorecard to specific entities:

    • Selection type and Entity types: Choose to include or exclude specific entity types. By default, the Scorecard will apply to all entities of the selected type(s).

    • Click Advanced options to see additional options for refining entity selection. You can include or exclude groups. For example, if you're creating a services-based Scorecard, you might choose to include those that belong in a "tier 1 services" group.

Advanced filtering only works with data that exist within Cortex, like entity groups, owners, and other custom data, but does not extend to third-party integrations.

See the next section for steps on choosing a scoring type and writing rules.

Step 2: Add levels

  • Level progression

    • Establish progressive levels as you add rules to make it obvious which set of rules are the highest priority. Levels allow you to set up a gamified system to encourage developers to make progress toward goals and improve entities' performance.

  • Point-based rules

    • Use points to assign weighted values to each rule you add to a Scorecard.

To add a level:

  1. In the modal, configure your level:

    • Color: Click the star icon to select a color for this level.

    • Name: Enter a name that represents this level.

      • You might choose to go with classic levels names, such as Bronze, Silver, and Gold, or you can choose something more unique to your organization.

      • Levels are designed to inspire progress on important goals, so consider level names that will motivate your team members.

    • Description: Enter a description for the level.

  2. Click Add level.

You can add more levels to the Scorecard, or you can move on to the next step and add rules to each level.

Reorder levels

Step 3: Create a rule

There are two methods to build a rule: through a form in the Cortex UI or via the CQL editor.

The form is the fastest option, as you can choose from pre-configured rules for each integration. The CQL editor is best when writing complex rules or accessing data from multiple sources.

  1. While creating your Scorecard, in the "Define evaluation rules" section, click Add rule.

      • When using the "Level progression" scoring type, we recommend including the most essential rules in the first level so developers know to prioritize them.

  2. Configure the basic fields in the form:

    • If you chose Form:

      • Integration: Choose one of your configured integrations to apply this rule to.

      • Rule: Choose a rule.

      • The next fields vary depending on the type of rule you choose. For example, you may need to choose a boolean operation or set a file path.

    • If you chose CQL editor:

      • CQL expression: Enter your CQL rule. On the right side of the modal, you can use CQL Explorer for help.

      • Rule name: Enter a descriptive name. Names make it easier to understand the rule without having to read the CQL expression.

  3. Optionally configure the additional fields:

    • Description: Enter a description for the rule.

    • Failure message: Add links or instructions to tell users how they can remediate this rule if it's failing.

    • Restrict rule to specific groups: Select groups to include or exclude.

      • This option can be used when a group of entities should be evaluated by the Scorecard as a whole, but a specific rule in the Scorecard does not apply to that group.

    • Points: Enter how many points this rule is worth.

      • You can use points to signify each rule's importance. Rules worth more points are more critical than those with fewer points.

    • Schedule evaluation start date: If you want to schedule the evaluation of a rule to start on a specific date, schedule a start date here. When the field is blank, the rule will take effect immediately. If a start date is provided, the rule will take effect at the start of the specified day (UTC).

      • This is helpful when you want users to be aware of an upcoming rule, but you do not want to evaluate the Scorecard yet based on that rule.

      • Scheduled rules are visible to all users and will be included in the weekly team and user reports to help with socialization.

  4. Click Save rule.

You can repeat the steps above to continue adding rules to your Scorecard. You cannot duplicate a rule across levels, since developers will have already completed that task earlier. It is common, however, to have similar rules with different thresholds if you want developers to progress in stages. For example, a base level might require that the P50 latency for API requests is under 5 seconds, while a higher level may required that the same P50 is under 2 seconds.

Step 4: Save the Scorecard

After you have finished adding levels and rules to your Scorecard, you are ready to save the configuration.

  1. Choose whether to keep your Scorecard in draft form, visible only to users with permission to configure Scorecards. To keep it in draft form, enable the toggle next to Save as draft.

    • If you choose to keep your Scorecard in draft status, it will not appear in reports or send notifications.

    • If you choose to publish your Scorecard, it will be visible to all users who have the View Scorecards permission. Notifications, if enabled, will be triggered.

  2. Click Create Scorecard.

After you save the Scorecard, it will automatically evaluate all the entities that apply based on the rules you have configured, and you will be redirected to the Scorecard's page in Cortex.

Next steps

Create Initiative from Scorecard

Initiatives allow you to prioritize specific rules and set deadlines, making sure your team members complete higher priority tasks on time.

You can create an Initiative while viewing a Scorecard. Note that the Scorecard must be published - i.e., not in draft mode - in order to create an Initiative for it.

  1. In the upper right corner, click +Create initiative.

  2. Configure the Initiative form.

Viewing and editing Scorecards

View Scorecards list

All of your organization's Scorecards are listed under All, while Scorecards you created will appear under Mine.

Edit a Scorecard

You must have the Edit Scorecard permission.

To edit:

  1. Navigate to the Scorecard in Cortex.

  2. Click Edit in the upper right corner of the Scorecard page:

  3. Make changes to your Scorecard, then at the bottom of the page, click Save Scorecard.

Any time you edit and save your Scorecard, Cortex will automatically begin reevaluating the relevant entities to reflect your changes.

Review a Scorecard

When you navigate to a Scorecard's page in Cortex, you can see high-level information at the top of the page:

  • Median level entities have achieved

  • Percent of entities at the highest level

  • Percent of entities without level

  • Number of entities per level

  • The scheduled Scorecard calculation

  • Whether notifications or rule exemptions are enabled

On the side the page, you can see:

  • A description of the Scorecard

  • The total number of entities being scored

  • The number of scheduled rules

  • The next evaluation time for the Scorecard

On the Scorecard page, click through each of the tabs to learn more:

Under Scores, view each entity that is being scored via this Scorecard, the entity's current overall level, and how many points it scored per level.

For example, in the screen shot above, the first two entities achieved 50 points out of 50 possible points within the Bronze level of the Scorecard. They also each achieved 3 out of 3 possible points under the Silver level.

View more about each entity

Click an entity to open a side panel with more information:

  • Next to the entity name: A link to its entity details page

  • At the top of the panel: The entity's current level, score, rank, percentile, and an overview of its rule progress

  • Under the Rules tab: A list of failing rules, passing rules, and rules not evaluated yet

  • Under the Exemptions tab: A list of exempted rules

  • Under the Progress tab: A graph showing the entity's progress

Filter and sort the scores list

In the upper right corner of the list, click Filter to narrow the scope of the list. You can filter by domain, failing rules, groups, levels, owners, passing rules, and teams.

Click Level to select a different method to sort by, and choose whether to sort ascending or descending.

Share the Scorecard

Need to show progress to leadership? Click Share in the upper right corner. This copies a link to your clipboard, which you can share with anyone who has access to view this Scorecard in your workspace.

Under Overview, see which entity is the highest ranking, which entity is most at-risk, which rule is the most at-risk, the most improved entity, the entity with the worst drop, and the most-moved rule.

Under those metrics, see a graph displaying the progress of all rules. Click All rules above the graph to filter it by specific rules.

By default, the Trends and changes blocks show data from the last month. Click Last month to select a different timeframe.

Under Rules, view all of the Scorecard's rules, per level. Expand a rule to see its CQL expression:

Note that if you did not set a name for a rule when you created it, it will display here as a CQL expression.

Under Reports, click to view a Bird's Eye report, Progress report, or Report card.

Under Exemptions, view a list of all rule exemptions that have been requested.

Each request includes its current status, the rule, an expiration date, the reason for the request, and the user who requested it.

How exemptions affect scores

When a rule is exempted, entities are not marked as passing or failing — the rule simply does not apply to those entities.

Approving or denying exemptions

You must have the the Configure Scorecard Exemptions permission to approve or deny exemptions.

To approve an exemption, click the checkmark icon. To deny an exemption, click the X icon:

Under Initiatives, see all initiatives associated with the Scorecard.

Scorecard examples

Gamification motivates developers to not only progress through the levels of a Scorecard, but to maintain the quality of their entities over time.

Scorecard evaluation

Scorecard evaluations are also automatically initiated after Scorecard definition changes are processed via GitOps or other means. This evaluation is of lower priority than a manual evaluation and may take longer to return results. To see the re-evaluation results more quickly, you can cancel an automatic evaluation and trigger a manual evaluation.

Trigger an evaluation

To manually trigger a Scorecard evaluation:

  1. Navigate to the Scorecard's page in Cortex.

  2. In the details panel on the right side of the page, click Evaluate now.

Cancel an evaluation

If a Scorecard is in the process of being evaluated, you can cancel it:

  1. Navigate to the Scorecard's page in Cortex.

  2. Click Cancel at the top of the page.

You can configure Scorecards in the Cortex UI, or you can follow a GitOps approach. See for more information.

For information about common use cases and examples, please see .

Looking to dive deeper? Check out the , available to all Cortex customers and POVs.

Scorecards evaluate entities, so before using Scorecards, make sure you have .

You will also be able to when needed.

See for more information.

If you want to use all of the default rules from the template, you can skip ahead to . If you want to add any rules or levels, follow the next steps.

See for more information.

You can also include a CQL expression to make your entity type selection.

When you , there are two options to set up your Scorecard's evaluation:

If you choose to use only point-based rules in your Scorecard, then you do not need to create levels. You can skip to the next section: .

While creating your Scorecard, in the "Define evaluation rules" section, click Add level.

When you have more than one level, you can click the arrows on the right side to reorder them:

Level progression scoring type: Click Add rule within a level.

Point-based rule scoring type: Enable the toggle next to Point-based rules, then click Add rule:

The rule creation modal defaults to the Form type, where you can choose a pre-configured rule based on configured integrations. To write a CQL expression instead, click CQL:

Rule name: Enter a descriptive name. Names make it easier to understand the outcome of the rule without having to read the CQL expression.

Note: If writing an expression that references a Jira issues, Cortex will not append new filter logic to your default JQL query. See for more information.

See for more information on configuring a filter.

Users can choose to receive when a scheduled rule is added to a Scorecard evaluating an entity they own, and they can receive reminders before a scheduled rule impacting an owned entity goes into effect.

After you have established Scorecards, you can start using to drive progress across the organization on your goals. Initiatives allow you to prioritize specific rules within a Scorecard and set deadlines for team members to accomplish tasks.

For detailed instructions on creating an Initiative, see the .

You can also create an Initiative while viewing reports for the Scorecard under its .

Click in the main nav in Cortex.

If you have not , then you can edit a Scorecard in the Cortex UI. You can edit the name, description, levels, rules, draft status, filter criteria, and the entity's being evaluated by the Scorecard. You cannot edit a Scorecard's unique identifier.

See the to view common use cases and see how others are motivating their teams with Scorecards.

After you create and save a Scorecard, Cortex automatically evaluates the entities that the Scorecard applies to. How often the Scorecard is evaluated depends on the evaluation window you set when .

Scorecards as code
Cortex Academy course on Scorecards
entities in your catalogs
Scorecard rule exemptions
Scorecard rule exemptions
Scorecard rule filters
notifications
Initiatives
Initiatives documentation
Scorecards
Scorecard examples page
Scorecard examples
manually trigger an evaluation
Step 4: Save the Scorecard
create a Scorecard
Step 3: Create a rule
Reports tab
configuring the basic fields of the Scorecard
Adding filter logic to the default JQL query in a Scorecard
A list of scorecards is displayed
Click "Create Initiative" in the upper right.
At the top of the Scorecard list, there are tabs for All and Mine
Click "Edit" in the upper right corner.
View basic information on the Scorecard page
The Scores tab shows scores and levels per entity.
Click into an entity to view more information.
Click "Evaluate now" in the right side of a Scorecard
Click "Cancel" at the top of a Scorecard
disabled UI editing