Skip to main content

6. Creating and editing Scorecards

Overview​

Scorecards gather and organize existing data within Cortex to provide greater insight into the performance of your architecture. Use Scorecards to establish best practices, track migration, promote accountability among teams, enforce standardization across entities, or define maturity standards.

In a Scorecard, entities are scored against rules. Rules can reference entities' metadata within Cortex, and can pull data from third-party integrations. Use levels and points in Scorecards to gamify the process and encourage developers to make progress toward goals.

A list of scorecards is displayed

Common Scorecard use cases​

For information about common use cases and examples, please see Scorecard examples.

How to create Scorecards​

Prerequisites​

  • To create a Scorecard, you must have the admin or manager role, or you must have a custom role that includes the permission Edit Scorecards.
  • Scorecards evaluate entities, so before using Scorecards, make sure you have entities in your catalogs.
  • Make sure you have set up any integrations you want to write rules against.

Step 1: Configure the basic Scorecard fields​

When you create a new Scorecard, you have the option to use a Scorecard template or Start from scratch. See the tabs below for instructions on starting your Scorecard.

To help you get started more quickly, Cortex offers the following templates:

  • Onboarding Scorecard
  • Production Readiness
  • Service Maturity
  • Incident Preparedness
  • Security
  • Vulnerability Management
  • Eng Intelligence metrics
  • DORA metrics
  • Code Quality
  • JavaScript Best Practices
  • Secrets Management
  • Minimum Postgres version for Aurora
  • Minimum version for ElastiCache for Redis
  • Minimum Lambda runtime versions
  • Empty Scorecard with Gold/Silver/Bronze levels pre-configured

To create a Scorecard using a template:

  1. On the Scorecards page in Cortex, click Create Scorecard.
  2. Click Scorecard template.
  3. Click the template you want to use, then at the bottom of the page, click Continue.
    • You will be redirected to a page that shows the default configuration for your Scorecard, including the integrations used by the Scorecard and the template's levels and rules. You will be able to modify the Scorecard on the next page.
  4. Click Continue at the bottom of the Scorecard default configuration page.
  5. Configure the basic fields for your Scorecard:
    • Name: Enter a descriptive name for your Scorecard.
    • Identifier: Enter a unique identifier for your Scorecard.
    • Description: Enter a description for your Scorecard.
    • Evaluation window: By default, Scorecards are evaluated every 4 hours. If you would like to evaluate Scorecards less frequently, you can override the evaluation window and enter a new number.
      • If a Scorecard is evaluating such a large number of entities that it cannot complete the evaluation in a 4-hour window without invoking rate limits, then a longer evaluation window is recommended.
    • Enable notifications: Choose whether to enable notifications.
      • If notifications are enabled, users who own entities being evaluated by this Scorecard will receive alerts when there are relevant updates on the Scorecard and any corresponding initiatives.
      • Notify when scores drop: When enabled, you will be notified any time a score drops from the last time the Scorecard was evaluated.
    • Enable exemptions: Choose whether to enable Scorecard exemptions. If not set, it defaults to true.
    • Enable auto-approval exemptions: Choose whether to enable auto-approving of Scorecard rule exemptions. If not set, it defaults to false.
  6. Select whether to apply this Scorecard to specific entities.
    • You can choose to include or exclude specific entity types.
    • Click Advanced options to see additional options for refining entity selection. You can include or exclude groups, and you can include a CQL expression to make your entity type selection.

If you want to use all of the default rules from the template, you can skip ahead to Step 4: Save the Scorecard. If you want to add any rules or levels, follow the next steps.

caution

Advanced filtering only works with data that exist within Cortex, like entity groups, owners, and other custom data, but does not extend to third-party integrations.

Step 2: Add levels​

When you create a Scorecard, there are two options to set up your Scorecard's evaluation:

  • Level progression
    • Establish progressive levels as you add rules to make it obvious which set of rules are the highest priority. Levels allow you to set up a gamified system to encourage developers to make progress toward goals and improve entities' performance.
  • Point-based rules
    • Use points to assign weighted values to each rule you add to a Scorecard.
tip

If you choose to use only point-based rules in your Scorecard, then you do not need to create levels. You can skip to the next section: Step 3: Create a rule.

To add a level:

  1. While creating your Scorecard, in the "Define evaluation rules" section, select Level progression for the scoring type. Click Add level.
    Click Add level under the scoring type selection
  2. In the modal, configure your level:
    • Name: Enter a name that represents this level.
      • You might choose to go with classic levels names, such as Bronze, Silver, and Gold, or you can choose something more unique to your organization.
      • Levels are designed to inspire progress on important goals, so consider level names that will motivate your team members.
    • Description: Enter a description for the level.
    • Color: Click the star icon to select a color for this level.
  3. Click Save level.

You can add more levels to the Scorecard, or you can move on to the next step and add rules to each level.

Reorder levels​

When you have more than one level, you can click the arrows on the right side to reorder them:
Click the arrow on the right side of a level

Step 3: Create a rule​

There are two methods to build a rule: through a form in the Cortex UI or via the CQL editor.

The form is the fastest option, as you can choose from pre-configured rules for each integration. The CQL editor is best when writing complex rules or accessing data from multiple sources.

  1. While creating your Scorecard, in the "Define evaluation rules" section, click Add rule.
    • If you are using the "Level progression" scoring type, click Add rule within a level.
      • When using the "Level progression" scoring type, we recommend including the most essential rules in the first level so developers know to prioritize them. In a level titled "Level 1: Bronze" you can click "Add rule to Level 1 Bronze" in the lower left
    • If you are using the "Point-based rules" scoring type, click Add rule below the scoring type selection:
      Click Add rule under the scoring type selection
  2. In the upper right side of the rule modal, click Form to choose a pre-configured rule, or click CQL Editor to write a CQL expression from scratch.
    Choose Forms for the UI editor or CQL editor to write CQL rules
  3. Configure the basic fields in the form:
    • If you chose CQL Editor:
      • CQL expression: Enter your CQL rule. On the right side of the modal, you can use CQL Explorer for help.
      • Name: Enter a descriptive name. Names make it easier to understand the outcome of the rule without having to read the CQL expression.
    • If you chose Form:
      • Choose an integration: Choose one of your configured integrations to apply this rule to.
      • Choose rule: Choose a rule.
      • The next field varies depending on the type of rule you choose. For example, you may need to choose a boolean operation or set a file path.
      • Name: Enter a descriptive name. Names make it easier to understand the outcome of the rule without having to read the CQL expression.
        Configure the form to create a rule
  4. Optionally configure the additional fields:
    • Description: Enter a description for the rule.
    • Failure message: Add links or instructions to tell users how they can remediate this rule if it's failing.
    • Restrict rule to specific groups: Select groups to include or exclude.
      • This option can be used when a group of entities should be evaluated by the Scorecard as a whole, but a specific rule in the Scorecard does not apply to that group.
      • See Scorecard rule filters for more information on configuring a filter.
    • Points: Enter how many points this rule is worth.
      • You can use points to signify each rule's importance. Rules worth more points are more critical than those with fewer points.
    • Schedule evaluation start date: If you want to schedule the evaluation of a rule to start on a specific date, schedule a start date here. When the field is blank, the rule will take effect immediately. If a start date is provided, the rule will take effect at the start of the specified day (UTC).
      • This is helpful when you want users to be aware of an upcoming rule, but you do not want to evaluate the Scorecard yet based on that rule.
      • Scheduled rules are visible to all users and will be included in the weekly team and user reports to help with socialization.
      • Users can choose to receive notifications when a scheduled rule is added to a Scorecard evaluating an entity they own, and they can receive reminders before a scheduled rule impacting an owned entity goes into effect.
  5. Click Save rule.

You can repeat the steps above to continue adding rules to your Scorecard. You cannot duplicate a rule across levels, since developers will have already completed that task earlier. It is common, however, to have similar rules with different thresholds if you want developers to progress in stages. For example, a base level might require that the P50 latency for API requests is under 5 seconds, while a higher level may required that the same P50 is under 2 seconds.

Step 4: Save the Scorecard​

After you have finished adding levels and rules to your Scorecard, you are ready to save the configuration.

  1. Next to "All set?" choose whether to keep your Scorecard in Draft form, visible only to users with permission to configure Scorecards.
    • If you choose to keep your Scorecard in Draft status, it will not appear in reports or send notifications.
    • If you choose to publish your Scorecard, it will be visible to all users who have the View Scorecards permission. Notifications, if enabled, will be triggered.
  2. Click Save Scorecard.

After you save the Scorecard, it will automatically evaluate all the entities that apply based on the rules you have configured, and you will be redirected to the Scorecard's page in Cortex.

Next steps​

After you have established Scorecards, you can start using Initiatives to drive progress across the organization on your goals. Initiatives allow you to prioritize specific rules within a Scorecard and set deadlines for team members to accomplish tasks.

Viewing and editing Scorecards​

View Scorecards list​

Click Scorecards in the main nav in Cortex.

All of your organization's Scorecards are listed under All, while Scorecards you created will appear under Mine.

At the top of the Scorecard list, there are tabs for All and Mine

Edit a Scorecard​

You can edit a Scorecard's name, description, levels, rules, draft status, filter criteria, and the entity's being evaluated by the Scorecard. You cannot edit a Scorecard's unique identifier.

To edit:

  1. Navigate to the Scorecard in Cortex.
  2. Click Edit in the upper right corner of the Scorecard page:
    The Edit button is in the upper right corner of the Scorecard page
  3. Make changes to your Scorecard, then at the bottom of the page, click Save Scorecard.

Any time you edit and save your Scorecard, Cortex will automatically begin reevaluating the relevant entities to reflect your changes.

Review a Scorecard​

When you navigate to a Scorecard's page in Cortex, you can see high-level information at the top of the page:

  • Median level entities have achieved
  • Percent at the highest level
  • Percent without level
  • Number of entities
  • Number of scheduled rules

You can also see the Scorecard's description and its identifier from this page, as well as which entities the Scorecard applies to. A high level overview of a Scorecard

View scores​

In the Scores tab, you can see at a glance which entities have achieved which levels, and how many rules remain before that entity can progress to the next level. The Scores tab shows what entities are at each level

Click on any entity name to get more information about its progress. In the side panel, you can see the specific tasks the entity must pass to advance, the rules it's currently failing and passing, and its progress over time: The side panel shows failing and passing rules for an entity

View rules​

In the Rules tab, view all of the Scorecard's rules, per level. Expand a rule to see its CQL expression: View the rules and expand them to view the CQL

If you did not set a name for a rule when you created it, it will display here as a CQL expression.

View reports​

In the Reports tab, view a Bird's Eye report, Progress report, or Report Card. View the rules and expand them to view the CQL

View Rule Exemptions​

In the Rule Exemptions tab, see exemptions that have been requested for specific entities. When a rule is exempted, entities are not marked as passing or failing — the rule simply does not apply to those entities. View the rule exemptions

A user with the Manager or Admin role - or a user with a custom role containing the Configure Scorecard Exemptions permission - can deny, approve, or revoke exemptions.

Example: DORA Metrics Scorecard​

See a DORA Metrics Scorecard as an example:

scorecard 18

  • Let's say an entity's last commit was within 24 hours, there were zero rollbacks in the last 7 days, and the ratios of incidents and rollbacks to deploys in the last 7 days are both zero, but it hasn't averaged at least one deploy per day in the last week.
    • In this case, the entity would have No Level.
    • Even though it's passing all of the rules in the Bronze level, it is failing a rule in the Steel level; because of this, it has not achieved the Steel level, the first one that an entity can pass.
    • Once the entity averages at least one deploy a day in the last week, it would automatically achieve the Bronze level, since it's passing the other four rules required for that level.

This kind of gamification motivates developers to not only progress through the levels, but to maintain the quality of their entities over time.

You can add as many levels as you want to a Scorecard. You can also add as many rules to each level as makes sense for the Scorecard, but keep in mind that an entity must pass all rules in a given level in order to progress to the next one. You want to make sure each level is achievable, even if challenging, so developers stay motivated.