6. Creating and editing Scorecards
Scorecards are one of the most powerful ways in Cortex to promote accountability among teams and enforce standardization across entities.
Scorecards gather and organize existing data within Cortex to provide greater insight into the performance of your architecture. You can use Scorecards to establish best practices, track migration, or define maturity standards.
Before using Scorecards, make sure you have entities in your catalogs — Scorecards evaluate entities, so if none exist you won’t get much out of a Scorecard. You should also make sure essential integrations are set up, like Git.
You can find Scorecards in the main nav.
As with other features in Cortex, you’ll see all of your organization's Scorecards populate under All, while Scorecards you created will appear under Mine.
Those with the admin or manager permission levels will see the option to Create Scorecard.
Starting from scratch
When you create a new Scorecard, you have the option to use a Scorecard template or Start from scratch. To start from scratch, select that option under Getting started, and click Continue.
The first step in creating a Scorecard is entering some key details about it, just like with entities. Begin by adding a name, identifier, and description for the Scorecard.
Keep in mind that the Scorecard's name and description will be visible across Cortex, so make sure it’s obvious what entities are being scored on. For example, if you name your Scorecard “Security,” it’s pretty clear what that Scorecard is designed to measure.
Beyond that, you may link to a Confluence wiki page in the description and include a brief summary of how you intend to use the Scorecard. The description features full markdown support, so you can enter anything that is relevant in that field.
By default, Scorecards will be refreshed every 4 hours, but you can adjust the timeframe under Evaluation window. That default can be changed under Settings > Scorecards > Default evaluation window.
If a Scorecard is evaluating such a large number of entities that it cannot complete the evaluation in a 4-hour window (without invoking rate limits), then a longer evaluation window is recommended.
You can also toggle Notifications on for your Scorecard. When notifications are enabled, users who own entities being evaluated by the Scorecard will receive alerts when there are relevant updates, like score changes.
By default, users will be able to request rule exemptions for evaluated entities. You can disable this ability by toggling off Enable exemptions.
If rule exemptions are enabled, you can also Enable auto-approval exemptions, which will enable Cortex to automatically approve rule exemption requests. This setting will be disabled by default, and the toggle will be grayed out if rule exemptions are disabled.
Apply to specific entities
In the Apply to specific entities section, you'll decide which entities the Scorecard will apply to.
First, decide whether you're going to include or exclude entity types. Then select at least one entity type from the Entity types dropdown.
By default, the Scorecard will apply to all entities of the selected type(s), but you can narrow down the selection to more specific subsets. Under Advanced options, you can choose to Include groups of entities or Exclude groups. For example, if you're creating a services-based Scorecard, you might choose to include those that belong in a "tier 1 services" group.
With advanced filtering, you can write CQL expressions that dictate the entities that are included or excluded from the Scorecard. You can write your custom query within the provided field or use the CQL Builder to further refine your selection.
Advanced filtering only works with data that exist within Cortex, like entity groups, owners, and other custom data, but does not extend to third-party integrations.
Define evaluation rules
Next, you’ll Define evaluation rules — these are the criteria your entities will be scored against. Rules can reference entities' metadata within Cortex, and can pull data from third-party integrations.
There are two ways to set up your Scorecard's evaluation: Level progression, which allows you to establish progressive levels as you’re adding rules, and Point-based rules, where each rule has a weighted value.
Writing rules
Regardless of which option you choose, you'll need to write some rules — otherwise, your Scorecard won't measure anything.
There are two ways you can build a rule: through the Cortex UI or with the CQL Editor. The Cortex UI is a great way to get started quickly, since you can choose from pre-configured rules for each integration.
To use the Form editor, first select an integration from the dropdown menu — your options here will depend on the integrations that have been set up already. Next, select a rule from the dropdown menu.
Finally, enter information in the required fields to finish configuring the rule. These fields will vary depending on the integration/rule you've selected.
At this stage, you can add more information to make the rule more user friendly, like adding a Title. If you do not add a title, your rule will display as a CQL expression — for this reason, we strongly recommend adding a title.
In the Failure message field, you can enter a message that will display when an entity is failing a rule. This is a good opportunity to give team members some direction on how they can remediate the failure.
You can Restrict rule to specific groups to apply a more stringent filter to the entities in your Scorecard. This would make sense if a group of entities should be evaluated by the Scorecard as a whole, but a specific rule in the Scorecard does not apply to that group.
In the Points section, you can adjust the number of points a certain rule is worth. By default, the point value is 1. You can use points to signify each rule's importance — rules worth more points are more critical than those with fewer points.
The "Points" section only applies to and is visible within point-based Scorecards.
Once you're familiar with Cortex and CQL, you can use the CQL Editor to write your own expressions for each rule. This option is best when writing complex rules and/or accessing data from multiple sources.
Most of the time, it’ll make sense to work with an integration through the Form editor, especially in the beginning. This is one of the reasons we recommend setting up integrations before creating a Scorecard — it's easy and intuitive to build rules based on integrations.
Once you've defined your rule through the CQL Editor, you can enter a readable title and description, failure message, filters, and points.
Level progression
Within a single Scorecard, you might define 10 or 20 different evaluation rules. Although you can use points to assign weighted values to each rule, level progression makes it immediately obvious which ones are most important. With levels, you can set up a gamified system that encourages developers to make progress toward goals and improve entities' performance.
To start building levels for your Scorecard, select Level progression in the Define evaluation rules section, and then Add level.
This will open a modal where you can enter a Name and Description for your level, and choose a custom color. Levels are fully customizable, so you can go with something classic — like Bronze, Silver, and Gold — or something unique that represents your organization.
Levels are designed to inspire progress on important goals, so think about level names that will motivate team members.
Once you've created a level, you can begin adding rules to it. Select Add rule to [level name] to add rules to a given level.
This will open the Form editor for adding Scorecard rules. Keep in mind that your first level should include the most essential rules, so developers know to prioritize these.
Once you've finished adding rules for the level, you can add another level and repeat the process. You can't duplicate a rule across levels, since developers will have already completed that task earlier. It is common, however, to have similar rules with different thresholds if you want developers to progress in stages. For example, a base level might require that the P50 latency for API requests is under 5 seconds, while a higher level may required that the same P50 is under 2 seconds.
Once you have more than one level, you can rearrange the order by using the up and down arrows next to each level.
Level progression is also cumulative, so an entity must maintain its passing status with rules in lower levels in order to maintain its level. When you add a level, it includes All rules from [the prior level]. An entity could pass all the rules in the second level, but if it's failing a rule from the first, it won't be marked as passing the second level.
Let's take a look at the DORA Metrics Scorecard as an example:
Let's say an entity's last commit was within 24 hours, there were zero rollbacks in the last 7 days, and the ratios of incidents and rollbacks to deploys in the last 7 days are both zero, but it hasn't averaged at least one deploy per day in the last week.
In this case, the entity would have No Level. Even though it's passing all of the rules in the Bronze level, it's failing a rule in the Steel level; because of this, it has not achieved the Steel level, the first one that an entity can pass. Once the entity averages at least one deploy a day in the last week, it would automatically achieve the Bronze level, since it's passing the other four rules required for that level.
This kind of gamification motivates developers to not only progress through the levels, but to maintain the quality of their entities over time.
You can add as many levels as you want to a Scorecard. You can also add as many rules to each level as makes sense for the Scorecard, but keep in mind that an entity must pass all rules in a given level in order to progress to the next one. You want to make sure each level is achievable, even if challenging, so developers stay motivated.
Starting with a Scorecard template
Scorecard templates are a great way to get started quickly as you get more familiar with Cortex, since they come with pre-built rules and levels. To work with a Scorecard template, select any of the Scorecard templates built into Cortex, and click Continue.
From there, you’ll be taken to an overview page where you can view the integrations used by the Scorecard, as well as the template’s Levels and rules.
If you don't have the necessary integrations established for the Scorecard template, you'll need to configure them in settings first. Alternatively, if you don't have a relevant integration or you're not ready yet, you can edit out the rules you wish to omit on the following page.
On the next page, you have the opportunity to edit the Scorecard's name and description. At this stage, you'll also filter entities for the Scorecard. Before saving your Scorecard, you can also make adjustments to the template's predefined levels.
The draft toggle allows you to keep your Scorecard private until it’s ready to be viewed more widely. While in draft status, your Scorecard will not appear in reports or send notifications and it’ll only be visible to admins and managers, so it won’t be noisy for developers.
If you’re testing a Scorecard or aren’t ready for others to see it, keep the draft toggle engaged.
Viewing and editing a Scorecard
Once you select Save Scorecard, your Scorecard will automatically evaluate all the entities that apply based on the filters you set up in Apply to specific entities and you'll be taken to the Scorecard's page.
At the top of the page, you'll see high-level information about the Scorecard:
- Median level entities have achieved
- Percent at the highest level
- Percent without level
- Number of entities
You'll also be able to find the Scorecard's description and its identifier from this page, as well as which entities the Scorecard applies to.
In the Scores tab, you can see at a glance which entities have achieved which levels, and how many rules remain before that entity can progress to the next level.
Click on any entity name to get more information about its progress. In the side panel, you can see the specific tasks the entity must pass to advance, the rules it's currently failing and passing, and its progress over time.
This makes it easy for developers to understand exactly what they should be doing and lays out the priorities in a clear way. The visibility enabled by Scorecards then allows developers to target specific attributes of specific entities for improvement.
In the Rules tab, you can find all of the Scorecard's rules laid out for easy readability.
If you didn't set a human-readable title for your rule, the CQL expressions will display under the Rules tab instead.
Click the caret next to any rule to see the expression behind the rule.
From the Reports tab, you can automatically view a Bird's Eye report, Progres report, or Report Card based on that Scorecard.
From the Rule Exemptions tab, you can see exemptions that have requested for specific entities. With rule exemptions, entities are not marked as passing or failing — the rule simply does not apply to those entities.
If you're the owner of the Scorecard, then you can also Deny or Approve requested exemptions, as well as Revoke Exemptions that have been granted.
Editing a Scorecard
To edit a Scorecard, click Edit on that Scorecard's home page, above the description.
The Edit Scorecard page is nearly identical to the Create Scorecard page. You can modify your Scorecard’s name and description, edit levels, add or delete rules, and toggle the draft status. You can also adjust the filter criteria and change the entities being evaluated by the Scorecard to fine-tune your insights over time. The only thing you cannot do is modify the Scorecard's identifier, which is immutable.
Any time you edit and save your Scorecard, Cortex will automatically begin reevaluating the relevant entities to reflect your changes.