Review and evaluate Scorecards
Last updated
Was this helpful?
Last updated
Was this helpful?
Was this helpful?
Reviewing a Scorecard's evaluation gives you insight into patterns across teams or entities, and areas to prioritize for improvement.
When an entity is failing a rule in a Scorecard, Cortex provides the information needed to review and remediate the issue efficiently. From there, teams can take action on the failing rule. Resolving these failures not only improves the health and maturity of individual entities but also strengthens the overall reliability and compliance posture of the engineering organization.
After you create and save a Scorecard, Cortex automatically evaluates the entities that the Scorecard applies to. How often the Scorecard is evaluated depends on the evaluation window you set when configuring the basic fields of the Scorecard.
Scorecard evaluations are also automatically initiated after Scorecard definition changes are processed via GitOps or other means.
When processing a Scorecard, the individual entity evaluations are spread out over the evaluation window you configured. The evaluation will finish before the next evaluation time is reached. Also note that CQL filters require their own independent evaluation, which can increase the time required to process the evaluation.
To manually trigger a Scorecard evaluation:
Navigate to the Scorecard's page in Cortex.
In the details panel on the right side of the page, click Evaluate now.
If a Scorecard is in the process of being evaluated, you can cancel it:
Navigate to the Scorecard's page in Cortex.
Click Cancel at the top of the page.
Use Cortex MCP to quickly gain insights into the performance of your Scorecard. Ask questions in natural language, such as: What is going on with my DORA Metrics Scorecard? or What steps can I take to improve my Eng Intelligence Scorecard scores?
First, ask a question about a Scorecard. The MCP will display which Cortex APIs are being used to find an answer:
Next, it will provide context to help you improve your scores:
For this example, after listing out the priorities, it also gave us a list of quick wins and a suggestion for a systematic approach:
From there, take action to remediate the issues and start meeting your defined standards.
Learn how to get started in Cortex MCP.
When you navigate to a Scorecard's page in Cortex, see high-level information, including the median levels achieved per entity, percent of entities at the highest level, and percent of entities that haven't achieved a level. In the side panel, see basic information, including the total number of services being evaluated, scheduled rules, and the next evaluation time.
When reviewing scores, identify where teams or entities have the most friction. Can you automate any processes via Cortex Workflows to improve efficiency?
Need to show progress to leadership? Click Share in the upper right corner. This copies a link to your clipboard, which you can share with anyone who has access to view this Scorecard in your workspace.
On the Scorecard page, click through each of the tabs to learn more about the Scorecard:
Under Scores, view each entity that is being scored via this Scorecard, the entity's current overall level, and its points or number of rules passed per level.
Point-based Scorecards will display progress based on points, while level-based Scorecards will display the number of passing rules.
For example, in the screen shot above, the first two entities achieved 50 points out of 50 possible points within the Bronze level of the Scorecard. They also each achieved 3 out of 3 possible points under the Silver level.
Click an entity to open a side panel with more information:
Next to the entity name: A link to its entity details page
At the top of the panel: The entity's current level, score, rank, percentile, and an overview of its rule progress
Under the Rules tab: A list of failing rules, passing rules, and rules not evaluated yet
Review this section closely to understand areas you can take action on.
Under the Exemptions tab: A list of exempted rules
Under the Progress tab: A graph showing the entity's progress
In the upper right corner of the list, click Filter to narrow the scope of the list. You can filter by domain, failing rules, groups, levels, owners, passing rules, and teams.
Click Level to select a different method to sort by, and choose whether to sort ascending or descending.
Under Overview, see which entity is the highest ranking, which entity is most at-risk, which rule is the most at-risk, the most improved entity, the entity with the worst drop, and the most-moved rule.
Under those metrics, see a graph displaying the progress of all rules. Click All rules above the graph to filter it by specific rules.
By default, the Trends and changes blocks show data from the last month. Click Last month to select a different timeframe.
Under Rules, view all of the Scorecard's rules, per level. Expand a rule to see its CQL expression:
Note that if you did not set a name for a rule when you created it, it will display here as a CQL expression.
Under Reports, click to view a Bird's Eye report, Progress report, or Report card.
Under Exemptions, view a list of all rule exemptions that have been requested.
Each request includes its current status, the rule, an expiration date, the reason for the request, and the user who requested it.
When a rule is exempted, entities are not marked as passing or failing — the rule simply does not apply to those entities.
You must have the the Configure Scorecard Exemptions permission to approve or deny exemptions.
To approve an exemption, click the checkmark icon. To deny an exemption, click the X icon:
Under Initiatives, see all Initiatives associated with the Scorecard.
A failed Scorecard rule gives teams actionable goals to make incremental improvements. Teams can focus on addressing the highest-impact failures first, track progress over time, and celebrate measurable wins, which reinforces a culture of improvement.
To remediate failed rules in a Scorecard:
Identify the failing rules
View the Scores tab on a Scorecard page to view a list of failing rules per entity. You can group by level to understand which entities have reached each level of the Scorecard. In this example, the group of entities has met the Bronze level, but they are failing rules for Silver and Gold:
Understand the cause
While viewing scores, click into an entity to open a side panel with more information about why rules are failing.
When configuring a rule, add a failure message to give clear steps on how you can remediate the rule if it fails.
Request exemptions if needed: In some cases, a rule may not apply to an entity. You can request to exempt an entity from the rule.
Track and prioritize issues
Use the Engineering homepage to see a prioritized to-do list of failing rules across your owned entities and Scorecards. This helps you focus on addressing the most impactful issues first.
Remediate the issues
Address the underlying issue for each failed rule. This could involve adding missing documentation, configuring a monitoring integration, or any other action required to meet the rule's criteria.
In the example screen shot above, the rule failed because PagerDuty is not configured for the entity. You can follow the instructions in Cortex's PagerDuty documentation to ensure that a PagerDuty service, schedule, or escalation policy is configured in the entity's descriptor YAML.
Use Initiatives to set specific deadlines for higher priority tasks. Cortex will notify owners and team members when deadlines are approaching, helping ensure accountability.
Re-evaluate
The Scorecard will automatically re-evaluate based on its schedule (or you can manually trigger an evaluation), allowing you to see the progress made and any remaining failing rules.
When reviewing your Scorecard evaluations, you may notice trends — some teams struggling in specific areas, or some entities often not meeting standards.
In addition to remediating individual rules, you may want to think about how you can improve larger processes to drive excellence in a scalable way across your organization.
Look for frequent challenging areas across teams. For example, is a team consistently failing to add runbooks to new services? Are there any failing rules relating to onboarding processes? When a standard is frequently being missed, it indicates an opportunity to automate that process with a Cortex Workflow.
You can use Workflows to streamline a set of repeatable steps, reducing human error and speeding efficiency for your teams.
For example, you could create a Workflow to streamline the process of Scaffolding new services, and include a User Input step that requires your users to ensure a runbook has been documented.
While viewing a group of entities that are failing rules, you can also look at organizational ways to support your teams or adjust processes.
Team-specific patterns
Are all of the failing entities owned by a newer team?
Ensure that the newer team members are not feeling overwhelmed and have completed any necessary training.
Ensure that the team understands their scope and priorities.
Does one team own more entities than other teams?
Ensure the workload is spread fairly across your teams.
If one teams owns more due to tribal knowledge, it is a signal to ensure team members are documenting their knowledge. It could also help you identify areas to conduct training sessions.
Are some teams or services successful across several areas?
When viewing teams that are successful or entities that meet all of their standards, find ways to recognize these achievements publicly. Public recognition reinforces good practices and motivates your team.
Service lifecycle patterns
Do older, legacy services have low scores, while newer ones align with standards?
This highlights modernization opportunities.
Adoption over time
Are scores trending upward across the quarter in line with company initiatives?
Tracking Scorecard trends across time shows whether your internal initiatives (like a push for better test coverage) are effective.
To promote specific standards, you can create an Initiative in Cortex based on a Scorecard, which will encourage remediation by a deadline.