Seasonal readiness Scorecard

For organizations that have a busy season, they can implement a seasonal readiness Scorecard to proactively ensure operational efficiency and consistency. You can use our example rules as a starting point, and you can learn more below about how customer H&R Block reduced bugs by 50% year-over-year by implementing a seasonal readiness Scorecard.

Example rules

The focus areas of this type of Scorecard may be be specific to your organization’s needs, but as general guidance, we would recommend creating a Scorecard that contains rules in these key focus areas:

Operational readiness

Example rules:

Entity has an owner
ownership != null
Runbook is linked
links("runbook").length > 0
Logs are linked
links("logs").length > 0

Reliability

Example rules:

SLO coverage
slos().length > 0
- Before configuring this rule, set up an integration that supports querying SLOs: Datadog, Dynatrace, Google, Lightstep, New Relic, Prometheus, Splunk Observability Cloud (SignalFx), or Sumo Logic.
Test coverage minimum met
captures("test-coverage", sonarqube.metric("coverage") >= 80)
- Before configuring this rule, set up an integration with SonarQube. The Codecov integration also supports querying code coverage. You can surface these code coverage metrics in failure messages for failed Scorecard rules.
Zero critical and high vulnerabilities in Mend
mend.vulnerabilities(risk = ["Critical", "High"]).length == 0
- Before configuring this rule, set up an integration with Mend. Other integrations that support querying on vulnerabilities include Apiiro, Checkmarx, GitHub, Mend, Semgrep, and Snyk.
On-call rotation has at least one escalation
oncall.numOfEscalations() > 1
- Before configuring this rule, set up an integration that supports querying on-call: Opsgenie, PagerDuty, Splunk On-Call (VictorOps), or xMatters.
Datadog monitor is set
datadog.monitors().length > 0
- Before configuring this rule, set up an integration with Datadog.

Security

Example rules:

At least one required approval to merge
git.numOfRequiredApprovals() > 0
- Before configuring this rule, set up a version control integration: Azure DevOps, Bitbucket, GitHub, or GitLab.
Code updated in the last week
git.lastCommit().freshness < duration("P7D")
- Before configuring this rule, set up a version control integration.
0 critical Snyk issues
snyk.issues(severity=["CRITICAL"], fixability=["FIXABLE"]) <= 0
- Before configuring this rule, set up an integration with Snyk. Other integrations that support querying on vulnerabilities include Apiiro, Checkmarx, GitHub, Mend, Semgrep, and Snyk.

Get started with an example Scorecard

You can use the example rules above as a starting point for your Scorecard. In this example Scorecard, the levels are organized by focus area. Scorecard levels are progressive, with the last level including the highest priority rules. Based on your organization’s needs, you may want to reorder rules, add or remove rules, or rename the levels. You can create this Scorecard in the UI, or you can upload the YAML file via API or GitOps.

UI
API or GitOps

Create example Scorecard via the UI

Step 1: Create the Scorecard and configure its basic details

In Cortex, navigate to Scorecards and click +Create Scorecard. Start with a blank Scorecard.
Configure the basic details.
- Include a name that helps your users understand the purpose of the Scorecard (e.g., Season readiness) and a description.
- Learn more about configuring basic fields for Scorecards in Create a Scorecard.

Step 2: Add levels and rules

Under Define evaluation rules, add levels. In our example, we added levels based on focus area:
- Operational readiness
- Reliability
- Security
Under a level, click +Add rule to add a rule.
- For each level, add the example rules listed earlier in these instructions.
At the bottom of the page, click Save Scorecard.

Create example Scorecard via GitOps or API

When following a GitOps approach, you can add a Scorecard YAML file to your .cortex/scorecards directory in your version control repository. Note that GitOps must be enabled for Scorecards in your GitOps settings.You could also use the Cortex API, where you can submit a Scorecard definition in YAML.

Seasonal readiness Scorecard YAML

Use the YAML file below to add this Scorecard to your workspace via the API or via a GitOps flow.

tag: seasonal-readiness
name: Seasonal Readiness
description: Ensuring services are ready before the busy season
draft: false
notifications:
  enabled: true
  scoreDropNotificationsEnabled: false
exemptions:
  enabled: true
  autoApprove: false
  userSpecificNotifications: false
ladder:
  name: Default Ladder
  levels:
  - name: Operational readiness
    rank: 1
    description: Minimum seasonal readiness
    color: "#EBA27E"
  - name: Reliability
    rank: 2
    description: Top level readiness rules
    color: "#B3B3B3"
  - name: Security
    rank: 3
    color: "#33393F"
rules:
- title: On-call rotation has at least one escalation
  expression: oncall.numOfEscalations() > 1
  weight: 1
  level: Reliability
- title: Logs are linked
  expression: links("logs").length > 0
  weight: 1
  level: Operational readiness
- title: Entity has owner
  description: "For accountability, ensure entities have clear ownership"
  expression: ownership != null
  weight: 1
  level: Operational readiness
- title: Test coverage minimum met
  expression: "captures(\"test-coverage\", sonarqube.metric(\"coverage\") >= 80)"
  weight: 1
  level: Reliability
- title: SLO coverage
  expression: slos().length > 0
  weight: 1
  level: Reliability
- title: Datadog monitoring is set
  expression: datadog.monitors().length > 0
  weight: 1
  level: Reliability
- title: At least one required approval to merge
  expression: git.numOfRequiredApprovals() > 0
  weight: 1
  level: Security
- title: Runbook is linked
  expression: links("runbook").length > 0
  weight: 1
  level: Operational readiness
- title: Low number of Snyk issues
  expression: snyk.issues().length < 5
  weight: 1
  level: Security
- title: Zero critical and high vulnerabilities
  expression: "captures(\"critical-vulns\", custom(\"critical_vulnerabilities\"))\
    \ == 0 AND captures(\"high-vulns\", custom(\"high_vulnerabilities\")) == 0"
  weight: 1
  level: Reliability
- title: Code updated in the last week
  expression: git.lastCommit().freshness < duration("P7D")
  weight: 1
  level: Security
filter:
  kind: GENERIC
  types:
    include:
    - service

Example: H&R Block’s Season Readiness Scorecard

As described in their IDPCON talk, H&R Block implemented a Season Readiness Scorecard ahead of tax season to drive operational consistency. Their Scorecard helped them automatically measure their organization’s top priority areas: Security, resiliency, code coverage, and software development lifecycle best practices. They included rules to verify on-call coverage, escalation depth, monitoring setup, SLOs, and package versions.

Outcome

They reduced incidents and bugs by 50% year over year, while automating readiness tracking previously handled manually by six program managers.

Their next steps

To continue proactively meeting organizational benchmarks, they also created Scaffolder templates with built-in compliance and coding standards. This allowed them to ensure that all new projects were scaffolded with the best practices baked in.

See the IDPCON session

H&R Block’s Manager of Technology spoke at IDPCON to explain how their organization leveled up developer experience with automation in Cortex:

​Example rules

​Get started with an example Scorecard

​Create example Scorecard via the UI

​Step 1: Create the Scorecard and configure its basic details

​Step 2: Add levels and rules

​Create example Scorecard via GitOps or API

​Example: H&R Block’s Season Readiness Scorecard

​Outcome

​Their next steps

​See the IDPCON session

Example rules

Get started with an example Scorecard

Create example Scorecard via the UI

Step 1: Create the Scorecard and configure its basic details

Step 2: Add levels and rules

Create example Scorecard via GitOps or API

Example: H&R Block’s Season Readiness Scorecard

Outcome

Their next steps

See the IDPCON session