Seasonal readiness Scorecard

For organizations that have a busy season, they can implement a seasonal readiness Scorecard to proactively ensure operational efficiency and consistency.

You can use our example rules as a starting point, and you can learn more below about how customer H&R Block reduced bugs by 50% year-over-year by implementing a seasonal readiness Scorecard.

Example rules

The focus areas of this type of Scorecard may be be specific to your organization's needs, but as general guidance, we would recommend creating a Scorecard that contains rules in these key focus areas:

Operational readiness

Example rules:

• Entity has an owner ownership != null

• Runbook is linked links("runbook").length > 0

• Logs are linked links("logs").length > 0

Reliability

Example rules:

• SLO coverage slos().length > 0

  • Before configuring this rule, set up an integration that supports querying SLOs: Datadog, Dynatrace, Google, Lightstep, New Relic, Prometheus, Splunk Observability Cloud (SignalFx), or Sumo Logic.

• Test coverage minimum met captures("test-coverage", sonarqube.metric("coverage") >= 80)

  • Before configuring this rule, set up an integration with SonarQube. The Codecov integration also supports querying code coverage. You can surface these code coverage metrics in failure messages for failed Scorecard rules.

• Zero critical and high vulnerabilities in Mend mend.vulnerabilities(risk = ["Critical", "High"]).length == 0

  • Before configuring this rule, set up an integration with Mend. Other integrations that support querying on vulnerabilities include Apiiro, Checkmarx, GitHub, Mend, Semgrep, and Snyk.

• On-call rotation has at least one escalation oncall.numOfEscalations() > 1

  • Before configuring this rule, set up an integration that supports querying on-call: Opsgenie, PagerDuty, Splunk On-Call (VictorOps), or xMatters.

• Datadog monitor is set datadog.monitors().length > 0

  • Before configuring this rule, set up an integration with Datadog.

Security

Example rules:

• At least one required approval to merge git.numOfRequiredApprovals() > 0

  • Before configuring this rule, set up a version control integration: Azure DevOps, Bitbucket, GitHub, or GitLab.

• Code updated in the last week git.lastCommit().freshness < duration("P7D")

  • Before configuring this rule, set up a version control integration.

• 0 critical Snyk issues snyk.issues(severity=["CRITICAL"], fixability=["FIXABLE"]) <= 0

  • Before configuring this rule, set up an integration with Snyk. Other integrations that support querying on vulnerabilities include Apiiro, Checkmarx, GitHub, Mend, Semgrep, and Snyk.

Get started with an example Scorecard

You can use the example rules above as a starting point for your Scorecard.

In this example Scorecard, the levels are organized by focus area. Scorecard levels are progressive, with the last level including the highest priority rules. Based on your organization's needs, you may want to reorder rules, add or remove rules, or rename the levels.

You can create this Scorecard in the UI, or you can upload the YAML file via API or GitOps.

UI

Create example Scorecard via the UI

Step 1: Create the Scorecard and configure its basic details

1. In Cortex, navigate to Scorecards and click +Create Scorecard. Start with a blank Scorecard.

2. Configure the basic details.

   • Include a name that helps your users understand the purpose of the Scorecard (e.g., Season readiness) and a description.

   • Learn more about configuring basic fields for Scorecards in Create a Scorecard.

Step 2: Add levels and rules

1. Under Define evaluation rules, add levels. In our example, we added levels based on focus area:

   • Operational readiness

   • Reliability

   • Security

2. Under a level, click +Add rule to add a rule.

   • For each level, add the example rules listed earlier in these instructions.

3. At the bottom of the page, click Save Scorecard.

API or GitOps

Example: H&R Block's Season Readiness Scorecard

As described in their IDPCON talk, H&R Block implemented a Season Readiness Scorecard ahead of tax season to drive operational consistency. Their Scorecard helped them automatically measure their organization's top priority areas: Security, resiliency, code coverage, and software development lifecycle best practices.

They included rules to verify on-call coverage, escalation depth, monitoring setup, SLOs, and package versions.

Outcome

They reduced incidents and bugs by 50% year over year, while automating readiness tracking previously handled manually by six program managers.

Their next steps

To continue proactively meeting organizational benchmarks, they also created Scaffolder templates with built-in compliance and coding standards. This allowed them to ensure that all new projects were scaffolded with the best practices baked in.

See the IDPCON session

H&R Block's Manager of Technology spoke at IDPCON to explain how their organization leveled up developer experience with automation in Cortex: