What good looks like: A real example
After rolling out an AI maturity Scorecard, an engineering org notices something unexpected: change failure rate is climbing, even as AI-assisted coding adoption increases. After talking to teams, the cause is clear: developers have started trusting AI-generated code without reviewing it as carefully. Here’s how they use Cortex to address it:- They add a Scorecard rule to the Silver level: Change failure rate stays below 5%.
- A week later, several teams are still failing the rule. They create an Initiative (“Improve quality of PR reviews”) with a 30-day deadline and automated reminders.
- Jira tickets are auto-generated for each failing team, so there’s no ambiguity about what’s required.
- By the end of the month, change failure rate is back down, and the Scorecard rule passes across all affected services.
1
Measure
2
Identify gaps
3
Drive action
4
Verify improvement
Day-to-day: Staying on top of Scorecard gaps
Scorecard failures surface in multiple places so nothing slips through:- Engineering homepage - Engineers see their failing Scorecard rules and open action items each time they log in. No need to go looking for gaps; they’re front and center. See Engineering homepage
- Notifications - Configure Slack, Microsoft Teams, or email alerts when a rule starts failing. See Notifications
- Scorecard reports - View aggregate progress across teams in reports, or drill into a specific Scorecard in Evaluate Scorecards.
Remediating a failing rule
There are three ways to remediate a failing rule:- Use Cortex MCP - Ask What are quick wins for my AI Maturity Scorecard? and get a prioritized remediation list without leaving your editor.
- Create an Initiative - Initiatives let you assign a specific fix to the responsible team with a deadline and automated reminders. You can also auto-generate issues in Jira, ClickUp, Azure DevOps, or GitHub based on Initiative tasks. See Creating issues from Initiatives
- Fix it directly - For simple gaps (a missing file, an unset integration), the engineer can resolve it immediately from the homepage action item and the rule rechecks automatically.
Measuring impact with Eng Intelligence
Use Eng Intelligence to track whether AI adoption is translating into engineering performance gains.See before and after
Pull up the DORA dashboard or Velocity dashboard and compare metrics from before and after your AI maturity Scorecard launched. Concrete shifts in deployment frequency, lead time, or MTTR are the evidence leadership needs to justify further investment.Compare AI users vs. non-users
If you created user labels to tag teams using AI tools, use the Data Explorer to filter by those labels. Comparing cohorts side by side makes the productivity delta visible, and makes the case for broader rollout to teams that haven’t adopted yet.Spot regressions before they become problems
Watch for metrics that move in the wrong direction after AI tool adoption. A rise in change failure rate, a slowdown in review time, or a spike in MTTR often points to a behavior change worth investigating. When you spot one, add a Scorecard rule to track it and create an Initiative if teams need a nudge.Keeping the Scorecard current
AI tooling evolves fast. Revisit your AI maturity Scorecard regularly (at least quarterly) to:- Retire rules for tools your org has moved away from
- Add rules for new tools or practices you want to standardize
- Raise the bar on existing tiers as your baseline improves