AI readiness in action
Your Cortex workspace is configured; now the work shifts from setup to continuous improvement. This page covers how to identify and fix gaps, respond to Scorecard failures, and measure the impact of your AI readiness program over time.
Identifying and fixing Scorecard gaps
Finding failing rules
Cortex surfaces failing Scorecard rules in three places:
Engineering homepage - Each engineer sees a personalized view of their failing rules, open PRs, and action items. This is the default starting point for daily work. See Engineering homepage
Notifications - Set up Slack, Teams, or email alerts so engineers are proactively notified when a rule they own starts failing — rather than discovering it in a review. See Notifications
Scorecard reports - For a team- or org-wide view of pass rates and trends, use Scorecard reports or the Scorecard evaluate view.
Remediating failing rules
For quick individual fixes - Use Cortex MCP. Ask What are my quick wins for the AI Readiness Scorecard? and you'll get a prioritized list of rules your services are failing that are low effort to fix. This is useful for engineers who want to make incremental progress without a formal project.
For team-wide gaps - Create an Initiative. An Initiative assigns specific rules to teams with a deadline, sends automated reminders, and tracks progress in one place. You can also auto-create tickets in Jira, GitHub, ClickUp, or Azure DevOps from Initiative action items. See Creating issues from Initiatives
Example - Your security team notices that several AI model services are failing the "AI model security scanning" rule at the Gold level. Rather than chasing down individual owners, they create an Initiative — "Enable vulnerability scanning on all AI model services" — targeting that rule, with a deadline 3 weeks out. Cortex sends automated reminders to each service owner and tracks completion. The security team can see real-time progress without status meetings.
For systemic issues - Look for patterns across failing rules. If 15 services are all failing "Runbook linked," the fix isn't 15 separate tasks — it's a Workflow that scaffolds new services with a runbook template by default, and a one-time push to existing services. Use the Scorecard evaluate view to spot these patterns.
Example - A platform team reviews their Scorecard report and notices that every service created in the last 6 months is missing SLO configuration. Investigation reveals that the service creation Workflow never included SLO setup. They update the Scaffolder template to include SLO configuration, then create an Initiative targeting all existing services to backfill it within 30 days.
Measuring impact with Eng Intelligence
Eng Intelligence gives you the historical and org-wide view needed to answer the question: Is our AI Readiness program actually working?
Connecting Scorecard milestones to engineering metrics
The most useful thing you can do is compare metrics before and after your AI readiness Scorecard went live. Pull your baseline DORA metrics (deployment frequency, MTTR, change failure rate, cycle time) from Eng Intelligence, then watch how they shift as more services reach Bronze, Silver, and Gold.
What to look for
Deployment frequency increases as teams gain confidence from automated quality gates
MTTR decreases as better runbooks, on-call configuration, and ownership clarity reduce time to diagnose incidents
Change failure rate drops as test coverage and security scanning rules enforce quality at commit time
Cycle time shortens as Cortex MCP reduces time engineers spend searching for service context
Example
After launching an AI readiness Scorecard, an engineering org tracks MTTR weekly in the DORA dashboard. Over 8 weeks, as more services reach Silver level (which requires SLOs and on-call configuration), MTTR drops from 47 minutes to 28 minutes. Leadership uses this data to justify expanding the AI Readiness program to all business units.
When metrics aren't moving
If DORA metrics stay flat after several weeks, use Eng Intelligence to dig into why:
Low Scorecard pass rates suggest teams haven't engaged with the rules yet. Check notification settings and consider an Initiative with a tight deadline.
High pass rates but flat metrics suggest your Scorecard rules may not be targeting the right behaviors. Revisit which rules map to outcomes you care about
Metrics improving for some teams but not others. Use the team-level filter in Data Explorer to isolate which teams are lagging and why
Turning insights into action
When you spot a gap:
Identify the pattern in Eng Intelligence or Scorecard reports.
Determine whether it needs a Scorecard rule change, a new Initiative, or a Workflow fix.
Assign and track it—through an Initiative if it's a team task, or a Scorecard rule update if it's a standards change.
Measure again in 2–4 weeks.
Last updated
Was this helpful?