AI maturity
From AI tools to AI outcomes
AI coding assistants and internal AI tooling are only as effective as the infrastructure supporting them. Cortex gives you the visibility, automation, and measurement you need to drive real adoption, not just tool access.
AI maturity in Cortex means:
Every team knows which AI tools they're expected to use and whether they're meeting that standard
Access requests and onboarding are automated, not manual
Leaders can see whether AI adoption is actually improving engineering performance
Work through the five steps below to configure your workspace.
Step 1: Ingesting data and establishing ownership
AI maturity tracking only works if your services, infrastructure, and other entities are in Cortex and have clear owners. Without this foundation, Scorecards have nothing to evaluate, Workflows have no targets, and Eng Intelligence has no baselines.
What to do
Import your services, infrastructure, and other entities into Cortex. Assign an owner (team or individual) to each entity. Then connect the integrations that give Cortex visibility into how those entities are built, tested, and deployed.
For AI-specific entities like models or pipelines, consider creating a custom entity type (e.g. "AI Model") to group and track them separately.
Integrations to prioritize
Configure integrations in these categories to get the most signal on AI adoption:
Version control (GitHub, GitLab, Bitbucket, Azure DevOps) - Track PR size, review times, and merge frequency before and after AI tool rollout.
Project management (Jira, GitHub, ClickUp, Azure DevOps) - Compare lead time for AI-influenced work items vs. traditional work.
External docs - Link runbooks and documentation to your entities so teams can access AI diagnostics and guidelines directly from Cortex.
You'll know this step is done when...
Every entity in scope has an assigned owner, at least one version control integration is active, and your AI-specific entities (models, pipelines, etc.) are visible in the Cortex catalog.
Common issues
Ownership gaps - Entities get imported but ownership isn't set, so Scorecards can't route notifications and Eng Intelligence can't group data by team. Bulk-setting ownership via the API or a CSV import is faster than doing it entity by entity.
Integrations connected, but not fully scoped - GitHub might be connected at the org level but missing specific repos, so version control rules silently fail. Double-check that the integration covers all the repos your entities reference.
Step 2: Creating an AI maturity Scorecard
Scorecards automatically check whether your services and teams are meeting AI adoption standards. The AI maturity template gives you a starting point—a tiered set of rules (Bronze, Silver, Gold) based on common industry practices—that you can customize to fit your organization.
What to do
Create a Scorecard using the AI maturity template. The template includes rules that verify AI tooling is properly configured across your repositories. Examples include:
AI instructions file present
A file like AGENTS.md, .cursorrules, or CLAUDE.md exists in the repository
Copilot enabled
GitHub Copilot is enabled for the team owning this service
AI-assisted PR reviews configured
An AI code review tool (e.g. CodeRabbit, GitHub Copilot code review) is active on the repo
AI anomaly detection integrated
The service is connected to an AI-based monitoring or alerting tool
Customize for your stack
Delete rules that don't apply, re-weight rules to reflect your priorities, and add custom rules using CQL to check for tooling specific to your organization. Each rule can be assigned more points to signal its relative importance.
You'll know this step is done when...
The Scorecard is live, scoped to the right entities, and at least a handful of services are showing scores, even if many are failing initially.
Common issues
Scorecard is scoped to the wrong entities - If the entity filter is too broad, you'll get noise from entities that shouldn't be measured against AI standards. If it's too narrow, key services get missed. Review the filter before publishing.
Template rules reference integrations you haven't configured yet - For example, a Copilot adoption rule will fail for every entity if the GitHub integration isn't set up. Cross-check the template rules against your active integrations in Step 1 before going live.
Everything is failing on day one - This is normal. A Scorecard full of red on launch isn't a problem, it's the baseline. Communicate that to teams upfront so it doesn't cause alarm.
Step 3: Automating AI tooling requests with Workflows
A Workflow is an automated, multi-step process that runs entirely within Cortex. It lets teams define tasks, trigger actions, collect input, and route approvals in one place, turning complex, multi-tool operations into a single repeatable experience. For AI maturity, this means you can enforce standards automatically rather than relying on manual checklists or Slack threads.
Identifying good candidates for Workflow automation
Before building, look at where friction exists in your AI maturity program. Scorecards and Eng Intelligence are useful here: if a team consistently fails the same rules, or cycle time data shows work getting stuck at a particular handoff, that's a strong signal that the underlying process should be a Workflow.
What to do
Create a Workflow, configure its settings, and add blocks. You can create a Workflow based on a template or create one from scratch.
Two high-value Workflows to set up
Automating access requests
Build a Workflow that handles GitHub Copilot (or other AI tool) access requests end-to-end: a developer submits a request, the Workflow routes it for approval, and access is provisioned automatically on approval. See the example Workflow for Copilot access automation.
Enforcing standards at service creation
Use Scaffolder templates to ensure every new service is created with baseline AI standards already in place: AI instructions files, anomaly detection integrations, and links to AI diagnostic runbooks.
You'll know this step is done when...
Your AI tool access request Workflow (e.g. GitHub Copilot) is active and has completed at least one successful end-to-end run—request, approval, provisioning—and your Scaffolder template includes AI-specific baseline standards so new services start with them in place. If new services still need manual setup after creation, or engineers are requesting AI tool access informally over Slack, this step isn't done.
Common issues
Undefined approval chain - If the Workflow routes a request but nobody is designated to approve it, requests stall and engineers abandon the process and find workarounds.
Scaffolder templates that reference AI tools or integrations not yet fully rolled out - A template that includes AI-based anomaly detection configuration for a tool your org hasn't standardized on yet will confuse service owners immediately.
Bypassing the Scaffolder by cloning repos or creating services directly - The Scorecard catches the gaps eventually, but you'll be cleaning up after the fact rather than preventing them.
Step 4: Configuring Cortex MCP
Cortex MCP lets engineers query your Cortex workspace in natural language from their MCP client (Claude, Cursor, VS Code, and others). Instead of switching to a browser to look up service owners, on-call rotations, or Scorecard status, engineers get answers inline while they work.
Relevant for AI Maturity specifically:
Find ownership fast - Ask Who owns the embeddings pipeline? to immediately identify which team is accountable for an AI service, without digging through wikis or Slack history.
Check Scorecard status - Ask What are the failing rules for the inference service AI Maturity Scorecard? to get a prioritized list of gaps to fix.
Get remediation suggestions - Ask What are quick wins for my AI Maturity Scorecard? and MCP will return actionable next steps based on your current scores.
What to do
Configure the Cortex MCP. It can be hosted locally or remotely. Refer to Using the Cortex MCP for information on starting a new chat, crafting effective prompts, and using Eng Intelligence metrics in chat.
You'll know this step is done when...
At least one engineer on your team has MCP configured and can successfully query Cortex in natural language from their editor.
Common issues
MCP not connected to the right workspace - If the MCP is pointed at the wrong Cortex workspace or authenticated with credentials that have limited permissions, queries will return incomplete or no results. Verify the connection returns accurate data for a service you know well before rolling it out.
User labels haven't been configured yet (Step 5 prerequisite) - MCP queries about "which teams are using AI tools" won't return useful results since Cortex doesn't know who those teams are yet. See user labels
Questions that are too vague - MCP works best with specific, scoped questions. How are my AI services doing? returns less useful results than What are quick wins for my AI Governance Scorecard? Train teams on the kinds of questions that get actionable answers.
Teams not aware it exists - MCP adoption tends to be low when it's announced once and forgotten. Engineers default to existing habits (Slack, wikis, asking a colleague) unless MCP is introduced in the context of a workflow they already use, like a Scorecard remediation or an incident investigation.
Step 5: Establishing Eng Intelligence baselines
Before you roll out AI tooling broadly, capture baselines in DORA metrics (deployment frequency, MTTR, change failure rate, lead time) and velocity metrics. These numbers are your before state and you'll need them to show impact.
What to do
Apply user labels to identify teams. User labels let you group engineers into cohorts and compare metrics side by side, e.g. deployment frequency for Copilot users vs. non-users over the same period.
This makes it straightforward to answer the question leadership will ask: Is the AI tooling investment actually working?
You'll know this step is done when...
You have baseline DORA and velocity metrics recorded, and user labels are applied to any teams already using AI tools.
Common issues
No baseline captured before rollout - Once AI tools are adopted and metrics shift, you lose the before/after comparison if you didn't pull numbers first.
User labels not configured - The comparison between AI-adopting teams and non-adopting teams is one of the most valuable things Eng Intelligence can show you, but it requires labels to be set up and assigned before the measurement period starts. Setting them up after the fact means your cohorts aren't clean.
Tracking the wrong metrics for an AI maturity use case - Cycle time and MTTR are useful, but the more telling signals are PR size, review time, and merge frequency from your version control integration. These show whether AI coding tools are actually changing how engineers work. If those integrations weren't connected in Step 1, they won't be available here.
No regular reviewer - Eng Intelligence surfaces trends, but someone needs to own the cadence of reviewing them and acting on regressions. Without that ownership established now, the dashboards go unread.
Next steps
Once your workspace is configured, learn how to keep the momentum going with AI maturity in action.
Last updated
Was this helpful?