LogoLogo
Login to CortexBook a DemoCortex Academycortex.io
  • Cortex Docs
  • Cortex Quick Start
  • Ingesting data into Cortex
    • Managing Entities
      • Adding entities
        • Add services
        • Add domains
        • Add teams
        • Add custom entity types
        • Defining dependencies
      • Entity details page
      • Defining ownership
      • Defining relationship types
      • Grouping entities
      • Adding external documentation
      • Adding Deploy data
      • Adding custom data
      • Viewing discovered entities
      • Archiving entities
      • Relationship graph
      • Using On-call Assistant for incidents
      • Managing Terraform infra in Cortex
    • Managing Catalogs
    • Integrations
      • Internally hosted integrations
      • ArgoCD
      • AWS
      • Azure DevOps
      • Azure Resources
      • BambooHR
      • Bitbucket
      • BugSnag
      • Buildkite
      • Checkmarx
      • CircleCI
      • ClickUp
      • Codecov
      • Coralogix
      • Custom webhook integrations
      • Datadog
      • Dynatrace
      • Entra ID (Azure AD)
      • FireHydrant
      • GitHub
      • GitLab
      • Google
      • Grafana
      • incident.io
      • Instana
      • Jenkins
      • Jira
      • Kubernetes
      • LaunchDarkly
      • Lightstep
      • Mend
      • Microsoft Teams
      • New Relic
      • Okta
      • Opsgenie
      • PagerDuty
      • Prometheus
      • Rollbar
      • Rootly
      • Sentry
      • ServiceNow
      • Slack
      • Snyk
      • SonarQube
      • Splunk Observability Cloud (SignalFx)
      • Splunk On-Call (VictorOps)
      • Sumo Logic
      • Veracode
      • Wiz
      • Workday
      • xMatters
  • Scorecards
    • Initiatives and Action items
      • Creating issues based on Initiatives
    • Scorecard rule exemptions
    • Scorecard rule filters
    • Scorecard examples
    • Scorecards as code
  • Reports
    • Executive report
    • All Scorecards report
    • Bird's eye report
    • Progress report
    • Report card
  • Eng Intelligence
    • Custom Metrics
    • Jira Metrics
    • Metrics Explorer (Beta)
  • Cortex Query Language (CQL)
    • Using CQL reports
    • Using JQ in Cortex
  • Workflows
    • Creating a Workflow
      • Workflows as code
    • Blocks
    • Running a Workflow
    • Registering a Scaffolder template
      • Scaffolder advanced usage
    • Using a Workflow to sync in ArgoCD
    • Kicking off a Jenkins pipeline in a Workflow
    • Calling internal service endpoints in a Workflow
  • Plugins
    • Creating a plugin
      • Creating a plugin proxy
    • Migrating Backstage plugins to Cortex
  • Engineering homepage
  • Workspace Settings
    • Using GitOps for Cortex
      • GitOps logs
    • Managing users
      • Roles and permissions
        • Custom roles
        • Team ownership entity editing
      • Configuring SSO
        • Microsoft Entra ID
        • Google
        • Other OIDC providers
        • Okta
          • Okta SCIM
      • Configuring identity mappings
      • Onboarding management
    • API keys, secrets, and tokens
      • Secrets
      • Personal tokens
    • Audit logs
    • Entity settings
      • Data verification
      • Auto archiving entities
    • IP allowlist
    • Notifications
      • Notification logs
    • Customizing your workspace
    • Using search in Cortex
  • Cortex API
    • REST API operations
      • API Keys
      • Audit Logs
      • Catalog Entities
      • Custom Data
        • Custom Data (Advanced)
      • Custom Events
      • Custom Metrics
      • Dependencies
      • Deploys
      • Discovery Audit
      • Docs
      • Eng Intel: User Labels
      • Entity Relationship Types (Beta)
      • Entity Relationships (Beta)
      • Entity Types
      • GitOps Logs
      • Groups
      • Initiatives
      • Integrations APIs
        • Azure Active Directory (Entra ID) API
        • Azure Resources API
        • AWS API
        • Azure DevOps API
        • CircleCI API
        • Coralogix API
        • Datadog API
        • GitHub API
        • GitLab API
        • incident.io API
        • LaunchDarkly API
        • New Relic API
        • PagerDuty API
        • Prometheus API
        • SonarQube API
      • IP Allowlist
      • Notification Logs
      • On call
      • Packages
      • Plugins
      • Queries
      • SCIM
      • Scorecards
      • Secrets
      • Team Hierarchies
      • Teams
      • Workflows
Powered by GitBook
On this page
  • How to configure PagerDuty with Cortex
  • Prerequisites
  • Configure the integration in Cortex
  • Enabling the On-call Assistant
  • How to connect Cortex entities to PagerDuty
  • Discovery
  • Considerations for registering PagerDuty entities
  • Editing the entity descriptor
  • Identity mappings
  • Expected results
  • Scorecards and CQL
  • Trigger an incident
  • Background sync
  • Still need help?​

Was this helpful?

Export as PDF
  1. Ingesting data into Cortex
  2. Integrations

PagerDuty

Last updated 1 month ago

Was this helpful?

is an incident response platform that allows developers to manage alerts, schedule rotations, define escalation policies, and more.

Integrating PagerDuty with Cortex allows you to:

  • Pull in PagerDuty services, on-call schedules, and escalation policies

    • The on-call user or team will appear in the Current On-call block on an entity's details page.

    • You can also view on-call information on an entity page in its side panel under Integrations > On-call.

  • directly from Cortex

  • Automatically surface the most vital information about entity health and metadata when an incident is triggered by using Cortex's On-Call Assistant tool

    • The On-Call Assistant automatically notifies users via Slack when an incident is triggered. The notifications include runbooks, links, dependencies, and key information about the affected entity.

  • View incidents from PagerDuty in an entity's event timeline

  • View on-call information from PagerDuty in the dev homepage

  • Use PagerDuty metrics in Eng Intelligence to understand key metrics and gain insight into services, incident response, and more.

  • Create that track progress and drive alignment on projects involving your on-call schedules and alerts

How to configure PagerDuty with Cortex

Prerequisites

Before getting started:

  • Create a .

    • When adding the API key, you have the option to set read or write permissions.

      • Read: Enables Cortex to read any and all data from PagerDuty

      • Write: Allows users to from an entity page in Cortex, and enables On-Call Assistant

Configure the integration in Cortex

    1. In Cortex, click your avatar in the lower left corner, then click Settings.

    2. Under "Integrations", click PagerDuty.

  1. Click Add configuration.

  2. Configure the integration:

    • API key: Enter the API key you created in PagerDuty.

      • If the Read-only API key option is togged off, Cortex will use assume the provided API key has write permissions.

  3. Click Save.

If you’ve set everything up correctly, you’ll see the option to Remove Integration in settings.

You can also use the Test configuration button to confirm that the configuration was successful. If your configuration is valid, you’ll see a banner that says “Configuration is valid. If you see issues, please see documentation or reach out to Cortex support.”

Enabling the On-call Assistant

How to connect Cortex entities to PagerDuty

Discovery

If your PagerDuty projects don’t cleanly match the Cortex entity tag or name, you can override this in the Cortex entity descriptor.

Considerations for registering PagerDuty entities

Cortex recommends setting up PagerDuty at the service level by registering service entities with PagerDuty services, rather than configuring team entities with a PagerDuty schedule.

If PagerDuty is set up on a service level, you can see current on-call information listed within a given service's page. If PagerDuty is set up on the team level, you will only be able to view on-call rotation information from a team page.

Other benefits to setting up PagerDuty on a service level include:

  • Structuring PagerDuty 1-1 with services enables better alert routing and analytics, something that organizations struggle more with when PagerDuty is set up on a team level.

  • With a service-level setup, it’s easier to enforce all services to have a compliant on-call policy enacted in PagerDuty, especially when making use of Scorecards.

  • The service-level setup is less reliant on team members tagging incidents with service information because services and incidents are already linked.

  • You will gain the ability to get data from your Cortex catalog into PagerDuty, such as tier/criticality. By tying the service entities in the catalog with those in PagerDuty, you can automate processes and streamline severity protocols.

Editing the entity descriptor

For a given entity, you can define the PagerDuty service, schedule, or escalation policy within the entity’s YAML. You can only set up one of these three options per entity.

Each of these has the same field definitions.

Field
Description
Required

id

PagerDuty ID for service, schedule, or escalation policy

✓

type

SERVICE, SCHEDULE or ESCALATION_POLICY

✓

Define a PagerDuty service

Find the service ID value in PagerDuty under Configuration > Services. The URL for the service will contain the ID, for example: https://cortexapp.pagerduty.com/services/. You can only configure one service ID per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: SERVICE

Define a schedule

Find a schedule ID in PagerDuty under People > On-call schedules. Click the desired schedule to view its ID in the URL, for example: https://cortexapp.pagerduty.com/schedules#. You can only configure one schedule per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: SCHEDULE

Define an escalation policy

Find the escalation policy ID in PagerDuty under People > Escalation Policies. Click the desired policy to view its ID in the URL, for example: https://cortexapp.pagerduty.com/escalation_policies#. You can only configure one escalation policy per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: ESCALATION_POLICY

You can only set up one of the three options above per entity.

Identity mappings

Expected results

Entity pages

The escalation policy and PagerDuty service details are hyperlinked to the corresponding pages in your PagerDuty instance.

Dev homepage

The PagerDuty integration enables Cortex to pull on-call information into the on-call block on the Dev homepage. On-call data from PagerDuty is refreshed every 60 minutes.

Eng Intelligence

Cortex also pulls in metrics from PagerDuty for Eng Intelligence. This tool will display MTTR, incidents opened, and incidents opened per week.

Notifications

Scorecards and CQL

With the integration, you can create Scorecard rules and write CQL queries based on .

Check if on-call is set

Check if entity has a registered service, schedule, or escalation policy. If the service does not have any registrations in its entity descriptor, Cortex searches for PagerDuty services matching the tag defined in the entity's x-cortex-tag field.

Definition: oncall (==/!=) null

Example

For a Scorecard focused an production readiness, you can use this expression to make sure on-call is defined for entities:

oncall != null

This rule will pass if an entity has a service, schedule, or escalation policy set.

Forbidden contact methods

Number of users in each entity's escalation policy with missing or forbidden contact methods.

Allowed contact methods:

  • "SMS"

  • "PHONE"

  • "EMAIL"

  • "PUSH_NOTIFICATION"

  • "SLACK"

Definition: oncall.usersWithoutContactMethods(allowed=<allowed>, onlyCurrentOncall=<boolean>).length

Example

For a Scorecard focused on ownership, you can use this expression to make sure users have required contact methods enabled:

oncall.usersWithoutContactMethods(allowed=["SMS", "PHONE"]).length == 0

This rule will pass if every user in an associated escalation policy has either SMS or phone calls enabled as their contact method.

You can also use this expression in the Query builder to find users that lack the required contact method:

oncall.usersWithoutContactMethods(allowed=["EMAIL"]) > 0

This query will surface users without email addresses.

If you want to check only current on-call users, you can use the onlyCurrentOncall parameter:

oncall.usersWithoutContactMethods(allowed=["EMAIL"], onlyCurrentOncall=true) > 0

When this parameter is set to false or omitted, the expression will check all users in the associated escalation policy for the next 3 months.

Incident response analysis
  • Mean assignment count

  • Mean engaged seconds

  • Mean engaged user count

  • Mean seconds to engage

  • Mean seconds to first ack

  • Mean seconds to mobilize

  • Mean seconds to resolve

  • Total business-hour erruptions

  • Total engaged seconds

  • Total escalation count

  • Total off-hour erruptions

  • Total sleep-hour erruptions

  • Total snoozed seconds

  • Total incident count

  • Up time percent

PagerDuty updates its analytics data once per day, and it can take up to 24 hours before new incidents appear in the analytics API.

Only works if entity has a registered PagerDuty service ID or if the PagerDuty service name matches the entity tag.

Definition: oncall.analysis(lookback = <duration>, priority = <List<String>>)

Examples

PagerDuty analytics can easily be used to craft rules for a DORA metrics Scorecard.

For mean time to acknowledge, you can use the meanSecondsToFirstAck schema definition:

oncall.analysis(lookback = duration("P7D"), priority = ["P1", "P2"]).meanSecondsToFirstAck <= 300

Entities will pass this rule if incidents in the last week were acknowledged within 5 minutes.

For mean time to resolve, you can use meanSecondsToResolve to make sure that incidents were handled within an hour:

oncall.analysis(lookback = duration("P7D"), priority = ["P1"]).meanSecondsToResolve < 3600

You can also use this expression to write a rule checking entities' change failure rate:

oncall.analysis(lookback = duration("P7D")).totalIncidentCount == 0

This rule will pass if there weren't any incidents in the last week.

Incidents

Get incident data for each entity:

  • Assignee ID

  • Created at

  • Incident ID

  • Last updated

  • Resolved at

  • Service ID

  • Status

Only works if entity has a registered PagerDuty service ID or if the PagerDuty service name matches the entity tag.

Definition: oncall.incidents(lookback = <duration>)

Examples

For a Scorecard focused on service maturity or quality, you can use this expression to check the number of incidents opened in the last month:

oncall.incidents(lookback = duration("P1M")).length < 15

Entities will pass this rule if they have fewer than 15 incidents opened in the last month.

You can also use this expression to make sure there aren't incidents that remain open over the last month:

oncall.incidents(lookback=duration("P1M")).filter((incident) => incident.status.matches("TRIGGERED|ACKNOWLEDGED")).length < 1

Or you can check for incidents that took a certain amount of time to resolve:

oncall.incidents(lookback=duration("P1M")).filter((incident) => incident.createdAt.until(incident.resolvedAt) > duration("P-2D")).length < 2

Entities will pass this rule if there were 0 or 1 incidents in the last month that took more than 2 days to resolve.

Number of escalations

Number of escalation tiers in escalation policy.

Definition: oncall.numOfEscalations()

Example

This expression could be used in a Scorecard focused on production readiness or service maturity:

oncall.numOfEscalations() >= 2

This rule checks that there are at least two tiers in an escalation policy for a given entity, so that if the first on-call does not ack, there is a backup.

While making sure an on-call policy set is a rule that would be defined in a Scorecard's first level, a rule focused on escalation tiers would make more sense in a higher level.

On-call metadata

On-call metadata, including type, id, and name.

Definition: oncall.details()

Examples

To find all entities with a schedule-type on-call registration, you can use this expression in the Query builder:

oncall.details().type == "schedule"

If you're migrating on-call policies, you could use this rule to check for outdated policies. Let's say, for example, all outdated PagerDuty policies start with "Legacy" in their titles.

oncall.details().id.matches("Legacy*") == false

Entities with on-call policies that start with "Legacy" will fail, while those with other policy names will pass.

Trigger an incident

Your PagerDuty API key must include the write permission in order to trigger incidents from an entity.

While viewing an entity in Cortex, follow these steps to trigger an incident in PagerDuty:

  1. In Cortex, navigate to an entity. On the left side of an entity details page, click On-call & incidents.

  2. In the upper right side of the entity's "On-call" page, click Trigger incident.

  3. Configure the incident modal:

    • Summary: Enter a title for the incident.

    • Description: Enter a description of the incident.

    • Severity: Select a severity level.

  4. At the bottom of the modal, click Trigger incident.

    • A confirmation screen will appear. In the confirmation, click the link to view the incident in PagerDuty.

Background sync

PagerDuty performs the following background jobs:

  • On-call: On-call information displayed on the developer homepage is refreshed every 60 minutes.

  • Services and incidents: Services used for automapping and active incidents viewable in the catalog are fetched approximately every 5 minutes, or however long the refresh takes.

  • Users: User data for identity mapping is synced daily at 10 a.m. UTC.

The following options are available to get assistance from the Cortex Customer Engineering team:

  • Chat: Available in the Resource Center

  • Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.

In Cortex, navigate to the :

At this stage, you can enable the Cortex On-call Assistant, which notifies users via Slack when an incident is triggered in PagerDuty. See the documentation for instructions: .

Note that On-Call Assistant will only work for since these notifications are related to affected services.

By default, Cortex will use the (e.g. my-entity) or its name as the "best guess" for PagerDuty projects. For example, if your entity tag is my-entity, then the corresponding project in PagerDuty should also be my-entity.

Cortex maps email addresses in your PagerDuty instance to email addresses that belong to team members in Cortex. When is set up, users will be able to see their personal on-call status from the developer homepage.

Once the PagerDuty integration is set up, you’ll be able to view current on-call information in the "on-call" block on an . In the left sidebar of an entity, click On-call & incidents to view on-call information, escalation policy, service, and incidents.

If you have a Slack integration set up, you can also use the /cortex oncall to retrieve current on-call information. This feature works for both services and teams with registered PagerDuty schedules or escalation policies.

See more examples in the in Cortex.

Get detailed for each entity:

As described above under , a given entity can have a PagerDuty service, schedule, or escalation policy defined. Only entities with a PagerDuty service defined will include the option to trigger an incident directly from Cortex.

Still need help?

Email: , or open a support ticket in the in app Resource Center

PagerDuty settings page
On-Call Assistant
identity mapping
entity details page
CQL Explorer
on-call analysis stats
​
help@cortex.io
PagerDuty
PagerDuty API key
Trigger incidents in PagerDuty
Scorecards
trigger incidents
service-level PagerDuty registrations
Editing the entity descriptor
entity tag
Slack Bot command