PagerDuty

CatalogScorecards

PagerDuty is an incident response platform that allows developers to manage alerts, schedule rotations, define escalation policies, and more.

Integrating PagerDuty with Cortex allows you to:

• Pull in PagerDuty services, on-call schedules, and escalation policies
  • The on-call user or team will appear in the Current On-call block on an entity's details page.
  • You can also view on-call information on an entity page in its side panel under Integrations > On-call.
• Trigger incidents in PagerDuty directly from Cortex
• Automatically surface the most vital information about entity health and metadata when an incident is triggered by using Cortex's On-Call Assistant tool
  • The On-Call Assistant automatically notifies users via Slack when an incident is triggered. The notifications include runbooks, links, dependencies, and key information about the affected entity.
• View incidents from PagerDuty in an entity's event timeline
• View on-call information from PagerDuty in the dev homepage
• Use PagerDuty metrics in Eng Intelligence to understand key metrics and gain insight into services, incident response, and more.
• Create Scorecards that track progress and drive alignment on projects involving your on-call schedules and alerts

How to configure PagerDuty with Cortex

caution

If you do not see the settings page you're looking for, you may not have permission to access that page. Please contact your admin for assistance.

Prerequisites

Before getting started:

• Create a PagerDuty API key.
  • When adding the API key, you have the option to set read or write permissions.
    • Read: Enables Cortex to read any and all data from PagerDuty
    • Write: Allows users to trigger incidents from an entity page in Cortex, and enables On-Call Assistant

Configure the integration in Cortex

1. In Cortex, navigate to the PagerDuty settings page:
   1. In Cortex, click your avatar in the lower left corner, then click Settings.
   2. Under "Integrations", click PagerDuty.
2. Click Add PagerDuty configuration.
3. Configure the integration:
   • API key: Enter the API key you created in PagerDuty.
     • If the Read-only API key option is togged off, Cortex will use assume the provided API key has write permissions.
4. Click Save.

If you’ve set everything up correctly, you’ll see the option to Remove Integration in settings.

You can also use the Test configuration button to confirm that the configuration was successful. If your configuration is valid, you’ll see a banner that says “Configuration is valid. If you see issues, please see documentation or reach out to Cortex support.”

Enabling the On-call Assistant

At this stage, you can enable the Cortex On-call Assistant, which notifies users via Slack when an incident is triggered in PagerDuty. See the documentation for instructions: On-Call Assistant.

Note that On-Call Assistant will only work for service-level PagerDuty registrations since these notifications are related to affected services.

How to connect Cortex entities to PagerDuty

Discovery

By default, Cortex will use the entity tag (e.g. my-entity) or its name as the "best guess" for PagerDuty projects. For example, if your entity tag is my-entity, then the corresponding project in PagerDuty should also be my-entity.

If your PagerDuty projects don’t cleanly match the Cortex entity tag or name, you can override this in the Cortex entity descriptor.

Considerations for registering PagerDuty entities

Cortex recommends setting up PagerDuty at the service level by registering service entities with PagerDuty services, rather than configuring team entities with a PagerDuty schedule.

If PagerDuty is set up on a service level, you can see current on-call information listed within a given service's page. If PagerDuty is set up on the team level, you will only be able to view on-call rotation information from a team page.

Other benefits to setting up PagerDuty on a service level include:

• Structuring PagerDuty 1-1 with services enables better alert routing and analytics, something that organizations struggle more with when PagerDuty is set up on a team level.
• With a service-level setup, it’s easier to enforce all services to have a compliant on-call policy enacted in PagerDuty, especially when making use of Scorecards.
• The service-level setup is less reliant on team members tagging incidents with service information because services and incidents are already linked.
• You will gain the ability to get data from your Cortex catalog into PagerDuty, such as tier/criticality. By tying the service entities in the catalog with those in PagerDuty, you can automate processes and streamline severity protocols.

Editing the entity descriptor

For a given entity, you can define the PagerDuty service, schedule, or escalation policy within the entity’s YAML. You can only set up one of these three options per entity.

Each of these has the same field definitions.

Field	Description	Required
id	PagerDuty ID for service, schedule, or escalation policy	✓
type	SERVICE, SCHEDULE or ESCALATION_POLICY	✓

Define a PagerDuty service

Find the service ID value in PagerDuty under Configuration > Services. The URL for the service will contain the ID, for example: https://cortexapp.pagerduty.com/services/<ID>. You can only configure one service ID per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: SERVICE

Define a schedule

Find a schedule ID in PagerDuty under People > On-call schedules. Click the desired schedule to view its ID in the URL, for example: https://cortexapp.pagerduty.com/schedules#<ID>. You can only configure one schedule per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: SCHEDULE

Define an escalation policy

Find the escalation policy ID in PagerDuty under People > Escalation Policies. Click the desired policy to view its ID in the URL, for example: https://cortexapp.pagerduty.com/escalation_policies#<ID>. You can only configure one escalation policy per entity.

x-cortex-oncall:
  pagerduty:
    id: ASDF1234
    type: ESCALATION_POLICY

caution

You can only set up one of the three options above per entity.

Identity mappings

Cortex maps email addresses in your PagerDuty instance to email addresses that belong to team members in Cortex. When identity mapping is set up, users will be able to see their personal on-call status from the developer homepage.

Expected results

Entity pages

Once the PagerDuty integration is set up, you’ll be able to view on-call information on entity pages:

• Current on-call for an entity
• Escalation policy
• Service

tip

The escalation policy and PagerDuty service details are hyperlinked to the corresponding pages in your PagerDuty instance.

Dev homepage

The PagerDuty integration enables Cortex to pull on-call information into the on-call block on the Dev homepage. On-call data from PagerDuty is refreshed every 60 minutes.

Eng Intelligence

Cortex also pulls in metrics from PagerDuty for Eng Intelligence. This tool will display MTTR, incidents opened, and incidents opened per week.

Notifications

If you have a Slack integration set up, you can also use the /cortex oncall <tag> command to retrieve current on-call information. This feature works for both services and teams with registered PagerDuty schedules or escalation policies.

Scorecards and CQL

With the integration, you can create Scorecard rules and write CQL queries based on .

See more examples in the CQL Explorer in Cortex.

Check if on-call is set

Check if entity has a registered service, schedule, or escalation policy. If the service does not have any registrations in its entity descriptor, Cortex searches for PagerDuty services matching the tag defined in the entity's x-cortex-tag field.

Definition: oncall (==/!=) null

Example

For a Scorecard focused an production readiness, you can use this expression to make sure on-call is defined for entities:

oncall != null

This rule will pass if an entity has a service, schedule, or escalation policy set.

Forbidden contact methods

Number of users in each entity's escalation policy with missing or forbidden contact methods.

Allowed contact methods:

• "SMS"

• "PHONE"

• "EMAIL"

• "PUSH_NOTIFICATION"

• "SLACK"

  Definition: oncall.usersWithoutContactMethods(allowed=<allowed>, onlyCurrentOncall=<boolean>).length

Examples

For a Scorecard focused on ownership, you can use this expression to make sure users have required contact methods enabled:

oncall.usersWithoutContactMethods(allowed=["SMS", "PHONE"]).length == 0

This rule will pass if every user in an associated escalation policy has either SMS or phone calls enabled as their contact method.

You can also use this expression in the Query builder to find users that lack the required contact method:

oncall.usersWithoutContactMethods(allowed=["EMAIL"]) > 0

This query will surface users without email addresses.

If you want to check only current on-call users, you can use the onlyCurrentOncall parameter:

oncall.usersWithoutContactMethods(allowed=["EMAIL"], onlyCurrentOncall=true) > 0

When this parameter is set to false or omitted, the expression will check all users in the associated escalation policy for the next 3 months.

Incident response analysis

Get detailed on-call analysis stats for each entity:

• Mean assignment count
• Mean engaged seconds
• Mean engaged user count
• Mean seconds to engage
• Mean seconds to first ack
• Mean seconds to mobilize
• Mean seconds to resolve
• Total business-hour erruptions
• Total engaged seconds
• Total escalation count
• Total off-hour erruptions
• Total sleep-hour erruptions
• Total snoozed seconds
• Total incident count
• Up time percent

PagerDuty updates its analytics data once per day, and it can take up to 24 hours before new incidents appear in the analytics API.

Only works if entity has a registered PagerDuty service ID or if the PagerDuty service name matches the entity tag.

Definition: oncall.analysis(lookback = <duration>, priority = <List<String>>)

Examples

PagerDuty analytics can easily be used to craft rules for a DORA metrics Scorecard.

For mean time to acknowledge, you can use the meanSecondsToFirstAck schema definition:

oncall.analysis(lookback = duration("P7D"), priority = ["P1", "P2"]).meanSecondsToFirstAck <= 300

Entities will pass this rule if incidents in the last week were acknowledged within 5 minutes.

For mean time to resolve, you can use meanSecondsToResolve to make sure that incidents were handled within an hour:

oncall.analysis(lookback = duration("P7D"), priority = ["P1"]).meanSecondsToResolve < 3600

You can also use this expression to write a rule checking entities' change failure rate:

oncall.analysis(lookback = duration("P7D")).totalIncidentCount == 0

This rule will pass if there weren't any incidents in the last week.

Incidents

Get incident data for each entity:

• Assignee ID
• Created at
• Incident ID
• Last updated
• Resolved at
• Service ID
• Status

Only works if entity has a registered PagerDuty service ID or if the PagerDuty service name matches the entity tag.

Definition: oncall.incidents(lookback = <duration>)

Examples

For a Scorecard focused on service maturity or quality, you can use this expression to check the number of incidents opened in the last month:

oncall.incidents(lookback = duration("P1M")).length < 15

Entities will pass this rule if they have fewer than 15 incidents opened in the last month.

You can also use this expression to make sure there aren't incidents that remain open over the last month:

oncall.incidents(lookback=duration("P1M")).filter((incident) => incident.status.matches("TRIGGERED|ACKNOWLEDGED")).length < 1

Or you can check for incidents that took a certain amount of time to resolve:

oncall.incidents(lookback=duration("P1M")).filter((incident) => incident.createdAt.until(incident.resolvedAt) > duration("P-2D")).length < 2

Entities will pass this rule if there were 0 or 1 incidents in the last month that took more than 2 days to resolve.

Number of escalations

Number of escalation tiers in escalation policy.

Definition: oncall.numOfEscalations()

Example

This expression could be used in a Scorecard focused on production readiness or service maturity:

oncall.numOfEscalations() >= 2

This rule checks that there are at least two tiers in an escalation policy for a given entity, so that if the first on-call does not ack, there is a backup.

While making sure an on-call policy set is a rule that would be defined in a Scorecard's first level, a rule focused on escalation tiers would make more sense in a higher level.

On-call metadata

On-call metadata, including type, id, and name.

Definition: oncall.details()

Examples

To find all entities with a schedule-type on-call registration, you can use this expression in the Query builder:

oncall.details().type == "schedule"

If you're migrating on-call policies, you could use this rule to check for outdated policies. Let's say, for example, all outdated PagerDuty policies start with "Legacy" in their titles.

oncall.details().id.matches("Legacy*") == false

Entities with on-call policies that start with "Legacy" will fail, while those with other policy names will pass.

Trigger an incident

As described above under Editing the entity descriptor, a given entity can have a PagerDuty service, schedule, or escalation policy defined. Only entities with a PagerDuty service defined will include the option to trigger an incident directly from Cortex.

Your PagerDuty API key must include the write permission in order to trigger incidents from an entity.

While viewing an entity in Cortex, follow these steps to trigger an incident in PagerDuty:

1. In Cortex, navigate to an entity. On the left side of an entity details page, click Integrations > On-call.
2. In the upper right side of the entity's "On-call" page, click Trigger incident.
3. Configure the incident modal:
   • Summary: Enter a title for the incident.
   • Description: Enter a description of the incident.
   • Severity: Select a severity level.
4. At the bottom of the modal, click Trigger incident.
   • A confirmation screen will appear. In the confirmation, click the link to view the incident in PagerDuty.

Background sync

PagerDuty performs the following background jobs:

• On-call: On-call information displayed on the developer homepage is refreshed every 60 minutes.
• Services and incidents: Services used for automapping and active incidents viewable in the catalog are fetched approximately every 5 minutes, or however long the refresh takes.
• Users: User data for identity mapping is synced daily at 10 a.m. UTC.

Still need help?

The following are all the ways to get assistance from our customer engineering team. Please use the option that is best for your users:

• Email: help@cortex.io, or open a support ticket in the in app Resource Center
• Chat: Available in the Resource Center
• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your customer success manager.