Using On-call Assistant for incident notifications

Cortex’s On-call Assistant leverages the PagerDuty integration to automatically surface the most vital information about entity health and metadata when an incident is triggered. On-call Assistant notifies the user(s) responsible for an incident via Slack, including information about the affected entity, recent deployments, ownership, and links to get more details, including dependencies, runbooks, and logs.

On-call Assistant helps users respond to incidents in real time, simplifying the incident response process and helping to reduce MTTR. It can also drive adoption and engagement through links to the catalogs and Scorecards.

On-call Assistant only works for service-level PagerDuty registrations, since these notifications are related to affected services. Refer to this article for more information.

How On-call Assistant notifications are sent in Slack

When a PagerDuty incident is triggered, Cortex identifies the responsible on-call user and notifies them in Slack. The alert includes information about the affected entity, deploy details, and ownership information. A direct link is included to view the alert in PagerDuty, so the incident can be quickly assessed from its source.

Individual users - On-call Assistant sends a direct message (DM) to the on-call user. Users don't need to install the Cortex Slack app or select a channel. Cortex looks up the user's Slack account using the email address on their Cortex account, then sends the DM directly. The only requirement is that the user's email in Cortex matches their Slack workspace email.

Teams - Notifications are sent to the Slack channel or channels configured on the entity.

On-call assistant notifies users via Slack.

Viewing runbooks and other links

Viewing dependencies

Enabling On-call Assistant

Prerequisites

1. An API key in PagerDuty with the Write permission.

   • If you choose to bypass this prerequisite and create an API key with Read-only permissions, you also need to configure a webhook to get the On-call Assistant working.

2. The PagerDuty integration must be configured.

   • If using an API key with Read-only permission, be sure to toggle on Read-only API key in the PagerDuty side panel.

     
3. If sending notifications to teams, Slack channels must be configured on the entity.

Enabling On-call Assistant in Cortex

1. From the main sidebar, select Integrations.

2. Locate PagerDuty, then click Settings.

3. Toggle on Enable On-call Assistant.

   

Configuring a webhook for read-only API keys

If your PagerDuty API key has Read-only permissions, you must also configure a webhook subscription.

Step 1: Enabling On-call Assistant in Cortex

1. From the main sidebar, select Integrations.

2. Locate PagerDuty, then click Settings.

3. Toggle on Enable On-call Assistant. The Configure webhook side panel opens.

Step 2: Adding a webhook in PagerDuty

1. In PagerDuty, add a new webhook:

   1. Under Webhook URL, paste the Cortex webhook URL. The Cortex webhook URL can be found in the Configure webhook side panel.

      
   2. From the Scope type dropdown, select Account.

   3. Under Event Subscription, clear all checkboxes EXCEPT:

      • incident.escalated

      • incident.reopened

      • incident.triggered

      • incident.unacknowledged

   4. Click Add webhook. The Webhook subscription created window opens.

   5. Copy the webhook secret. Don't skip this step! You'll need the webhook secret to complete the setup.

   6. Click OK.

Step 3: Completing setup in Cortex

1. Go back to the browser window where your Cortex instance is open.

2. In the Configure webhook side panel, paste the webhook secret in the Secret field.

   
3. Click Save.