Using On-call Assistant for incidents

Cortex’s On-call Assistant leverages the PagerDuty integration to automatically surface the most vital information about entity health and metadata when an incident is triggered. On-call Assistant notifies the user(s) responsible for an incident via Slack, including information about the affected entity, recent deployments, ownership, and links to get more details, including dependencies, runbooks, and logs.

On-call Assistant helps users respond to incidents in real time, simplifying the incident response process and helping to reduce MTTR. It can also drive adoption and engagement through links to the catalogs and Scorecards.

How On-call Assistant works

Notifications

When an incident is triggered in PagerDuty, On-call Assistant will notify relevant users via Slack. This alert will include information about the affected entity, deploy details, and ownership information so an on-call team member can reach out to other relevant parties about the incident.

On-call assistant notifies users via Slack.

Developers can access entity information that is already in Cortex directly from the Slack notification to quickly resolve issues. On-call Assistant provides a direct link to view the alert in PagerDuty, so you can also quickly access the incident from its source.

View runbooks and other links

The notification includes runbooks and other links.

View dependencies

The notification includes dependencies.

Enabling On-call Assistant

Prerequisites

• You must have the PagerDuty integration configured.

• You must create an API key in PagerDuty with the Write permission.

  • If you create an API key with Read-only permissions, you will also need to configure a webhook to get the On-call Assistant working.

Enable On-call Assistant in Cortex

To enable, navigate to Settings > PagerDuty and toggle on Enable On-call Assistant.

If you added PagerDuty API key with Write permissions, enabling the On-Call Assistant will create a webhook subscription in PagerDuty, allowing Cortex to receive events when incidents are triggered, escalated, or unacknowledged.

Configure a webhook for read-only API keys

If you added PagerDuty API key with Read-only permissions, you must also configure a webhook subscription.

1. In Cortex, on the PagerDuty settings page, click Configure webhook.

2. Copy the webhook URL. You will need this in the next steps.

3. In PagerDuty, add a new webhook.

   • Paste the Cortex webhook URL into the Webhook URL field.

   • Choose Account for scope type.

   • Select only the following in Event Subscription:

     • incident.escalated

     • incident.reopened

     • incident.triggered

     • incident.unacknowledged

4. A secret will be generated. Copy the secret. You will need it in the next step.

5. Navigate back to the browser window where your Cortex instance is open. In the Webhook configuration Secret field, enter the secret that you generated in PagerDuty.

6. Click Save at the bottom of the side panel.