Splunk On-Call (VictorOps)
Splunk On-Call (formerly known as VictorOps) is an alert and on-call management platform. By integrating Splunk On-Call with Cortex, you drive insights into on-call rotations and escalation policies.
Setup and configuration
Getting started
In order to connect Cortex to your Splunk On-Call instance, you’ll need to create a Splunk On-Call API key.
If the key is granted Read-only permissions, Cortex will only perform GET requests.
You'll also need your API ID to set up the integration. Navigate to the Integrations page in your Splunk On-Call portal and go to the API tab. Your API ID will be available above your API keys (including the one you just created).
Configuration
Once you have your API ID and an API key, go to Splunk On-Call settings in Cortex.
If you do not see the settings page you're looking for, you likely don't have the proper permissions and need to contact your admin.
Enter the following information to set up the integration:
- API ID: API ID available under Integrations → API in your Splunk On-Call portal.
- API key: API key you generated in the prior step. Also available under Integrations → API in your Splunk On-Call portal.
- Organization slug: Splunk On-Call organization slug. Your org slug can be found at the end of the URL for your Splunk On-Call portal (e.g.
https://portal.victorops.com/dash/<EXAMPLE-SLUG>)
If you’ve set everything up correctly, you’ll see the option to Remove Integration in settings.
You can also use the Test configuration button to confirm that the configuration was successful. If your configuration is valid, you’ll see a banner that says “Configuration is valid. If you see issues, please see documentation or reach out to Cortex support.”
Registration
Entity descriptor
With the Splunk On-Call integration, you can tie on-call rotations to entities by defining the x-cortex-oncall block with your schedule metadata.
x-cortex-oncall:
victorops:
type: SCHEDULE
id: team-abcd12345
| Field | Description | Required |
|---|---|---|
type | Type of on-call data (in this case, SCHEDULE) | ✓ |
id | ID for the team assigned to the given schedule | ✓ |
You can find the team ID in the Splunk On-Call portal on the teams page (e.g. https://portal.victorops.com/dash/cortex-app#/team/<TEAM_ID>/users).
Expected Results
Entity pages
Once a Splunk On-Call schedule is defined in an entity descriptor, the user or team who is on call will appear in the Current On-call block on that entity's details page.
You can also find on-call information for a given entity on the On-call page under the Integrations tab.
Scorecards and CQL
With the Splunk On-Call integration, you can create Scorecard rules and write CQL queries based on Splunk On-Call schedules.
Check if on-call is set
Check if entity has a registered team.
Definition: oncall (==/!=) null
Example
For a Scorecard focused an production readiness, you can use this expression to make sure on-call is defined for entities:
oncall != null
This rule will pass if an entity has a service, schedule, or escalation policy set.
Number of escalations
Number of escalation tiers in escalation policy.
Definition: oncall.numOfEscalations()
Example
This expression could be used in a Scorecard focused on production readiness or service maturity:
oncall.numOfEscalations() >= 2
This rule checks that there are at least two tiers in an escalation policy for a given entity, so that if the first on-call does not ack, there is a backup.
On-call metadata
On-call metadata, including type, ID, and name.
Definition: oncall.details()
Example
You can use this expression in the Query builder to find all entities with an on-call rotation that includes a specific team. Let's say we want to find all entities that the "Sample Team" team is on-call for and the team's ID in Splunk On-Call is sample-team1234. Our query would then be:
oncall.details().id == "sample-team1234"
Still need help?
The following are all the ways to get assistance from our customer engineering team. Please use the option that is best for your users:
- Email: help@cortex.io, or open a support ticket in the in app Resource Center
- Chat: Available in the Resource Center
- Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a
:ticket:reaction to a question in Slack, and the team will respond directly.
Don’t have a Slack channel? Talk with your customer success manager.