# Splunk On-Call (VictorOps)

{% hint style="info" %}
Cortex connects to many third-party vendors whose system interfaces frequently change. As a result, integration behavior or configuration steps may shift without notice. If you encounter unexpected issues, check with your system administrator or refer to the vendor's documentation for the most current information. Additionally, integration sync times vary and are subject to scheduling overrides and timing variance.
{% endhint %}

Splunk On-Call (formerly known as [VictorOps](https://www.splunk.com/en_us/about-splunk/acquisitions/splunk-on-call.html)) is an alert and on-call management platform.

Integrating Cortex with Splunk On-Call allows you to:

* Pull in on-call rotation data and escalation policies
  * The on-call user or team will appear in the **Current On-call** block on an entity's details page.
  * You can also view on-call information on an entity page in its side panel under **Integrations > On-call**.
* Create [Scorecards](#scorecards-and-cql) that track progress and drive alignment on projects involving your on-call schedule

## How to configure Splunk On-Call with Cortex

### Prerequisites

Before getting started:

* Create a [Splunk On-Call API key](https://help.victorops.com/knowledge-base/api/).
  * Note: If the key is granted `Read-only` permissions, Cortex will only perform `GET` requests.
* Obtain your Splunk API ID.
  * In your Splunk On-Call portal, navigate to the **Integrations** page then click the **API** tab. Your API ID is displayed above your API keys.

### Configure the integration in Cortex

1. In Cortex, navigate to the [Splunk On-call settings page](https://app.getcortexapp.com/admin/integrations/splunk-oncall):
   1. Click **Integrations** from the main nav. Search for and select **Splunk On-call**.
2. Click **Add configuration**.
3. Configure the VictorOps integration form:
   * **API ID**: Enter your API ID from Splunk On-Call.
   * **API key**: Enter your API key from Splunk On-Call.
   * **Organization slug**: Enter your Splunk On-Call organization slug.
     * This can be found at the end of the URL for your Splunk On-Call portal (e.g., `https://portal.victorops.com/dash/`)
   * **Client ID** and **Client secret**: Enter the client ID and secret associated with the application link you created in the previous steps.
4. Click **Save**.

To modify the integration configuration, see [Modifying an existing integration configuration](https://docs.cortex.io/ingesting-data-into-cortex/integrations/..#modifying-an-existing-integration-configuration).

## How to connect Cortex entities to Splunk On-Call

### Editing the entity descriptor

With the Splunk On-Call integration, you can tie on-call rotations to entities under the `x-cortex-oncall` block with your schedule metadata. You can use the team ID, or you can filter beyond the team ID and use the policy ID. Using the policy ID is helpful if your team has multiple policies available.

Team ID in entity descriptor:

```yaml
x-cortex-oncall:
  victorops:
    type: SCHEDULE
    id: team-abcd12345
```

Policy ID in entity descriptor:

```yaml
x-cortex-oncall:
  victorops:
    type: SCHEDULE
    id: team-abcd12345
    policyIds:
      - 12345
      - 56789
```

| Field  | Description                                     | Required |
| ------ | ----------------------------------------------- | :------: |
| `type` | Type of on-call data (in this case, `SCHEDULE`) |   **✓**  |
| `id`   | ID for the team assigned to the given schedule  |   **✓**  |

You can find the team ID in the Splunk On-Call portal on the teams page (e.g., `https://portal.victorops.com/dash/cortex-app#/team//users`).

## Using the Splunk On-Call integration

#### Entity pages

Once a Splunk On-Call schedule is defined in an entity descriptor, the user or team who is on call will appear in the **Current On-call** block on that [entity's details page](https://docs.cortex.io/ingesting-data-into-cortex/entities-overview/entities/details).

You can also find on-call information for a given entity on the **On-call & incidents** page in the entity's sidebar.

### Scorecards and CQL

With the Splunk On-Call integration, you can create Scorecard rules and write CQL queries based on Splunk On-Call schedules.

See more examples in the [CQL Explorer](https://app.getcortexapp.com/admin/cql-explorer) in Cortex.

<details>

<summary>Check if on-call is set</summary>

Check if entity has a registered team.

**Definition:** `oncall (==/!=) null`

**Example**

For a Scorecard focused an production readiness, you can use this expression to make sure on-call is defined for entities:

```
oncall != null
```

This rule will pass if an entity has a service, schedule, or escalation policy set.

</details>

<details>

<summary>Number of escalations</summary>

Number of escalation tiers in escalation policy.

**Definition:** `oncall.numOfEscalations()`

**Example**

This expression could be used in a Scorecard focused on production readiness or service maturity:

```
oncall.numOfEscalations() >= 2
```

This rule checks that there are at least two tiers in an escalation policy for a given entity, so that if the first on-call does not ack, there is a backup.

</details>

<details>

<summary>On-call metadata</summary>

On-call metadata, including type, ID, and name.

**Definition:** `oncall.details()`

**Example**

You can use this expression in the Query builder to find all entities with an on-call rotation that includes a specific team. Let's say we want to find all entities that the "Sample Team" team is on-call for and the team's ID in Splunk On-Call is `sample-team1234`. Our query would then be:

```
oncall.details().id == "sample-team1234"
```

</details>

### View integration logs <a href="#still-need-help" id="still-need-help"></a>

## Still need help?[​](https://docs.cortex.io/docs/reference/integrations/aws#still-need-help) <a href="#still-need-help" id="still-need-help"></a>

The following options are available to get assistance from the Cortex Customer Engineering team:

* **Email**: <help@cortex.io>, or open a support ticket in the in app Resource Center
* **Slack**: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a `:ticket:` reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cortex.io/ingesting-data-into-cortex/integrations/splunk-oncall.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.