# Splunk Observability Cloud (SignalFx)

{% hint style="info" %}
Cortex connects to many third-party vendors whose system interfaces frequently change. As a result, integration behavior or configuration steps may shift without notice. If you encounter unexpected issues, check with your system administrator or refer to the vendor's documentation for the most current information. Additionally, integration sync times vary and are subject to scheduling overrides and timing variance.
{% endhint %}

Splunk Observability Cloud ([formerly known as SignalFx](https://www.splunk.com/en_us/newsroom/press-releases/2019/splunk-closes-acquisition-of-cloud-monitoring-leader-signalfx.html)) is a monitoring and analytics platform that allows customers to evaluate, visualize, automate, and alert on metrics.

Integrating Splunk Observability Cloud with Cortex allows you to:

* [View SLO information on entity pages](#viewing-slo-information-on-an-entity) in Cortex
* Create [Scorecards](#scorecards-and-cql) that include rules related to your Splunk Observability Cloud SLOs

## How to configure Splunk Observability Cloud with Cortex

### Prerequisites

Before getting started:

* Create an [access token](https://docs.splunk.com/observability/en/admin/authentication/authentication-tokens/org-tokens.html#create-an-access-token) in Splunk.

### Configure the integration in Cortex

1. In Cortex, navigate to the [Splunk Observability settings page](https://app.getcortexapp.com/admin/integrations/splunk-observability):
   1. Click **Integrations** from the main nav. Search for and select **Splunk Observability**.
2. Click **Add configuration**.
3. Configure the integration form:
   * **Realm**: Enter your Splunk Observability realm.
     * You can find it in the URL for your instance, e.g., `https://app.{REALM}.signalfx.com/#/metrics`.
   * **Access token**: Enter the access token you generated in Splunk.
4. Click **Save**.

After saving your configuration, you are redirected to the Splunk Observability integration settings page in Cortex. In the upper right corner of the page, click **Test configuration** to ensure Splunk Observability was configured properly.

To modify the integration configuration, see [Modifying an existing integration configuration](/ingesting-data-into-cortex/integrations.md#modifying-an-existing-integration-configuration).

## How to connect Cortex entities to Splunk Observability SLOs

### Editing the entity descriptor

SLOs can be defined in the [entity descriptor](/ingesting-data-into-cortex/entities-overview/entities.md#defining-entities-via-yaml-file).

You can pull in data about SLOs and manage them in Cortex by defining relevant SLIs through queries, along with a threshold and a comparator. Cortex will pull data from Splunk Observability Cloud and roll up the query with a specified `rollup` function.

```yaml
x-cortex-slos:
  signalfx:
    - query: sf_metric:"jvm.memory.max" AND area:"nonheap"
      rollup: AVERAGE
      target: 5120000
      lookback: P1Y
      operation: "<="
```

| Field       | Description                                                                                                                                                                                                                                                                                           | Required |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------: |
| `query`     | [Elasticsearch query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html) for your metric: Use `sf_metric` to filter by the given metric and add additional dimensions to narrow the searchQueries resulting in multiple datasets will be rolled up according to `rollup` |   **✓**  |
| `rollup`    | `SUM` or `AVERAGE`                                                                                                                                                                                                                                                                                    |   **✓**  |
| `target`    | Target number for the SLO                                                                                                                                                                                                                                                                             |   **✓**  |
| `lookback`  | [ISO-8601 duration](https://www.digi.com/resources/documentation/digidocs/90001488-13/reference/r_iso_8601_duration_format.htm) (`P[n]Y[n]M[n]DT[n]H[n]M[n]S`)                                                                                                                                        |   **✓**  |
| `operation` | `>`, `<`, `=`, `=`                                                                                                                                                                                                                                                                                    |   **✓**  |

## Using the Splunk Observability integration

### Viewing SLO information on an entity

When an SLO is defined in an entity's descriptor, SLO information appears in the **Monitoring** block on an [entity details page's](/ingesting-data-into-cortex/entities-overview/entities/details.md) overview.

**Monitoring**

* Click **Monitoring > Data overview** in the entity's sidebar to see the query, target, and current value for each SLO from any integrations you're pulling SLOs from. Each SLO also includes the period of time the SLO is being calculated for. For example, if the time listed is "7 days ago," then the SLO is looking at the time range starting 7 days ago to now.\\

  <div align="left"><figure><img src="/files/TX5CWyQphq5wQoGuSdot" alt="View SLOs in the entity sidebar under Monitoring > Data overview." width="563"><figcaption></figcaption></figure></div>
* Click **Monitoring > Splunk Observability** to view a graph of SLO performance over time.\\

  <div align="left"><figure><img src="/files/kmV2AxYU1wYfnCz1mery" alt="View a graph of SLO performance."><figcaption></figcaption></figure></div>

### Scorecards and CQL

With the Splunk Observability Cloud integration, you can create Scorecard rules and write CQL queries based on Splunk Observability Cloud SLOs.

See more examples in the [CQL Explorer](https://app.getcortexapp.com/admin/cql-explorer) in Cortex.

<details>

<summary>SLOs</summary>

SLOs associated with the entity via ID or tags. You can use this data to check whether an entity has SLOs associated with it, and if those SLOs are passing.

**Definition:** `slos: List<SLO>`

**Example**

In a Scorecard, you can use this expression to make sure an entity is passing its SLOs:

```
slos().all((slo) => slo.passing) == true
```

Use this expression to make sure latency Service Level Indicator (SLI) value is above 99.99%:

```
slos().filter((slo) => slo.name.matchesIn("latency") and slo.sliValue >= 0.9999).length > 0
```

</details>

### View integration logs <a href="#still-need-help" id="still-need-help"></a>

{% hint style="info" %}
This feature is available in Cortex cloud.
{% endhint %}

While viewing an integration's settings page, click the **Logs** tab to view error logs from the last 7 days. You can filter the logs list by configuration and by operation (for example, you could filter to view errors surfaced only via Scorecards).

<div align="left" data-with-frame="true"><figure><img src="/files/x8JmoPqXZTJ7YHeFJpOA" alt="The &#x27;Logs&#x27; tab on an integration&#x27;s settings page shows error information over the past 7 days."><figcaption></figcaption></figure></div>

Click into a row to get more information, including time stamp, status code, full error, and request path.

## Still need help?[​](https://docs.cortex.io/docs/reference/integrations/aws#still-need-help) <a href="#still-need-help" id="still-need-help"></a>

The following options are available to get assistance from the Cortex Customer Engineering team:

* **Email**: <help@cortex.io>, or open a support ticket in the in app Resource Center
* **Slack**: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a `:ticket:` reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cortex.io/ingesting-data-into-cortex/integrations/splunk-observability.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
