Splunk Observability Cloud (SignalFx)

Cortex connects to many third-party vendors whose system interfaces frequently change. As a result, integration behavior or configuration steps may shift without notice. If you encounter unexpected issues, check with your system administrator or refer to the vendor's documentation for the most current information. Additionally, integration sync times vary and are subject to scheduling overrides and timing variance.

Splunk Observability Cloud (formerly known as SignalFx) is a monitoring and analytics platform that allows customers to evaluate, visualize, automate, and alert on metrics.

Integrating Splunk Observability Cloud with Cortex allows you to:

• View SLO information on entity pages in Cortex

• Create Scorecards that include rules related to your Splunk Observability Cloud SLOs

How to configure Splunk Observability Cloud with Cortex

Prerequisites

Before getting started:

• Create an access token in Splunk.

Configure the integration in Cortex

1. In Cortex, navigate to the Splunk Observability settings page:

   1. Click Integrations from the main nav. Search for and select Splunk Observability.

2. Click Add configuration.

3. Configure the integration form:

   • Realm: Enter your Splunk Observability realm.

     • You can find it in the URL for your instance, e.g., https://app.{REALM}.signalfx.com/#/metrics.

   • Access token: Enter the access token you generated in Splunk.

4. Click Save.

After saving your configuration, you are redirected to the Splunk Observability integration settings page in Cortex. In the upper right corner of the page, click Test configuration to ensure Splunk Observability was configured properly.

How to connect Cortex entities to Splunk Observability SLOs

Editing the entity descriptor

SLOs can be defined in the entity descriptor.

You can pull in data about SLOs and manage them in Cortex by defining relevant SLIs through queries, along with a threshold and a comparator. Cortex will pull data from Splunk Observability Cloud and roll up the query with a specified rollup function.

x-cortex-slos:
  signalfx:
    - query: sf_metric:"jvm.memory.max" AND area:"nonheap"
      rollup: AVERAGE
      target: 5120000
      lookback: P1Y
      operation: "<="

Field	Description	Required
query	Elasticsearch query for your metric: Use sf_metric to filter by the given metric and add additional dimensions to narrow the searchQueries resulting in multiple datasets will be rolled up according to rollup	✓
rollup	SUM or AVERAGE	✓
target	Target number for the SLO	✓
lookback	ISO-8601 duration (P[n]Y[n]M[n]DT[n]H[n]M[n]S)	✓
operation	>, <, =, =	✓

Using the Splunk Observability integration

Viewing SLO information on an entity

When an SLO is defined in an entity's descriptor, SLO information appears in the Monitoring block on an entity details page's overview.

Monitoring

• Click Monitoring > Data overview in the entity's sidebar to see the query, target, and current value for each SLO from any integrations you're pulling SLOs from. Each SLO also includes the period of time the SLO is being calculated for. For example, if the time listed is "7 days ago," then the SLO is looking at the time range starting 7 days ago to now.\

  
• Click Monitoring > Splunk Observability to view a graph of SLO performance over time.\

  

Scorecards and CQL

With the Splunk Observability Cloud integration, you can create Scorecard rules and write CQL queries based on Splunk Observability Cloud SLOs.

See more examples in the CQL Explorer in Cortex.

SLOs

SLOs associated with the entity via ID or tags. You can use this data to check whether an entity has SLOs associated with it, and if those SLOs are passing.

Definition: slos: List<SLO>

Example

In a Scorecard, you can use this expression to make sure an entity is passing its SLOs:

slos().all((slo) => slo.passing) == true

Use this expression to make sure latency Service Level Indicator (SLI) value is above 99.99%:

slos().filter((slo) => slo.name.matchesIn("latency") and slo.sliValue >= 0.9999).length > 0

View integration logs

This feature is available in Cortex cloud.

While viewing an integration's settings page, select the Logs tab to view error logs from the last 7 days. You can filter the logs list by configuration and by operation (for example, you could filter to view errors surfaced only via Scorecards).

Click into a row to get more information, including time stamp, status code, full error, and request path.

Still need help?​

The following options are available to get assistance from the Cortex Customer Engineering team:

• Email: [e-mail], or open a support ticket in the in app Resource Center

• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your Customer Success Manager.