Splunk Observability Cloud (SignalFx)

CatalogScorecards

Splunk Observability Cloud (formerly known as SignalFx) is a monitoring and analytics platform that allows customers to evaluate, visualize, automate and alert on metrics. By integrating Splunk Observability Cloud with Cortex, you can drive insights into SLOs.

Setup and configuration

Getting started

In order to connect Cortex to your Splunk Observability Cloud instance, you’ll need to create an access token.

Configuration

Once you've created an access token, go to Splunk Observability Cloud settings in Cortex and add the following information:

• Realm: Your Splunk Observability Cloud realm, which you can find in the URL for your instance (e.g. https://app.{REALM}.signalfx.com/#/metrics).
• Access token: The access token you created in Splunk.

caution

If you do not see the settings page you're looking for, you likely don't have the proper permissions and need to contact your admin.

If you’ve set everything up correctly, you’ll see the option to Remove Integration in settings.

You can also use the Test configuration button to confirm that the configuration was successful. If your configuration is valid, you’ll see a banner that says “Configuration is valid. If you see issues, please see documentation or reach out to Cortex support.”

Registration

Entity descriptor

You can pull in data about SLOs and manage them in Cortex by defining relevant SLIs through queries, along with a threshold and a comparator. Cortex will pull data from Splunk Observability Cloud and roll up the query with a specified rollup function.

x-cortex-slos:
  signalfx:
    - query: sf_metric:"jvm.memory.max" AND area:"nonheap"
      rollup: AVERAGE
      target: 5120000
      lookback: P1Y
      operation: "<="

Field	Description	Required
query	Elasticsearch query for your metric: Use sf_metric to filter by the given metric and add additional dimensions to narrow the search Queries resulting in multiple datasets will be rolled up according to rollup	✓
rollup	SUM or AVERAGE	✓
target	Target number for the SLO	✓
lookback	ISO-8601 duration (P[n]Y[n]M[n]DT[n]H[n]M[n]S)	✓
operation	>, <, =, <=, or >=	✓

Expected results

Entity pages

When an SLO is defined in an entity's descriptor, you'll see high-level information in the Operations block under the Overview tab.

In the Operations tab, you can find more detailed data about SLOs in the Service level objectives block.

Click into the Pass or Fail blocks (if they have a nonzero value) to open a modal with more detailed information: the SLO query, its target(s), and its current value.

Integrations - Splunk Observability Cloud

You can access more detailed information about your SLO(s) in the Splunk Observability Cloud tab under Integrations in the sidebar.

In addition to the query, target(s), and current value for each SLO, you'll find a graph of its performance over time.

Event timeline

Scorecards and CQL

With the Splunk Observability Cloud integration, you can create Scorecard rules and write CQL queries based on Splunk Observability Cloud SLOs.

SLOs

SLOs associated with the entity via ID or tags. You can use this data to check whether an entity has SLOs associated with it and if those SLOs are passing.

• History

• ID

• Name

• Operation

• Remaining budget

• SLI value

  • Datum
  • Timeseries

• SLO target

• Source

• Thresholds

  • Name
  • Threshold

  Definition: slos()

Examples

For a Scorecard focused on operational maturity, this expression can be used to make sure an entity has associated SLOs in Datadog:

slos().length > 0

This rule checks that there is at least one SLO is set up. While this rule makes sense in a Scorecard's first level, a rule checking the status of the SLO would make sense in a higher level:

slos().all((slo) => slo.passing)

Entities will pass this rule if all SLOs associated with it have the "passing" status.

Still need help?

The following are all the ways to get assistance from our customer engineering team. Please use the option that is best for your users:

• Email: help@cortex.io, or open a support ticket in the in app Resource Center
• Chat: Available in the Resource Center
• Slack: Users with a connected Slack channel will have a workflow added to their account. From here, you can either @CortexTechnicalSupport or add a :ticket: reaction to a question in Slack, and the team will respond directly.

Don’t have a Slack channel? Talk with your customer success manager.