Datadog
Summary
Datadog is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services. You can use Datadog to drive insights into values such as:
- Service discovery
- Monitors
- SLOs
Setup
In order to connect Cortex to your Datadog instance, you’ll need to add a Datadog Application Key and create a Datadog API key, and add it under Settings → Datadog.
Service Registration
Discovery
By default, Cortex will use the service identifier (e.g. my-service) as the "best guess" for the Datadog tags. For example, if your service identifier is “my-service”, then the corresponding tag in Datadog should also be “service:my-service”.
If your Datadog tags don’t cleanly match the Cortex service identifier, you can override this in the Cortex Service Descriptor.
Service Descriptor
If you need to override the automatic discovery, you can define the following block in your Cortex Service Descriptor.
x-cortex-apm:
datadog:
serviceTags: # List of tags & values
- tag: service
value: brain
- tag: backend
value: brain
tip
These tags are used to "discover" your monitors and SLOs. Cortex will look up monitors and SLOs by querying for tag:value OR tag:value ...
If you want to hardcode/override discovery altogether, use the following sections:
Monitors
Adding monitors let you see information about their current status directly in the Catalog, under the Datadog
section for each service. You can find your monitors at https://app.datadoghq.com/monitors/manage.
The ID of a monitor is found in the URL when you click on a monitor in your Datadog dashboard, for example https://app.datadoghq.com/monitors/**<MONITOR_ID>**
.
info:
x-cortex-apm:
datadog: # List of monitor IDs.
monitors:
- 12345
- 67890
SLOs
Datadog SLOs can be listed in the Catalog for each service under the Datadog
section. You can find the SLOs at https://app.datadoghq.com/slo.
The ID for the SLO can be found in the URL when you click on an SLO in the datadog dashboard, for example https://app.datadoghq.com/slo?slo_id=**<SLO_ID>**&timeframe=7d&tab=status_and_history
.
info:
x-cortex-slos:
datadog: # List of SLO ids
- id: 0b73859a3e2504bf09ad23a161702654
- id: 228499184a9efe34d4e4e9df838c7fa1
Cortex will do an OR
query on Datadog using service:brain OR backend:brain
to find the SLOs & Monitors.