# Datadog {% hint style="info" %} Cortex connects to many third-party vendors whose system interfaces frequently change. As a result, integration behavior or configuration steps may shift without notice. If you encounter unexpected issues, check with your system administrator or refer to the vendor's documentation for the most current information. Additionally, integration sync times vary and are subject to scheduling overrides and timing variance. {% endhint %} [Datadog](https://www.datadoghq.com/) is an application performance monitoring platform that provides real-time observability into entities, servers, databases, and tools, providing developers with a comprehensive understanding of their infrastructure as well as the ability to identify areas for improvement. Cortex is uniquely equipped to augment Datadog's tools, providing greater visibility into your entities. In this guide, you'll learn how to set up the Datadog integration to pull in services and metrics for entities: * Monitors * SLOs * Dependencies ## How to configure Datadog with Cortex ### Prerequisites Before getting started, make sure you have created the following: * [Datadog application key](https://docs.datadoghq.com/account_management/api-app-keys/#application-keys) * You can create this in Datadog under **Organizational Settings > Applications Key**. * Include the following scopes: * `monitors_read` * `apm_api_catalog_read` * `dashboards_read` * `metrics_read` * `timeseries_query` * `apm_service_catalog_read` * `slos_read` * `apm_read` * [Datadog API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) * You can create this in Datadog under **Organizational Settings > API Key**. ### Configure the integration in Cortex 1. In Cortex, navigate to the [Datadog settings page](https://app.getcortexapp.com/admin/integrations/datadog). * Click **Integrations** from the main nav. Search for and select Datadog. 2. Click **Add configuration**. 3. Configure the Datadog integration form: * **Account alias**: Enter a name that Cortex will associate this configuration with. * **App key**: Enter the application key you generated in Datadog. * This key appears under the **Key** column in Datadog while viewing your list of Application Keys.\\
* **API key**: Enter the API key you generated in Datadog. * **Region**: Select your [Datadog region](https://docs.datadoghq.com/getting_started/site/) from the dropdown. * **Custom subdomain**: Enter the custom subdomain for your Datadog instance. * This field only takes the subdomain, not the entire URL. For example, this field would take `cortex-docs` from `https://cortex-docs.datadoghq.com`. * **Environments**: Optionally, enter environment tags for Datadog entities. * If you set an environment tag here, make sure to set the [`env` dropdown in Datadog](https://docs.datadoghq.com/service_catalog/navigating/#performance-view) to match. 4. Click **Save**. After saving your configuration, you are redirected to the Datadog integration settings page in Cortex. In the upper right corner of the page, click **Test configuration** to ensure Datadog was configured properly. #### **Configure the integration for multiple Datadog accounts**[**​**](https://docs.cortex.io/docs/reference/integrations/datadog#configure-the-integration-for-multiple-propsintegration-accounts) The Datadog integration has multi-account support. You can add a configuration for each additional by repeating the process above. Each configuration requires an alias, which Cortex uses to correlate the designated with registrations for various entities. Registrations can also use a default configuration without a listed alias. You can edit aliases and default configurations from the Datadog page in your Cortex settings. Select the edit icon next to a given configuration and toggle **Set as default** on. If you only have one configuration, it will automatically be set as the default.How to connect Cortex entities to Datadog ## How to connect Cortex entities to Datadog ### Tag discovery By default, Cortex will use the [Cortex tag](https://docs.cortex.io/entities#cortex-tag) (e.g. `my-entity`) as the "best guess" for Datadog tag. For example, if your Cortex tag is `my-entity`, then the corresponding tag in Datadog should also be `my-entity`. If your Datadog tags don’t cleanly match the Cortex tag, you can override this in the Cortex entity descriptor. ### Import entities from Datadog See the [Create services documentation](https://docs.cortex.io/entities/adding-entities/add-services#creating-services) for instructions on importing entities. ### Editing the entity descriptor You can use service tags to connect Datadog services to Cortex entities. ```yaml x-cortex-apm: datadog: serviceTags: - tag: entity value: brain alias: my-default-alias - tag: entity value: cerebrum alias: my-other-alias ``` | Field | Description | Required | | ------- | --------------------------------------------------------------------------------------------------- | -------- | | `tag` | Tag for the project in Datadog | **✓** | | `value` | Value for the project; Cortex will find monitors and SLOs by querying `tag:value OR tag:value2 ...` | **✓** | | `alias` | Alias for the configuration in Cortex (only needed if you have opted into multi-account support) | | These tags are used to "discover" your monitors and SLOs. Cortex will find monitors and SLOs by querying `tag:value OR tag:value2 ...` If you want to hard code and/or override discovery, you can define a monitor or SLOs block in the entity descriptor, as described below. #### **Monitors and SLOs** Adding monitors let you see information about their current status directly from a catalog - via the Monitors column - and under the `Operations` section of an entity page. You can find your monitors from Datadog's [Manage Monitors page](https://app.datadoghq.com/monitors/manage). The ID of a monitor is found in the URL when you click on a monitor in your Datadog dashboard i.e., `https://app.datadoghq.com/monitors/****`. ```yaml info: x-cortex-apm: datadog: monitors: - id: 12345 alias: my-default-alias - id: 67890 alias: my-other-alias ``` Like monitors, Datadog SLOs can be found in the `Operations` section of an entity page. You can find the SLOs for your instance on Datadog's [SLO status page](https://app.datadoghq.com/slo). The ID for the SLO can be found in the URL when you click on an SLO in the Datadog dashboard. For example, `https://app.datadoghq.com/slo?slo_id=****&timeframe=7d&tab=status_and_history`. ```yaml info: x-cortex-slos: datadog: - id: 0b73859a3e2504bf09ad23a161702654 alias: my-default-alias - id: 228499184a9efe34d4e4e9df838c7fa1 alias: my-other-alias ``` Monitors and SLOs have the same field definitions. | Field | Description | Required | | ------- | ------------------------------------------------------------------------------------------------ | :------: | | `id` | Datadog ID for the monitor or SLO | **✓** | | `alias` | Alias for the configuration in Cortex (only needed if you have opted into multi-account support) | | ### Dependency mapping Cortex automatically syncs dependencies from Datadog's [Service Map](https://docs.datadoghq.com/tracing/services/services_map/), using the entity identifier (`x-cortex-tag`) to map entities found in the Service Map. The relationships Cortex discovers through the integration will feed directly into the [Relationships graph](https://app.getcortexapp.com/admin/graph), so you can easily visualize the connections between your entities. If you have two entities - for example, `entity-one` and `entity-two` - that have a dependency edge in Datadog's Service Map, both entities should exist in Cortex with the same identifiers ([Cortex tag](https://docs.cortex.io/entities#cortex-tag)). You can override this by [defining Cortex tags](#editing-the-entity-descriptor) where `tag` = `entity` and `value` = `entity name in Datadog Service Map`. {% hint style="warning" %} If the Cortex tag does not exactly match the entity identifier in Datadog, the dependencies will not automatically sync. You can override automatic discovery by defining values in the entity descriptor. {% endhint %} You can connect an APM service for dependency mapping in the entity descriptor: ```yaml x-cortex-apm: datadog: serviceName: cortex-gateway.gateway ``` ## Using the Datadog integration ### View Datadog monitors and SLOs on entity pages With the Datadog integration, you'll be able to find monitors and SLOs on an entity's home page. High-level information about monitors and SLOs appears in the **Overview** tab. Click **Monitoring** in the entity's sidebar to see more detailed data about both monitors and SLOS. Both sections display tags for Pass, Fail, Warning, and No Data for each monitor or SLO. * The **SLOs** column shows each SLO, its target(s), the current value for that entity, its status. and the period of time the SLO is being calculated for. For example, if the time listed is "7 days ago," then the SLO is looking at the time range starting 7 days ago to now\.. * The **Monitors** column shows the title for each monitor, its query (if available), and its status. Clicking any block with a nonzero value will open a modal with more detailed information. The monitor modals will list all monitors with the applicable status. The SLO modals will also display targets for each SLO that is passing or failing. ### Scorecards and CQL With the Datadog integration, you can create Scorecard rules and write CQL queries based on Datadog metrics, monitors, and SLOS. See more examples in the [CQL Explorer](https://app.getcortexapp.com/admin/cql-explorer) in Cortex. You can read more about Datadog's [metrics](https://docs.datadoghq.com/metrics/types/) and [custom metrics](https://docs.datadoghq.com/metrics/custom_metrics/) in their docs.
Metrics [Timeseries data](https://docs.datadoghq.com/api/latest/metrics/#query-timeseries-data-across-multiple-products) from Datadog. * Metric * Timestamp **Definition:** `datadog.metrics(query: Text, lookback: Duration, alias: Text | Null)` **Example** You can use the `datadog.metrics()` expression to evaluate the health of your entities in a Scorecard: ``` datadog.metrics(query="system.cpu.usage{service:" + datadog.serviceNames().join(" OR service:") + "}",lookback=duration("P2D")).averageBy((point) => point.metricValue) < 0.10 ``` This rule makes sure that a given entity's average CPU usage is less than 10% over the last two days.
Monitors Monitors associated with a given entity via ID or tags. You can use these data to check whether an entity has monitors associated with it, or whether an entity has the right types of monitors. * Created at * Creator email * Creator name * Name * Overall state * Query * Tags * URL **Definition:** `datadog.monitors()` **Example** For a Scorecard focused on operational maturity, this expression can be used to make sure an entity has at least one Datadog monitor set up: ``` datadog.monitors().length >= 1 ```
SLOs SLOs associated with a given entity via ID or tags. You can use these data to check whether an entity has SLOs associated with it and if those SLOs are passing. * History * ID * Name * Operation * Remaining budget * SLI value * Datum * Timeseries * SLO target * Source * Thresholds * Name * Threshold **Definition:** `slos()` **Examples** For a Scorecard focused on operational maturity, this expression can be used to make sure an entity has associated SLOs in Datadog: ``` slos().length > 0 ``` This rule checks that there is at least one SLO is set up. While this rule makes sense in a Scorecard's first level, a rule checking the status of the SLO would make sense in a higher level: ``` slos().all((slo) => slo.passing) ``` Entities will pass this rule if all SLOs associated with it have "passing" status.
### Discovery audit Cortex will pull recent changes from your Datadog environment into the [discovered entities list](https://docs.cortex.io/ingesting-data-into-cortex/entities/discovery-audit). Here, you can find new entities in Datadog that have not been imported into the catalog - these will have the tag **New APM Resource** - as well as entities in the catalog that no longer exist in Datadog - these will have the tag **APM Resource Not Detected**. ### View integration logs ## Background sync The dependency sync runs automatically each day at 12 a.m. UTC, and can be run manually via the [**Sync dependencies** button](https://docs.cortex.io/entities/adding-entities/dependencies#sync-dependencies) in the [Relationship Graph](https://docs.cortex.io/ingesting-data-into-cortex/entities/relationship-graph). ## FAQs and troubleshooting **Can I set a Scorecard rule to monitor Datadog monitors/SLOs based on tags?** Yes, you can [specify key-value pairs](#entity-descriptor) that allow Cortex to discover your SLOs and monitors, and use these tags in Scorecard rules. **How does Datadog work with other dependency sources?** When leveraging multiple dependency sources such as Datadog and a catalog entity's YAML, all the sources would be merged together and Cortex will de-duplicate the dependencies. For example, if an entity YAML indicates `X → Y` and Datadog indicates `X → Y` and `X → Z`, the entity will display two edges presented as `X → Y` and `X → Z`. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.cortex.io/ingesting-data-into-cortex/integrations/datadog.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.