Custom data

If the integrations that Cortex provides out-of-the-box isn't sufficient, you can easily extend the metadata using custom data.

Summary

Custom data allows you to include additional information about all the entities in your catalog, to be used for:

• Querying with CQL
• Applying Scorecards rules to data from internal tools
• Viewing extra details in the Catalog

Custom data can be added to entities programmatically via a REST API or defined manually in the entity descriptor.

Use cases

Cataloging

The Catalog contains lots of useful metadata about entities, including ownership and data from integrations. However, you may have your own custom fields that you want to include that would make exploring the catalog easier.

These can include things like:

• Which AWS zones is this deployed in?
• What databases does the entity consume?
• When was the last successful CI run?

tip

If the answers to these questions fit a pre-enumerated list, consider using groups. Custom data for cataloging makes the most sense when the data is more freeform/flexible.

The data in the Catalog can then be queried against through our CQL-powered Query Builder, or even to quickly view information directly in the service or resource homepage.

Scorecards

You may want to write Scorecard rules that apply to information outside of data provided by our out-of-the-box integrations. For example, you may want to write rules that depend on data from your own internal tools, or vulnerability scan data that happens as part of a CI pipeline.

Using the custom data API to push data into Cortex is a great way to extend your Scorecards. You can even send in entire JSON payloads, and use jq to process them in a Scorecard or even as an input to a custom OPA Policy as a Scorecard rule.

Defining custom data

There are a few ways you can add custom data to an entity, including the entity descriptor, a POST REST endpoint, and a simple webhook.

Entity descriptor

The simplest way to describe information in the YAML is to define an object.

x-cortex-custom-metadata:
  passed-vulnerability-scan: 
   value: true
   description: Scan runs nightly
  code-coverage: 91
  circle-ci-images: 
     - cimg/go:1.16
     - circleci/mongo:3.2.20
  openapi:
    is_present: true
    is_valid: true
    version: 1.0.0

Custom data can be of any type, including scalars (strings, numbers, booleans), objects, and lists.

Adding descriptions

If you want to add a description to one of your pieces of data, just define it with the value key set explicitly.

x-cortex-custom-metadata:
  my-key:
    value: the actual value for the key
    description: this is the description
  passed-vulnerability-scan:
     value: true
     description: Scan runs nightly

API

You can also pipe custom data directly into Cortex if you know the entity tag, by POSTing to /api/v1/catalog/{tag}/custom-data, where the entity-identifier is the x-cortex-tag of the entity.

The request body is the following, in JSON format. See API Docs for authentication details.

Field	Type	Description
key	string	The key for custom data
value	object (string, boolean, number, list, ...)	Value to set for the key
description	string (optional)	An optional description for the data to be displayed in the dashboard.

YAML is the source of truth

If a key has already been set through the entity descriptor, an API call will NOT overwrite the existing value. Instead, it will simply return the existing value - you'll notice the source value in the response body is YAML.

To explicitly overwrite values set in the YAML file, use the force=true flag as a query param. If you find yourself using the force flag, you should likely remove the field from the YAML file since it is no longer the source of truth, or update the value in the cortex.yaml file instead.

Read more about the Source Hierarchy.

Bulk upload

If you're trying to upload multiple keys for a single entity, or many keys for many entities, you can use the bulk upload endpoint.

Just PUT data in the following format to /api/v1/catalog/custom-data:

{
  "values": {
    "<tag>": [
      {
        "key":"foo",
        "value":"bar",
        "description":"foobar"
      }
    ],
    "<another tag>": [
      {
        "key":"foo",
        "value":{
          "my-data":"my-value",
          "complex": "data",
          "v": 0
        }
      }
    ]
  }
}

For each tag, you can include multiple key/value objects. The object conforms to the same shape as the single upload API.

Webhook

In some cases, you may not have obvious access to the entity tag, or ability to add authentication headers. For example, if you're unable to connect Cortex directly to your SonarQube instance, you may want to dump SonarQube webhook data into Cortex as custom data.

"Custom data Webhooks" give you the ability to create unique URLs that you can use to directly POST arbitrary JSON payloads without auth or an explicit entity tag in the URL. Instead, you can tell Cortex how to process the payload to map it to an entity (e.g., the payload may include a data.tag field that corresponds to the entity tag).

To create a Webhook URL:

1. Visit Settings → Custom Integrations
2. Create a new Custom Integration
3. Give this webhook a human-readable name, like SonarQube Webhook.
4. Define a jq expression that Cortex will use to extract the entity tag from the payload, for example .data.entity.tag. If the tag that Cortex extracts from the payload is not exactly equal to the entity tag, the endpoint will throw a 400.
5. Add a key under which the data will be stored for entities, which is the same as the key when using the entity descriptor or REST API.
6. Save the integration, and copy the provided webhook URL.
7. cURL any JSON your heart desires to this URL!

The data can be used exactly the same way as custom data defined through the entity descriptor or the API.

Changing the mappings for entities

In the case where the webhook payload does not contain a value that maps to the exact entity tag (for example, a SonarQube payload where the project key in SonarQube is com.my.project but the x-cortex-tag of the entity is my-project), you can tell Cortex alternative mappings to look up when processing payloads.

x-cortex-custom-integration-mappings:
  - my-alternative-mapping
  - another-project-alias

When processing a payload, Cortex takes the output of the jq command in step 3 above, and first looks up if there's an entity with that value as a tag. If no such entity exists, we check whether an entity has registered the value as an alternative mapping, and if so, attach the payload to that entity.

Data source hierarchy

What happens when you try to define the same key from multiple sources? There's a hierarchy that's applied:

1. The entity descriptor is the source of truth. If a key is defined there, the API or webhooks cannot override the key.
2. The API and Webhook approaches are at the same level and can override each other.

You can override keys defined in the YAML by adding a force=true param when using the API, but there's no guarantee – when the entity descriptor is eventually re-processed, the data provided via the API will be overwritten.