Custom data

Custom data extends the out-of-the-box metadata that Cortex provides via integrations, enhancing your use of catalogs and augmenting details available for your entities. This data can also be used to write CQL queries and Scorecard rules.

Custom data can be defined manually in the entity descriptor, added to entities programmatically via a REST API, or sent through a webhook.

Defining custom data

There are a few ways you can add custom data to an entity: the entity descriptor, a POST REST endpoint, or a simple webhook.

Adding custom data via entity descriptor is optimal for low-volume, human-maintained data because it requires updating the entity's YAML when the data changes.

Using a REST API is a better option for data that comes from an automated process, like a CI/CD pipeline.

Editing the entity descriptor

The simplest way to describe information in the YAML is to define an object.

x-cortex-custom-metadata:
  key-example: value-example
  arbitrary-custom-data: hippocampus
  regions-of-brain:
    value: 3
    description: cerebrum, cerebellum, brainstem
  cognition: true

Field	Description	Required
key	Key or title for the custom data. Anything defined before the : will serve as the key.	✓
value	Value for the custom data. Anything defined after the : is the value.	✓

Custom data can be of any type, including scalars (strings, numbers, booleans), objects, and lists.

When you add custom data to an entity's YAML, you'll see the key and value pairs on an entity's Custom data page.

info

Custom data added via entity descriptor will display with a YAML tag.

Adding descriptions

You can add a description to data by explicitly defining it along with the key and value pairs. While key and value pairs can be defined with any terms when a description is not added, value: must explicitly be defined when a description is included.

x-cortex-custom-metadata:
  regions-of-brain:
    value: 3
    description: cerebrum, cerebellum, brainstem
  brain-hemispheres:
    value: 2
    description: The brain has a left and a right hemisphere.

Field	Description	Required
key	Key or title for the custom data. Anything defined before the : will serve as the key.	✓
value	Value for the key; should be defined explicitly with value:	✓
description	Description of the custom data	✓

note

While the description field is always optional, it is technically "required" to add a description to custom data.

Adding custom data via the Cortex API

You can pipe custom data directly into Cortex by POSTing to /api/v1/catalog/{tag}/custom-data where {tag}is the x-cortex-tag for a given entity.

The request body requires the following in JSON format:

Field	Type	Description	Required
key	string	Key or title for the custom data	✓
value	object	Value for the custom data	✓
description	string	Description of the custom data

See API Docs for authentication details.

If a key has already been set through the entity descriptor, an API call will NOT overwrite the existing value. Instead, it will simply return the existing value.

warning

If a key is defined in the entity descriptor, the API cannot override the key. You'll see YAML as the source value in the response body if you encounter this situation.

To explicitly overwrite values set in the YAML file, use the force=true flag as a query parameter.

tip

If you find yourself using the force flag, it may be better to remove the field from the YAML file or update the value in the cortex.yaml file instead to maintain the source of truth. Read more about source hierarchy here.

info

Custom data added via API will display with a API tag.

Bulk upload entity keys

You can use the bulk upload endpoint to upload multiple keys for one or more entities.

PUT data in the following format to /api/v1/catalog/custom-data:

{
  "values": {
    "<entity-tag>": [
      {
        "key":"example-key",
        "value":"example value",
        "description":"A description of these data."
      }
    ],
    "<another-entity-tag>": [
      {
        "key":"example",
        "value":{
          "my-data":"my-value",
          "complex": "data",
          "v": 0
        }
      }
    ]
  }
}

You can include multiple key and value objects for each tag. The object conforms to the same shape as the single upload API.

Adding custom data via a webhook

You can use custom data webhooks to create unique URLs that you can use to directly POST arbitrary JSON payloads without auth or an explicit entity tag in the URL if you do not have obvious access to the entity tag or the ability to add authentication headers.

With this method, you can tell Cortex how to process the payload and map it to an entity. For example, the payload may include a data.tag field that corresponds to the entity tag.

Creating a webhook URL

1. Go to Custom integrations settings in Cortex.
2. Create a custom integration for the webhook:

• Name: A human-readable name for the webhook.
• Entity tag JQ: The JQ expression that Cortex will use to extract the entity tag from the payload (e.g. .data.entityTag). **If the tag Cortex extracts from the payload is not identical to the entity tag in Cortex, the endpoint will throw a 400.
• Key: The unique key that will be used in CQL queries with the syntax integration(key). Data will be stored under the key for entities, just like when adding data via entity descriptor or REST API.

3. Save the integration and copy the provided webhook URL.
4. cURL any JSON your heart desires to this URL!

The data can be used exactly the same way as custom data defined through the entity descriptor or the API.

info

Custom data added via webhook will display with an INTEGRATION tag.

Changing the mappings for entities

If the webhook payload does not contain a value that maps to the exact entity tag you can tell Cortex alternative mappings to look up when processing payloads.

x-cortex-custom-integration-mappings:
  - left-brain
  - right-brain

When processing a payload, Cortex takes the output of the Entity tag JQ field in step 2 above and searches for an entity with that value as a tag. If no such entity exists, Cortex checks whether an entity has registered the value as an alternative mapping; if so, the payload is attached to that entity.

Data source hierarchy

Cortex applies a data source hierarchy to determine how to handle a case where the same key is defined from multiple sources.

1. The entity descriptor is the source of truth. If a key is defined there, the API or webhooks cannot override the key.
2. The API and webhook approaches are at the same level and can override each other.

You can override keys defined in the YAML by adding a force=true parameter when using an API, but when the entity descriptor is eventually re-processed, the data provided via the API will be overwritten.

Use cases

Cataloging

Catalogs contain a lot of useful metadata about entities, including ownership and data from integrations. However, you may have your own custom fields that you want to include that would make exploring the catalog easier.

These can include things like:

• Which AWS zones is this deployed in?
• What databases does the entity consume?
• When was the last successful CI run?

tip

If the answers to these questions fit a pre-enumerated list, consider using groups. Groups will display on entities' detail pages and can easily be applied in catalog filters.

Custom data for cataloging makes the most sense when the data are more freeform/flexible.

Custom data can then be queried against by using the Query builder, explored with CQL reports, or viewed on an entity's details page.

Scorecards

Cortex makes it easy to write Scorecard rules based on custom data that exists in your catalogs. When adding a rule, you can select Custom data from Choose an integration in the form editor and follow the flow. Cortex will automatically supply variables based on the custom data you've defined.

tip

Using the custom data API to push data into Cortex is a great way to extend your Scorecards. You can send in entire JSON payloads and use jq to process them in a Scorecard or use it as an input to a custom OPA Policy as a Scorecard rule.