Validate data after migration

Data verifications ensure that entity data in the catalog (e.g., ownership and metadata) stays accurate over time.

As a best practice after migrating data (for example, migrating services from spreadsheets into Cortex), we recommend scheduling a data verification to ensure your entities are up to date.

Set up a data verification after a migration

You must have the Configure verification periods permission to set up a verification.

1. In Cortex, navigate to the Entity verification settings page. Click +Add verification period.

2. In the side panel, fill in the fields:

   • Start date and End date: Enter the start and end dates for the verification period.

     • The start date cannot be in the past. The end date must be at least one day after the start date.

     • Choose a timeframe that is reasonable based on the number of entities that each team will need to verify.

   • Name: Enter a descriptive name, e.g., Validate data after migration.

   • Verification instructions: Enter clear and concise instructions that will appear on the entity's details page, e.g., These entities were recently imported from spreadsheets. Please review and verify whether they're accurate.

   • Require reason for verification: Check this box to require the user to include a reason if they confirm that the entity's data is invalid.

     • We recommend requiring a reason, as it will allow you to quickly understand what needs to be fixed on an entity.

   • Team member roles responsible for verification: Select which Cortex users will be responsible for verifying data.

     • If you select multiple roles, any of those users can verify data.

     • If you do not select a team member role, any user who owns the entity can verify data.

   • Entities to verify: Specify which entities the verification policy applies to.

     • You can select a specific entity type to include (for example, you could apply the verification to all entities of the type service) or if your entities have groups assigned, you can filter by specific groups.

     • Under Advanced options, you can use CQL to refine which entities you want to verify. For example, you might narrow your selection to entities created in the last two weeks: entity.created() < duration("P2W")

3. Click Create verification period.

Within a few minutes of saving the verification period, it will be visible to admins in the list of verifications on the Entity verification settings page. Entity owners will receive notifications to verify their entities.

Remediate entities with incorrect data

After setting up the verification period, you can create a CQL report to view all entities that have been verified as incorrect using a prebuilt CQL report template.

Cortex also offers a prebuilt Scorecard template to flag entities that have not yet been verified.