AWS

CatalogDiscovery

Summary

Amazon Web Services, Inc. (AWS) that provides on-demand cloud computing platforms and APIs. You can use AWS to drive insights into values such as:

Catalog Discovery
Dependencies

There is a limitation with the AWS integration for discovery of RDS databases. Only Amazon Aurora DB clusters and Multi-AZ DB clusters can be discovered.

Setup

In order to connect Cortex to your AWS resources, you’ll need to add your AWS Account ID and IAM Role in Settings → AWS. For Cloud Cortex, we support multiple accounts. Each will share a common externalId that can be rotated. For each account:

If you do not see the Settings page you're looking for in the sidebar, you likely don't have the proper permissions and need to contact your admin.

Log onto the AWS management console and navigate to IAM
Create a new IAM policy by navigating to Policies | Create Policy.

Switch to the JSON editor and insert the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:Describe*",
        "dynamodb:List*",
        "ecs:Describe*",
        "ecs:List*",
        "elasticloadbalancing:Describe*",
        "kafka:Describe*",
        "kafka:List*",
        "lambda:Get*",
        "lambda:List*",
        "rds:Describe*",
        "rds:List*",
        "s3:Describe*",
        "s3:List*",
        "s3:GetBucketLocation",
        "s3:GetBucketTagging",
        "tag:GetResources",
        "tag:GetTagKeys",
        "tag:GetTagValues"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Click Review Policy, name it whatever you like, then Create Policy.

Cloud

Next, let's create a new role and attach the new policy to it. Navigate to Roles | Create Role.
Select Another AWS account as the trusted entity type.
For Account ID, fill in the Cortex AWS account ID from Settings | AWS.
Click Require External ID and insert the Cortex external ID from Settings | AWS, then click Next: Permissions.
Select your newly created policy, and click Next: Tags, if you'd like to add any, otherwise skip the next page with Next: Review.
Name your role, and click Create Role.
Finally, copy your new role name and AWS account ID into the Cortex AWS settings page.

On-Prem

If you are deploying Cortex in an AWS environment, Cortex will automatically try to discover AWS credentials in that environment, following Amazon's default credential retrieval order. For example, Cortex can retrieve credentials from the ECS container or EC2 instance that it's running in. Note that if you have already set up the access key id, access key secret, and account number environment variables described below, you'll need to remove them for the credential providers to be used.

If you don't want to use the credentials provided by the environment, you can manually specify the credentials in Cortex. To do so, create access users along with access keys for each AWS account you'd like access to. They should have all the permissions listed above. For each set of access keys, set these environment variables (access key id, access key secret, and account number) in your ConfigMap:

AMAZON_ACCESS_0_ID
AMAZON_ACCESS_0_SECRET
AMAZON_ACCESS_0_ACCOUNT
AMAZON_ACCESS_1_ID
AMAZON_ACCESS_1_SECRET
AMAZON_ACCESS_1_ACCOUNT
...
AMAZON_ACCESS_N_ID
AMAZON_ACCESS_N_SECRET
AMAZON_ACCESS_N_ACCOUNT

Registration

Catalog Descriptor

You can associate a Cortex entity with one or more AWS entities. For certain AWS resource types, Cortex will display those AWS entities' metadata on the Cortex entity page. If you want to define a dependency from a Cortex service to an AWS entity, please see Dependencies and the Dependencies section on this page.

x-cortex-infra:
  aws:
    arns:
      - arn:aws:rds:us-east-1:229540587644:cluster:alpha
      - arn:aws:rds:us-east-1:229540587644:cluster:bravo

Dependencies

Cortex automatically discovers dependencies between your services and resources by scanning for resources with specific AWS tags. By default, a service will have dependencies on any Cortex resource that has a corresponding AWS resource with AWS tag key = "service" and tag value = the service's Cortex tag. In AWS settings, you can customize the tag key name, or leave it blank to use "service" as the key name.

You can also use explicit tag key/value pairs in the x-cortex-dependency block for AWS dependency discovery. Instead of making a service depend on a resource based on service tags, Cortex will make a service depend on a resource if any of the resource's tags match the explicitly defined key/value pairs in the service's x-cortex-dependency block.

x-cortex-dependency:
  aws:
    tags:
      - key: my-key-1
        value: my-value-1
      - key: my-key-2
        value: my-value-2