Nov 9, 2021
Auto-remediation for missing tags in AWS using Cloud Custodian — Part 1
Automate the missing tag remediation task using the Cloud Custodian.
Cloud Custodian provides several ways of solving the missing tag problem for both existing and newly created AWS resources. Among them, the auto-tag-user action item is very powerful in automatically tagging the resources with missing owner tags. This saves time for the analyst to identify all the resources that are missing the mandatory tag requirement and, more importantly, identify the individual who stood them up to take corrective actions. The auto-tag-user action item is supported for both the public cloud providers AWS and Azure. However, in this story, we will just cover AWS.
We will discuss the benefits and drawbacks of using auto-tag-user action items in the policy through the console and continuous integration (CI/CD) pipeline. The auto-tag-user policy can only be applied to the newly created AWS resources. When you deploy the auto-tag-user policy, it will create a cloud watch event rule (AWS API call via CloudTrail) and lambda function. A CloudWatch log group will get created as soon as the event rule triggers via CloudTrail. An assumed role used by Cloud Custodian to call various APIs must have appropriate read-write permissions to the resources.
The below screenshot shows the schema for auto-tag-user action items.
When we analyze the schema, there are two required attributes type and tag. type invokes the cloud custodian auto-tag-user action. tag is where you specify the name for the key. ex- email-address, owner, CreatorEmail, etc. Within the properties, the cloud custodian also supports principal_id_tag. AWS has the concept of principalId. A principalId is an internal unique identifier that is making a call.
principalId”: “AROAXXXXXXXXABCD12345:abcdegh@example.com”
Where does Cloud Custodian learn about the tag and principal_id_tag? A very important concept introduced by AWS is the principalId. It comprises two elements — AROA and user. Both are separated using the colon symbol (:) as shown below in screenshot-1.
As an example, take the action of S3 bucket creation by a user. Cloud Custodian will look at the CloudTrail events to grab the Value for auto-tagging. It will then auto-tag the value from the second half of the principalId as shown below in the screenshot-2
Below are examples of auto-tagging for some of the AWS resources.
Example- 1 AWS Certificate Manager
policies:- name: misc-na-acm-certificate-auto-tag
resource: aws.acm-certificate
comments: |
Find ACM Certificate that has not been tagged with mandatory
owner tag on-creation. Tag ACM Certificate with the user who
created it. This policy does not apply on existing ACM Certs.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: acm.amazonaws.com
event: RequestCertificate
ids: requestParameters.domainName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Further explanation- It is very important to have the correct event mode which includes source, event, and ids for the auto-tag policy to work.
Example- 2 AMI Image
policies:- name: misc-na-ami-auto-tag
resource: aws.ami
comments: |
Find AMI that has not been tagged with mandatory owner tag on-
creation. Tag AMI with the user who created it. This policy does
not apply on existing ACM Certs.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ec2.amazonaws.com
event: CreateImage
ids: requestParameters.imageId
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 3 AWS Auto Scaling Group (ASG)
policies:- name: auto-scaling-group-auto-tag
resource: aws.asg
comments: |
Find ASG that has not been tagged with mandatory
owner tag on-creation. Tag ASG with the user who
created it. This policy does not apply on existing ASG.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: autoscaling.amazonaws.com
event: CreateAutoScalingGroup
ids: requestParameters.autoScalingGroupName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
If you are using the Terraform or any other CICD tools, you should consider reading Part-2 of the story to know the underlying challenges you may face before you start using the auto-tag-user action item.
Example- 4 RestAPI Gateway
policies:- name: rest-api-auto-tag
resource: aws.rest-api
comments: |
Find API Gateway RestApi that has not been tagged with mandatory
owner tag on-creation. Tag RestApi with the user who
created it. This policy does not apply on existing RestApi.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: apigateway.amazonaws.com
event: CreateRestApi
ids: responseElements.restapiResources.restApiId
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 5 Backup Plan
policies:- name: backup-plan-auto-tag
resource: aws.backup-plan
comments: |
Find API Gateway RestApi that has not been tagged with mandatory
owner tag on-creation. Tag RestApi with the user who
created it. This policy does not apply on existing RestApi.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: backup.amazonaws.com
event: CreateBackupPlan
ids: responseElements.backupPlanArn
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 6 CloudFormation Stack
policies:- name: cloudformation-auto-tag
resource: aws.cfn
comments: |
Find CloudFormation that has not been tagged with mandatory
owner tag on-creation. Tag cfn stack with the user who
created it. This policy does not apply on existing cfn stack.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: cloudformation.amazonaws.com
event: CreateStack
ids: responseParameters.stackName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 7 CloudTrail
policies:- name: cloudtrail-auto-tag
resource: aws.cloudtrail
comments: |
Find CloudTrail that has not been tagged with mandatory
owner tag on-creation. Tag cloudtrail stack with the user who
created it. This policy does not apply on existing cloudtrail.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: cloudtrail.amazonaws.com
event: CreateTrail
ids: responseParameters.name
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 8 Customer Gateway
policies:- name: customer-gateway-auto-tag
resource: aws.customer-gateway
comments: |
Find Customer Gateway that has not been tagged with mandatory
owner tag on-creation. Tag customer gateway stack with the user
who created it. This policy does not apply on existing customer
gateway.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ec2.amazonaws.com
event: CreateCustomerGateway
ids: responseElements.customerGateway.customerGatewayId
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 9 CloudFront Distribution
policies:- name: cloudfront-distr-streaming-auto-tag
resource: aws.distribution
comments: |
Find Cloudfront distribution that has not been tagged with
mandatory owner tag on-creation. Tag cloudfront distribution
stack with the user who created it. This policy does not apply
on existing cloudfront distribution.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: cloudfront.amazonaws.com
event: CreateDistribution
ids: responseElements.distribution.id
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example-10 Datapipeline
policies:- name: datapipeline-auto-tag
resource: aws.datapipeline
comments: |
Find Datapipeline that has not been tagged with mandatory owner
tag on-creation. Tag datapipeline with the user who created it.
This policy does not apply on existing datapipeline.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: datapipeline.amazonaws.com
event: CreatePipeline
ids: responseParameters.name
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 11 DirectConnect
policies:- name: directconnect-auto-tag
resource: aws.directconnect
comments: |
Find Directconnect that has not been tagged with mandatory owner
tag on-creation. Tag directconnect with the user who created it.
This policy does not apply on existing directconnect.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: directconnect.amazonaws.com
event: CreateDirectConnectGateway
ids: responseElements.directConnectGateway.directConnectGatewayId
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 12 DynamoDB Table
policies:- name: dynamodb-table-auto-tag
resource: aws.dynamodb-table
comments: |
Find DynamoDB table that has not been tagged with mandatory
owner tag on-creation. Tag dynamodb table with the user who
created it. This policy does not apply on existing dynamodb
table.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: dynamodb.amazonaws.com
event: CreateTable
ids: requestParameters.tableName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 13 EC2 Instances
policies:- name: ec2-auto-tag
resource: aws.ec2
comments: |
Find ec2 that has not been tagged with mandatory owner
tag on-creation. Tag ec2 with the user who created it.
This policy does not apply on existing ec2.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ec2.amazonaws.com
event: RunInstances
ids: responseElements.instancesSet.items[].instanceId
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: owner
principal_id_tag: principalid
Example- 14 Elastic Container Registries (ECR)
policies:- name: ecr-auto-tag
resource: aws.ecr
comments: |
Find ECR that has not been tagged with mandatory owner tag on-
creation. Tag ECR with the user who created it. This policy
does not apply on existing ECR.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ecr.amazonaws.com
event: CreateRepository
ids: responseElements.repository.repositoryName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: auto-owner
principal_id_tag: principalid
Example-15 Elastic Container Service (ECS)
policies:- name: ecs-auto-tag
resource: aws.ecs
comments: |
Find ECS that has not been tagged with mandatory owner tag on-
creation. Tag ECS with the user who created it. This policy
does not apply on existing ECS.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ecs.amazonaws.com
event: CreateCluster
ids: responseElements.cluster.clusterName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: auto-owner
principal_id_tag: principalid
Example- 16 ECS Service
policies:- name: ecs-service-auto-tag
resource: aws.ecs-service
comments: |
Find ECS service that has not been tagged with mandatory owner
tag on-creation. Tag ECS service with the user who created it.
This policy does not apply on existing ECS service.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ecs.amazonaws.com
event: CreateService
ids: responseElements.service.serviceName
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: auto-owner
principal_id_tag: principalid
Example- 17 ECS Task Definition
policies:- name: ecs-task-definition-auto-tag
resource: aws.ecs-task-definition
comments: |
Find ECS task definition that has not been tagged with mandatory
owner tag on-creation. Tag ECS task definition with the user who
created it. This policy does not apply on existing ECS task
definition.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: ecs.amazonaws.com
event: RegisterTaskDefinition
ids: responseElements.taskDefinition.taskDefinitionArn
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: auto-owner
principal_id_tag: principalid
Example- 18 Elastic File System
policies:- name: elastic-file-system-auto-tag
resource: aws.efs
comments: |
Find EFS that has not been tagged with mandatory owner tag on-
creation. Tag EFS with the user who created it. This policy
does not apply on existing EFS.
filters:
- "tag:owner": absent
mode:
type: cloudtrail
events:
- source: elasticfilesystem.amazonaws.com
event: CreateFileSystem
ids: responseElements.fileSystemArn
execution-options:
output_dir: s3://s3bucket/cclogs/{account_id}/
runtime: python3.8
actions:
- type: auto-tag-user
tag: auto-owner
principal_id_tag: principalid
In the second part of the story, we have discussed the benefits and challenges of using the auto-tag-user based on the method you are standing up the resources either via console or via CICD pipeline. We will also discuss how the auto-tag-user action item conflicts with your Terraform state file. Please continue to read Part — 2.
Other Stories: Auto-tagging in Azure, Cloud Custodian Policies for WebApp, Cloud Custodian Policies for SQL Database.
