AWS Resources Inventory using Cloud Custodian

Use Cloud Custodian to count the resources and visualize in the Sumo Logic

In no time, you will realize that your organization has hundreds of AWS accounts and several hundreds of resources within it. It is very important to know the inventory, ownership, and purposes of those resources. For the purpose of this story, we assume that the readers already have the basic knowledge of the Cloud Custodian. In this story, we will discuss one method of doing asset inventory for AWS resources using the Cloud Custodian Policy and the Sumo Logic (SIEM solution).

Setup

1)Deploy the Cloud Custodian YAML policies to the AWS account. 2) This policy goes as Lambda Function and runs as serverless 3) Based on the frequency, the Cloud Custodian policy will execute and it will produce the three GZ output files. 4) We have defined within the policy to drop the file into the S3 account (one management account). 5) Make sure the IAM role has permission to drop the file to the S3 bucket. 6) Ingest these logs into the Sumo Logic (SIEM) solution. 7) You can write various queries to draw the dashboard and visualization.

Cloud Custodian Output Workflow

The below dashboard from Sumo Logic shows the inventory of all the resources. You must have the Cloud Custodian policy for each resource running in every single account. Since this is a hygiene type policy, you may want to run these policies once every 1 week. Ingest all the data from your S3 to Sumo Logic and then write queries within.

Inventory Dashboard in Sumo Logic

In the below query, we have shown an example for Lambda Functions. It will produce the result in tabular format and other charts. Add this query to your asset inventory dashboard to see the visualization.

_sourceCategory="sourcecategory-name"
AND _sourceName=*CustodianLogs/*/sec-n-lambda-function-count/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"FunctionName\":\s\"(?<FunctionName>.+?)\"" multi nodrop
| where FunctionName matches "*cis-*"
| count(FunctionName) group by FunctionName
| fields -_count
| count

Below we have written some example policies which identify and count the resources like AMIs, Cloud Formation Stack, EC2, EBS Volume, ECS, Elastic IPs, ENIs, RDS, and many more. You will ask the question “how do we know what resources are in use today?” The answer to this question is — you can check in your master billing account under cost explorer. You will see all the resources that are in use and the corresponding costs. This will help you understand the asset value, data classification, and security control that needs to be implemented to protect those assets (cost-benefit analysis). You have to write one policy for one resource to get the counting.

  1. Total Count of AMIs
- name: misc-n-ami-count
resource: aws.ami
comment: |
Find all Amazon Machine Image count. This provides us the count.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

2. Total Count of Cloud Formation Stack

- name: misc-n-cfn-stack-count
resource: aws.cfn
comment: |
Find all CloudFormation Stack count. This provides us the count.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

3. Total Count of Cloud Watch Event Rule

- name: misc-n-cw-event-rule-count
resource: aws.event-rule
comment: |
Identify and Count on CloudWatch Event Rule.
This will help us know when we are going to exceed the default
quota (300) or custom quota of 1000 event-rule. This is a
notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

4. Total Count of EC2 Instances

- name: misc-n-ec2-count
resource: aws.ec2
comment: |
Identify and Count EC2 Instances.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

5. Total Count of ECS

- name: misc-n-ecs-count
resource: aws.ecs
comment: |
Identify and Count Elastic Container Service (ECS).
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

6. Total Count of Elastic IP

- name: misc-n-elastic-ip-count
resource: aws.elastic-ip
comment: |
Identify and Count on Elastic IP.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

7. Total Count of ENI

- name: misc-n-eni-count
resource: aws.eni
comment: |
Identify and Count on Elastic Network Interface.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

8. Total Count of EBS Volume

- name: misc-n-ebs-volume-count
resource: aws.ebs
comment: |
Identify and Count on EBS Volume.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

9. Total Count of Elasticsearch

- name: misc-n-elasticsearch-count
resource: aws.elasticsearch
comment: |
Identify and Count on Elasticsearch.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

10. Total Count of ELB

- name: misc-n-elb-count
resource: aws.elb
comment: |
Identify and Count on Elastic Load Balancer.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

11. Total Count of ElasticCache

- name: misc-n-elasticache-cluster-count
resource: aws.cache-cluster
comment: |
Identify and Count Elastic Cache Cluster.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

12. Total Count of Firehose

- name: misc-n-firehose-count
resource: aws.firehose
comment: |
Identify and Count Firehose.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

13. Total Count of IAM Role

- name: misc-n-iam-role-count
resource: aws.iam-role
comment: |
Identify and Count IAM Role.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

14. Total Count of KeyPair

- name: misc-n-key-pair-count
resource: aws.key-pair
comment: |
Identify and Count Key Pair.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

15. Total Count of Kinesis

- name: misc-n-kinesis-count
resource: aws.kinesis
comment: |
Identify and Count Kinesis.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

16. Total Count of KMS Key

- name: misc-n-kms-key-count
resource: aws.kms-key
comment: |
Identify and Count KMS Key.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

17. Total Count of Lambda Function

- name: misc-n-lambda-function-count
resource: aws.lambda
comment: |
Identify and Count Lambda Function.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

18. Total Count of CloudWatch Log Group

- name: misc-n-log-group-count
resource: aws.log-group
comment: |
Identify and Count Cloudwatch log-group.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

19. Total Count of VPC Peering Connection

- name: misc-n-peering-connection-count
resource: aws.peering-connection
comment: |
Identify and Count VPC Peering Connection
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

20. Total Count of RDS

- name: misc-n-rds-count
resource: aws.rds
comment: |
Identify and Count RDS.
This is a notify only policy. This policy is schedule to run at
8:00 AM UTC / 3:00 AM CDT, every 3 days starting on 1st of every
month.
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://bucketname/cclogs/{account_id}/
tags: *sec-tags

Other Stories

Count the Azure Resources using the Cloud Custodian

Cloud Custodian Policy Health Checks

Ingesting Cloud Custodian Logs into Sumo Logic

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aakif Shaikh, CISSP, CEH, CHFI, CISA, GWAPT

Over 18 years of experience in a wide variety of technical domains within information security including information assurance, compliance, and risk management.