Relationship between the Cloud Custodian Lambda Function and the Cloudwatch Event Rule (Part 1)
How to solve the quota problem for cloud watch event rules while deploying the cloud custodian policies
Now that you have decided to use the Cloud Custodian for managing your Cloud Governance, it is very important to understand the different components and their working mechanism. In this story, we will discuss the problem related to the CloudWatch event rule while deploying the policies and how we can overcome using different approaches.
Let’s take an example, a developer is deploying a policy to an AWS account to identify the AWS Redshift cluster that is publicly accessible. The policy will run periodically every 3 days. We have also provided the exemption tags in case if the Organization has a reason to have the cluster accessible to the public.
Example- Identify AWS Redshift Cluster Publicly Accessiblepolicies:
- name: redshift-cluster-publicly-accessible
Find Redshift clusters that are publicly accessible.This is a
notify only policy. The policy runs every 3 days.
- "tag:redshift-publicly-accessible-exempt": absent
- PubliclyAccessible: true
schedule: "rate(3 days)"
The basic component of Cloud Custodian depending on your implementation includes— Lambda Function, CloudWatch Log Groups, and Cloud Watch Event Rules. Firstly, you write a policy in YAML as shown above, an example to identify the publicly accessible Redshift clusters. When you deploy the policy to the AWS account, the real magic happens. It creates the lambda function which includes the policy. It will then create the CloudWatch Log Groups. This is where you can check the log streams. Every time when the policy runs it creates a new log stream. This log stream contains the timestamp and debugging messages. You will also find if the policy filters are working correctly (see below screenshot).
Lastly, it creates the Cloud Watch Event Rule. This is where you can check how often the policy will run. It includes the event rule name, status, event schedule, and target.
CloudWatch Event Rule Quota Limitation
The Lambda function that the policy creates has a one-to-one relationship with the Cloud Watch event rule. It means the CloudWatch event rule can not be shared. AWS by default provides a limited quota to the number of CloudWatch event rules that you can create. The default value is 300. If you plan to deploy more than 300 policies the deployment will fail because of the CloudWatch event rule quota limitations.
Cloud Custodian Lambdas has 1:1 relation with CloudWatch Event Rule
There are different ways to handle this problem of CloudWatch event rule quota limitation. For the sake of this story, we will consider a scenario where we have 700 Cloud Custodian policies that get deployed via the CICD pipeline when the new AWS account is created. Due to the CloudWatch event rule default quota of 300, the pipeline fails due to the limitation. It only provisions the 300 policies and the remaining 400 policies are unsuccessful. One way to solve this problem is to use a Quota Request Template within Service Quotas. Make sure you create this template from your AWS Organization management account (Master+Billing Account). It is very important to note that you can only create 10 quotas (by default). This is the hard limit. You can not get this increased from AWS. Here is the instruction from AWS on how to create a quota template — Click Here. You can specify the desired value to be 1000. With this every time you create a new AWS account, the quota request template will automatically submit the Case ID to increase the limit to 1000. However, the AWS support team takes some time to process the request. Once done, this will allow your pipeline to provision all 700 policies and it will turn out to be green.