Relationship between the Cloud Custodian Lambda Function and the Cloudwatch Event Rule (Part 2)
An alternate approach to monitoring your CW Event Rule Quota
In this story, we will be discussing an alternate approach on how we monitor the service quota limit for the CloudWatch Event Rule. For the purpose of this story, we will consider that our readers have the basic knowledge of the Cloud Custodian.
- First of all, you must know what’s the service quota limit for the cloud watch event rule. The default value is 300. You may have increased a few times (say to 600 and then 1000). You can ask your AWS TAM to pull the number for you. At present, the API is not exposed and is not queryable. We have tried working with the AWS team and here is the GitHub issue- https://github.com/aws/aws-cli/issues/6629
- Once you know your service limit quota, you can write the Cloud Custodian policy to get the count of your CloudWatch event rule. The below policy helps you count it-
policies:- name: misc-n-cw-event-rule-count
Identify and Count on CloudWatch Event Rule.
This will help us know when we are going to exceed our quota of
XXXX event-rule. This is a notify only policy. This policy is
schedule to run at 8:00 AM UTC / 3:00 AM CDT, every 3 days
starting on 1st of every month..
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
3. As discussed in the previous story how we can ingest the Cloud Custodian logs into the SIEM solution (Sumo Logic). We will continue to use that scenario and ingest the logs for the above policy. The below query in Sumo Logic will help you pull all the counts for all accounts (in the tabular column). Replace the below query with your source category, source name, and policy name.
Sumo Logic Query
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"Arn\":\s\"(?<Arn>.+?)\"" multi //nodrop
| count (Arn) by account_id
You can set up an alert within Sumo Logic to send you an email if the count reaches XXXX. This will help you monitor the usage of cloud watch event rules. Hence, it keeps your pipeline from failing due to the CW event rule limit (as shown below in the screenshot).