Relationship between the Cloud Custodian Lambda Function and the Cloudwatch Event Rule (Part 2)

An alternate approach to monitoring your CW Event Rule Quota

In this story, we will be discussing an alternate approach on how we monitor the service quota limit for the CloudWatch Event Rule. For the purpose of this story, we will consider that our readers have the basic knowledge of the Cloud Custodian.

Method-

First of all, you must know what’s the service quota limit for the cloud watch event rule. The default value is 300. You may have increased a few times (say to 600 and then 1000). You can ask your AWS TAM to pull the number for you. At present, the API is not exposed and is not queryable. We have tried working with the AWS team and here is the GitHub issue- https://github.com/aws/aws-cli/issues/6629
Once you know your service limit quota, you can write the Cloud Custodian policy to get the count of your CloudWatch event rule. The below policy helps you count it-

policies:- name: misc-n-cw-event-rule-count
  resource: aws.event-rule
  comment: |
    Identify and Count on CloudWatch Event Rule.
    This will help us know when we are going to exceed our quota of 
    XXXX event-rule.  This is a notify only policy. This policy is 
    schedule to run at 8:00 AM UTC / 3:00 AM CDT, every 3 days 
    starting on 1st of every month..  
  mode:
    schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
    type: periodic
    execution-options:
      output_dir: s3://s3bucket-reports/cclogs/{account_id}/

3. As discussed in the previous story how we can ingest the Cloud Custodian logs into the SIEM solution (Sumo Logic). We will continue to use that scenario and ingest the logs for the above policy. The below query in Sumo Logic will help you pull all the counts for all accounts (in the tabular column). Replace the below query with your source category, source name, and policy name.

Sumo Logic Query
_sourceCategory="aws/cc/sec"
AND _sourceName=*cclogs/*/misc-n-cw-event-rule-count/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"Arn\":\s\"(?<Arn>.+?)\"" multi //nodrop
| count (Arn) by account_id

You can set up an alert within Sumo Logic to send you an email if the count reaches XXXX. This will help you monitor the usage of cloud watch event rules. Hence, it keeps your pipeline from failing due to the CW event rule limit (as shown below in the screenshot).