Relationship between the Cloud Custodian Lambda Function and the Cloudwatch Event Rule (Part 2)

An alternate approach to monitoring your CW Event Rule Quota

In this story, we will be discussing an alternate approach on how we monitor the service quota limit for the CloudWatch Event Rule. For the purpose of this story, we will consider that our readers have the basic knowledge of the Cloud Custodian.

CW Event Rule Service Quota

Method-

  1. First of all, you must know what’s the service quota limit for the cloud watch event rule. The default value is 300. You may have increased a few times (say to 600 and then 1000). You can ask your AWS TAM to pull the number for you. At present, the API is not exposed and is not queryable. We have tried working with the AWS team and here is the GitHub issue- https://github.com/aws/aws-cli/issues/6629
  2. Once you know your service limit quota, you can write the Cloud Custodian policy to get the count of your CloudWatch event rule. The below policy helps you count it-
policies:- name: misc-n-cw-event-rule-count
resource: aws.event-rule
comment: |
Identify and Count on CloudWatch Event Rule.
This will help us know when we are going to exceed our quota of
XXXX event-rule. This is a notify only policy. This policy is
schedule to run at 8:00 AM UTC / 3:00 AM CDT, every 3 days
starting on 1st of every month..
mode:
schedule: "cron(0 8 */3 * ? *)" # Policy runs every 3 days
type: periodic
execution-options:
output_dir: s3://s3bucket-reports/cclogs/{account_id}/

3. As discussed in the previous story how we can ingest the Cloud Custodian logs into the SIEM solution (Sumo Logic). We will continue to use that scenario and ingest the logs for the above policy. The below query in Sumo Logic will help you pull all the counts for all accounts (in the tabular column). Replace the below query with your source category, source name, and policy name.

Sumo Logic Query
_sourceCategory="aws/cc/sec"
AND _sourceName=*cclogs/*/misc-n-cw-event-rule-count/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"Arn\":\s\"(?<Arn>.+?)\"" multi //nodrop
| count (Arn) by account_id
Tabular Column Result from Sumo Logic

You can set up an alert within Sumo Logic to send you an email if the count reaches XXXX. This will help you monitor the usage of cloud watch event rules. Hence, it keeps your pipeline from failing due to the CW event rule limit (as shown below in the screenshot).

Error in the pipeline due to service quota limit

Other Stories

Ingesting Cloud Custodian Logs into Sumo Logic

Turn on and off your AWS resources using Cloud Custodian

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aakif Shaikh, CISSP, CEH, CHFI, CISA, GWAPT

Over 18 years of experience in a wide variety of technical domains within information security including information assurance, compliance, and risk management.