Ingesting Cloud Custodian Logs into Sumo Logic (Part 2)

The method used for Collecting the Logs from S3 to SumoLogic Source / Collector

One of the methods that we tried using within Sumo Logic to ingest was Collection> Source > Log File Discovery > Scan Interval Automatic (without using the SNS). This would grab the logs from the S3 bucket into Sumo Logic. We found that this kind of ingestion fails and gives errors. One of the reasons for failure is that the collector becomes overwhelmed as it receives data from 500+ policies (100s of AWS accounts) at a point in time (policies are run based on the defined CRON expression).

Sumo Logic Collection S3 Source Type
Warning — Collection Error
Sumo Logic S3 Event Notification Integration
  1. It is very important to remember that we have 3 output files from Cloud Custodian (as discussed here). We have to create 3 different Source Collections within SumoLogic (as discussed here — steps 4 and 5). We have to create 3 different SNS topics, 3 different SNS subscriptions, and 3 different S3 event notifications. This step is very important that will allow us to pipe the right data to the right collection sources. Otherwise, the logs will be messed and you won’t get accurate data. Here is the high-level instruction from Sumo Logic on configuring SNS with One S3 Bucket and Multiple Sources — Click Here

resources.json.gz file

Let’s say you have a policy for S3. The policy is scheduled to run every day at 6:00 AM CDT and the Output is saved in the S3 bucket. This policy is identifying all s3 buckets where HTTPS is not enforced via bucket policy. It is a notify-only policy.

- name: cis-s3-does-not-enforce-https
resource: aws.s3
comment: |
CIS Amazon Web Services Foundations v1.4.0 (2.1.2). Identify s3
where https is not enforced via bucket policy. By default,
Amazon S3 allows both HTTP and HTTPS requests. To achieve only
allowing access to Amazon S3 objects through HTTPS you also have
to explicitly deny access to HTTP requests. Bucket policies that
allow HTTPS requests without explicitly denying HTTP requests
will not comply with this recommendation. This policy runs every
day at 11:00 AM UTC which is every day at 6:00 AM CDT.
- not:
- type: has-statement
- Effect: Deny
Action: s3:GetObject
Principal: '*'
"aws:SecureTransport": false
schedule: "cron(0 11 * * ? *)"
type: periodic
output_dir: s3://bucketname/cclogs/{account_id}/
_sourceCategory="aws/sourcecategory" and
_source="cloudcustodian_resources_file" and
AND _sourceName=*CustodianLogs/*/cis-s3-does-not-enforce-https/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"Name\":\s\"(?<Name>.+?)\"" multi nodrop
| count (Name) group by account_id
The result from the Sumo Logic Query



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


Over 18 years of experience in a wide variety of technical domains within information security including information assurance, compliance, and risk management.