Ingesting Cloud Custodian Logs into Sumo Logic (Part 2)

Collection, Parsing, and Writing SumoLogic Queries

In the previous story, we have looked at different components that enable the ingestion of Cloud Custodian logs into Sumo Logic (SIEM solution). In this story, we will look at parsing and writing queries within the Sumo Logic. We will also draw some pretty dashboards for reporting purposes to the management and cloud governance committee. As you know, we have created 3 different “sources” to ingest the logs into Sumo Logic, we are going to parse them separately. Let’s start with the resources.json.gz file.

The method used for Collecting the Logs from S3 to SumoLogic Source / Collector

One of the methods that we tried using within Sumo Logic to ingest was Collection> Source > Log File Discovery > Scan Interval Automatic (without using the SNS). This would grab the logs from the S3 bucket into Sumo Logic. We found that this kind of ingestion fails and gives errors. One of the reasons for failure is that the collector becomes overwhelmed as it receives data from 500+ policies (100s of AWS accounts) at a point in time (policies are run based on the defined CRON expression).

Sumo Logic Collection S3 Source Type
Warning — Collection Error

We have solved this problem by using the SNS method. Follow the step-by-step instructions to configure the SNS topics, subscriptions, and s3 event notifications. We tried creating this manually within the console and it just keep giving the destination validation error. I found some articles on AWS related to this error- Click Here. We found that AWS wants things to be created in a certain way following certain patterns.

Sumo Logic S3 Event Notification Integration
  1. It is very important to remember that we have 3 output files from Cloud Custodian (as discussed here). We have to create 3 different Source Collections within SumoLogic (as discussed here — steps 4 and 5). We have to create 3 different SNS topics, 3 different SNS subscriptions, and 3 different S3 event notifications. This step is very important that will allow us to pipe the right data to the right collection sources. Otherwise, the logs will be messed and you won’t get accurate data. Here is the high-level instruction from Sumo Logic on configuring SNS with One S3 Bucket and Multiple Sources — Click Here

I will go through the steps for one output file. You must repeat the same steps for the other two files. Firstly, click on “Generate event-based polling template”. This gives you the ready-to-use cloud formation stack template for SNS and subscription deployment. (Note- we tried creating it manually within the console and it failed to connect with the s3 event notification because AWS wants things to be created in a certain way).

2. Download that YAML CloudFormation template file to your local desktop. Go to AWS Cloud Formation> Create Stack> With new resources (standard)>Template is ready > Upload from your local machine> Next> Stack Name: CloudCustodianToSumoLogicResourceFile (example)> Add Tags> Next> Create Stack. This will create the SNS topic and subscription.

3. Go to your S3 bucket> properties> event notification> create event notification> Event Name: LetSumoLogicKnowCCResourcesFile (example)> Keep prefix blank> Suffix: resources.json.gz> Event types: Check mark — All object create events> Destination: SNS Topic> From the drop-down select the topic name (CloudCustodianToSumoLogicResourceFile)> Save Changes. Make sure you connect SNS topic created for resources file to resources s3 event notification. Modify the suffix for the other two files to match the custodian output file name.

resources.json.gz file

Let’s say you have a policy for S3. The policy is scheduled to run every day at 6:00 AM CDT and the Output is saved in the S3 bucket. This policy is identifying all s3 buckets where HTTPS is not enforced via bucket policy. It is a notify-only policy.

- name: cis-s3-does-not-enforce-https
resource: aws.s3
comment: |
CIS Amazon Web Services Foundations v1.4.0 (2.1.2). Identify s3
where https is not enforced via bucket policy. By default,
Amazon S3 allows both HTTP and HTTPS requests. To achieve only
allowing access to Amazon S3 objects through HTTPS you also have
to explicitly deny access to HTTP requests. Bucket policies that
allow HTTPS requests without explicitly denying HTTP requests
will not comply with this recommendation. This policy runs every
day at 11:00 AM UTC which is every day at 6:00 AM CDT.
- not:
- type: has-statement
- Effect: Deny
Action: s3:GetObject
Principal: '*'
"aws:SecureTransport": false
schedule: "cron(0 11 * * ? *)"
type: periodic
output_dir: s3://bucketname/cclogs/{account_id}/

In the below query, declare your source category, source, collector, and source name. You are counting all the non-compliant s3 buckets and grouping them based on the account id. The sumo query will give you the result shown below in the screenshot.

_sourceCategory="aws/sourcecategory" and
_source="cloudcustodian_resources_file" and
AND _sourceName=*CustodianLogs/*/cis-s3-does-not-enforce-https/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policies_name, year, month, date, _min, crunlog nodrop
| parse regex "\"Name\":\s\"(?<Name>.+?)\"" multi nodrop
| count (Name) group by account_id
The result from the Sumo Logic Query

We have discussed more examples in “Dashboard for Cloud Custodian

Other Stories

Ingesting Cloud Custodian Logs into Sumo Logic

Cloud Custodian Policy Health Checks

Cloud Custodian Output Files



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store

Over 18 years of experience in a wide variety of technical domains within information security including information assurance, compliance, and risk management.