Ingesting Cloud Custodian Logs into Sumo Logic (Part 2)

Collection, Parsing, and Writing SumoLogic Queries

In the previous story, we have looked at different components that enable the ingestion of Cloud Custodian logs into Sumo Logic (SIEM solution). In this story, we will look at parsing and writing queries within the Sumo Logic. We will also draw some pretty dashboards for reporting purposes to the management and cloud governance committee. As you know, we have created 3 different “sources” to ingest the logs into Sumo Logic, we are going to parse them separately. Let’s start with the resources.json.gz file.

The method used for Collecting the Logs from S3 to SumoLogic Source / Collector

One of the methods that we tried using within Sumo Logic to ingest was Collection> Source > Log File Discovery > Scan Interval Automatic (without using the SNS). This would grab the logs from the S3 bucket into Sumo Logic. We found that this kind of ingestion fails and gives errors. One of the reasons for failure is that the collector becomes overwhelmed as it receives data from 500+ policies (100s of AWS accounts) at a point in time (policies are run based on the defined CRON expression).