Cloud Custodian [GZ] Output Files

policies:
- name: sec-n-elb-internet-facing
resource: aws.elb
description: |
This policy identifies all Load Balancers that are facing the
Internet.
filters:
- Scheme: internet-facing
mode:
type: periodic
schedule: "rate(3 days)"
execution-options:
output_dir: s3://example-bucket/cclogs/policy/{account_id}
runtime: python 3.8

What execution mode schema supports the option to have the output sent to a directory, say s3 bucket?

From the above policy example, you can see that we are declaring the output directory within the policy itself as to where the Custodian must drop the output file. All the AWS execution mode schema supports the availability of “execution options” except the “Pull” mode. See the below screenshot for an example.

What format are those files?

Cloud Custodian Produces the output in GZ format. This is because it has 3 different files which are compressed using the GNU ZIP compression algorithm. It reduces the file size and allows multiple files to be combined into one. With the advanced usage, Cloud Custodian can generate the reports in CSV or Text format using the “report” command. Click the links for more details on Reporting and Advanced Usage.

s3 bucket containing the output from the Cloud Custodian

What are those files?

Cloud Custodian produces 3 output files. The two files namely resources and metadata is in JSON while the run log file is text. Each file has its own importance and purpose that flags the non-compliant items, provides the policy health checks, and other details. Let’s go through each one of them and understand its purpose and how can we make use of it.

GZ and Uncompressed Files
// SumoLogic Query to identify the non-compliant items
// Find unsecure SSM Parameters.
_sourceCategory="cloud/aws/custodian/resources/security"
AND _source="resources.json.gz_file" and _collector="applications"
AND _sourceName=*custodian/*/sec-n-ssm-parameter-not-secure/*/*/*/*/resources.json.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policyname, year, month, date, minutes, crunlog nodrop
| parse regex "\"Name\":\s\"(?<Name>.+?)\""
| count (Name) group by account_id
Resource count metrics from the metadata file
// SumoLogic Query to look for value for ResourceCount
// Find unsecure SSM Parameters.
_sourceCategory="cloud/aws/custodian/resources/security"
AND _source="metadata.json.gz_file" and _collector="applications" sec-n-ssm-parameter-not-secure
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policyname, year, month, date, minutes, crunlog nodrop
| json field=_raw "metrics[0].MetricName" as metrics_0_MetricName
| where metrics_0_MetricName matches "ResourceCount"
| json field=_raw "metrics[0].Value" as metrics_0_Value
| sum(metrics_0_Value)
Debug Message
// SumoLogic Query to look for the Filtered from/to count.
// Find unsecure SSM Parameters.
_source="custodian-run.log.gz_file" and _collector="applications"
AND _sourceName=*CustodianLogs/*/*/*/*/*/*/custodian-run.log.gz Filtered
AND _sourceName=*custodian/*/sec-n-ssm-parameter-not-secure/*/*/*/*/custodian-run.log.gz
| parse field=_sourceName "*/*/*/*/*/*/*/*" as clogs, account_id, policyname, year, month, date, minutes, crunlog nodrop
| parse "DEBUG - Filtered from * to * ssmparameter" as fromcount,twocount
| sum(twocount)

Filtered “to” value from the run log file must match with the “value” of the resource count from the metadata file must match to the actual resource from the resource file.

We will discuss in another story on how to ingest the Custodian Logs into the SIEM solution and draw pretty Dashboards.

Other Stories Related to Cloud Custodian

Tried replacing the Cloud Custodian with Something else…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aakif Shaikh, CISSP, CEH, CHFI, CISA, GWAPT

Aakif Shaikh, CISSP, CEH, CHFI, CISA, GWAPT

67 Followers

Over 18 years of experience in a wide variety of technical domains within information security including information assurance, compliance, and risk management.