How to deal with custom recorder of AWS Config?

Disclaimer: I’m not a REGEX expert 😄

Lately, I was working for one of my customers on a custom configuration of AWS Config recorder.

My customer wanted to record using AWS Config All resources except a few of them:

  • 'AWS::EC2::Subnet'
  • 'AWS::EC2::VPC'
  • 'AWS::EC2::SecurityGroup'

Unfortunately, the AWS API and Console do not allow you to do this, you should cherry-pick manually which resource you want to record.

The trade-off of this method is that if a new AWS Config resource type came out, it won’t be recorded until you manually select it in your AWS Config recorder setting.

To deal with this, I’ve created a very simple python script that is using AWS documentation web-scraping to extract supported resource types, and then apply all resources except those which are blacklisted by my customer.

The idea is to schedule this script once a week to be up-to-date.

import boto3
import re
from urllib.request import urlopen
import logging

# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/config.html#ConfigService.Client.put_configuration_recorder

# Purpose:
# Activate Custom AWS Record for AWS Config
# Supported resource type: https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources
# Scraping AWS Docs using: https://realpython.com/python-web-scraping-practical-introduction/

# Get information about the current regional config recorder: aws configservice describe-configuration-recorders --region eu-west-1

# Logging
root = logging.getLogger()
if root.handlers:
    for handler in root.handlers:
        root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s',level=logging.INFO)

recorder_name = "<AWS Config recorder name>"
role_arn = "<role arn used for AWS Config>"

# Put here the AWS Config resources type to exclude
exclusion_list = [
    'AWS::EC2::Subnet',
    'AWS::EC2::VPC',
    'AWS::EC2::SecurityGroup'
]

def get_config_resources():
    url = "https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources"
    page = urlopen(url)
    html = page.read().decode("utf-8")

    # Target format: AWS::ApiGateway::Stage
    pattern = "AWS::.*"
    match_results = re.findall(pattern, html)
    cleaned_list = []
    count = 0

    for result in match_results:
        # remove HTML tags
        results = re.sub("<.*?>", "", result) 
        # remove ending *
        results = results.replace("*", "")
        # remove space
        results = results.replace(" ", "")
        # remove long items (sentences)
        if len(results) >= 60:
            continue
        # distinct list while preserving order
        list(dict.fromkeys(results))
        # Count items
        count += 1
        # Create the target cleaned list
        cleaned_list.append(results)
    logging.info("Scraped Config supported resources: %s", count)

    return cleaned_list

def apply_custom_recorder(config_resources):
    # Remove excluded resources from the globql list
    result_list = list(set(config_resources) - set(exclusion_list))

    # counter
    count_result = 0

    # Count resulted number of resource types (minus excluded types)
    for type in result_list:
        count_result += 1
    logging.info("result_types: %s", count_result)

    client = boto3.client('config')

    try:
        r = client.put_configuration_recorder(
                ConfigurationRecorder={
                    'name': recorder_name,
                    'roleARN': role_arn,
                    'recordingGroup': {
                        'allSupported': False,
                        'includeGlobalResourceTypes': False,
                        'resourceTypes': result_list
                    }
                }
            )
    except Exception as e:
        logging.error(e)
    logging.info("Response: %s", r)

if __name__ == "__main__":
    config_resources = get_config_resources()
    apply_custom_recorder(config_resources)

Gist version if you want to contribute.

I think this approach could help someone else, so I’m sharing it with you. Don’t hesitate to comment, and enhance it as you like.

That’s all folks!

zoph.


comments powered by Disqus