How to deal with custom recorder of AWS Config?

Disclaimer: I’m not a REGEX expert :smile:

Lately, I was working for one of my customers on a custom configuration of AWS Config recorder.

My customer wanted to record using AWS Config All resources except a few of them:

'AWS::EC2::Subnet'
'AWS::EC2::VPC'
'AWS::EC2::SecurityGroup'

Unfortunately, the AWS API and Console do not allow you to do this, you should cherry-pick manually which resource you want to record.

The trade-off of this method is that if a new AWS Config resource type came out, it won’t be recorded until you manually select it in your AWS Config recorder setting.

To deal with this, I’ve created a very simple python script that is using AWS documentation web-scraping to extract supported resource types, and then apply all resources except those which are blacklisted by my customer.

The idea is to schedule this script once a week to be up-to-date.

import boto3
import re
from urllib.request import urlopen
import logging

# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/config.html#ConfigService.Client.put_configuration_recorder

# Purpose:
# Activate Custom AWS Record for AWS Config
# Supported resource type: https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources
# Scraping AWS Docs using: https://realpython.com/python-web-scraping-practical-introduction/

# Get information about the current regional config recorder: aws configservice describe-configuration-recorders --region eu-west-1

# Logging
root = logging.getLogger()
if root.handlers:
    for handler in root.handlers:
        root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s',level=logging.INFO)

recorder_name = "<AWS Config recorder name>"
role_arn = "<role arn used for AWS Config>"

# Put here the AWS Config resources type to exclude
exclusion_list = [
    'AWS::EC2::Subnet',
    'AWS::EC2::VPC',
    'AWS::EC2::SecurityGroup'
]

def get_config_resources():
    url = "https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources"
    page = urlopen(url)
    html = page.read().decode("utf-8")

    # Target format: AWS::ApiGateway::Stage
    pattern = "AWS::.*"
    match_results = re.findall(pattern, html)
    cleaned_list = []
    count = 0

    for result in match_results:
        # remove HTML tags
        results = re.sub("<.*?>", "", result) 
        # remove ending *
        results = results.replace("*", "")
        # remove space
        results = results.replace(" ", "")
        # remove long items (sentences)
        if len(results) >= 60:
            continue
        # distinct list while preserving order
        list(dict.fromkeys(results))
        # Count items
        count += 1
        # Create the target cleaned list
        cleaned_list.append(results)
    logging.info("Scraped Config supported resources: %s", count)

    return cleaned_list

def apply_custom_recorder(config_resources):
    # Remove excluded resources from the globql list
    result_list = list(set(config_resources) - set(exclusion_list))

    # counter
    count_result = 0

    # Count resulted number of resource types (minus excluded types)
    for type in result_list:
        count_result += 1
    logging.info("result_types: %s", count_result)

    client = boto3.client('config')

    try:
        r = client.put_configuration_recorder(
                ConfigurationRecorder={
                    'name': recorder_name,
                    'roleARN': role_arn,
                    'recordingGroup': {
                        'allSupported': False,
                        'includeGlobalResourceTypes': False,
                        'resourceTypes': result_list
                    }
                }
            )
    except Exception as e:
        logging.error(e)
    logging.info("Response: %s", r)

if __name__ == "__main__":
    config_resources = get_config_resources()
    apply_custom_recorder(config_resources)

Gist version if you want to contribute.

I think this approach could help someone else, so I’m sharing it with you. Don’t hesitate to comment, and enhance it as you like.

That’s all folks!

zoph.