Disclaimer: I’m not a REGEX expert :smile:
Lately, I was working for one of my customers on a custom configuration of AWS Config recorder.
My customer wanted to record using AWS Config All resources except a few of them:
'AWS::EC2::Subnet''AWS::EC2::VPC''AWS::EC2::SecurityGroup'
Unfortunately, the AWS API and Console do not allow you to do this, you should cherry-pick manually which resource you want to record.
The trade-off of this method is that if a new AWS Config resource type came out, it won’t be recorded until you manually select it in your AWS Config recorder setting.
To deal with this, I’ve created a very simple python script that is using AWS documentation web-scraping to extract supported resource types, and then apply all resources except those which are blacklisted by my customer.
The idea is to schedule this script once a week to be up-to-date.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
import boto3
import re
from urllib.request import urlopen
import logging
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/config.html#ConfigService.Client.put_configuration_recorder
# Purpose:
# Activate Custom AWS Record for AWS Config
# Supported resource type: https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources
# Scraping AWS Docs using: https://realpython.com/python-web-scraping-practical-introduction/
# Get information about the current regional config recorder: aws configservice describe-configuration-recorders --region eu-west-1
# Logging
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s',level=logging.INFO)
recorder_name = "<AWS Config recorder name>"
role_arn = "<role arn used for AWS Config>"
# Put here the AWS Config resources type to exclude
exclusion_list = [
'AWS::EC2::Subnet',
'AWS::EC2::VPC',
'AWS::EC2::SecurityGroup'
]
def get_config_resources():
url = "https://docs.aws.amazon.com/config/latest/developerguide/resource-config-reference.html#supported-resources"
page = urlopen(url)
html = page.read().decode("utf-8")
# Target format: AWS::ApiGateway::Stage
pattern = "AWS::.*"
match_results = re.findall(pattern, html)
cleaned_list = []
count = 0
for result in match_results:
# remove HTML tags
results = re.sub("<.*?>", "", result)
# remove ending *
results = results.replace("*", "")
# remove space
results = results.replace(" ", "")
# remove long items (sentences)
if len(results) >= 60:
continue
# distinct list while preserving order
list(dict.fromkeys(results))
# Count items
count += 1
# Create the target cleaned list
cleaned_list.append(results)
logging.info("Scraped Config supported resources: %s", count)
return cleaned_list
def apply_custom_recorder(config_resources):
# Remove excluded resources from the globql list
result_list = list(set(config_resources) - set(exclusion_list))
# counter
count_result = 0
# Count resulted number of resource types (minus excluded types)
for type in result_list:
count_result += 1
logging.info("result_types: %s", count_result)
client = boto3.client('config')
try:
r = client.put_configuration_recorder(
ConfigurationRecorder={
'name': recorder_name,
'roleARN': role_arn,
'recordingGroup': {
'allSupported': False,
'includeGlobalResourceTypes': False,
'resourceTypes': result_list
}
}
)
except Exception as e:
logging.error(e)
logging.info("Response: %s", r)
if __name__ == "__main__":
config_resources = get_config_resources()
apply_custom_recorder(config_resources)
|
Gist version if you want to contribute.
I think this approach could help someone else, so I’m sharing it with you. Don’t hesitate to comment, and enhance it as you like.
That’s all folks!
zoph.