The day when the AWS Support got access to your S3 data
Update from: 2021-12-23
Official Security bulletin from AWS AWSSupportServiceRolePolicy Informational Update
You will find below details about the security incident that leads to this unattended access for millions of AWS customers.
22nd December 2021, AWS deployed a new version (
AWSSupportServiceRolePolicy used by a mandatory role:
AWSServiceRoleForSupport for AWS Support access to all AWS Accounts. In this policy, they added the action:
s3:getObject which gives access to all customer Amazon S3 data by AWS Support teams.
Immediately, @mamip_aws bot detected this change and tweeted about it.
AWSSupportServiceRolePolicy ... https://t.co/3t6zzEDrPe— MAMIP - Monitor AWS Managed IAM Policies (@mamip_aws) December 22, 2021
Then, Scott Piper and cloudsec community of experts contacted AWS Security teams to alert them about this security incident.
I've emailed firstname.lastname@example.org— Scott Piper (@0xdabbad00) December 22, 2021
A few hours later, in fact, nearly 10 hours later. AWS reverted this change and made the
v19 version the default version for everyone.
How we can audit events like this?
- You should enable CloudTrail S3 Data Events (❗ it could be costly for AWS accounts with a lot of S3 object movements)
- Using CloudTrail, review when the support role was last used (
- Enable server access logs - you will not have a full picture here as some info is missing.
CloudWatch Insights query
In this example, you will search for
AssumeRole events on CloudTrail events invokedBy AWS Support teams.
fields @timestamp, @message | filter eventSource = "sts.amazonaws.com" and eventName = "AssumeRole" and userIdentity.invokedBy = "support.amazonaws.com"
How to prevent this kind of incident?
- ❌ You can’t delete the Support role:
- ❌ You can’t use Service Control Policies (SCPs) on service-linked-role
- ❌ You can’t add inline or managed policy with
denystatement on this IAM Policy
- ✅ You can use KMS-CMK or client-side encryption to control access
- ✅ You can limit using resource policy (buckets policies)
- IAM is HARD, even AWS is failing at it.
- Changes made to IAM should always be peer-reviewed, manually, and using linting tools.
- Encrypt everything using your own customer-managed keys or client-side encryption.
AWS Security Digest
Don’t miss AWS Security related news with AWS Security Digest Newsletter (once a week): http://asd.zoph.io
Follow this army of AWS Twitter Bots:
- @mamip_aws: Monitor AWS Managed IAM Policies
- @mase_aws: Monitor AWS Services Endpoints and Regions
To be fair, a sum-up of AWS answers since:
- Rolled back because: rollback first, investigate, explain
- AWS Support internal systems are incapable of making an s3 get on customers objects “by design” - trust us
- The first intention with this change was to give the ability to make
s3:getObjectto get S3 metadata.
Read the following thread to better understand their positions:
The tweet below is false.— @email@example.com (@_msw_) December 24, 2021
Despite the permissions added in the AWSServiceRoleForSupport update that were subsequently rolled back, AWS Support *never* had access to customer data via S3 GetObject actions. https://t.co/2OBrh6D8CE
AWS just released a security bulletin about their addition of s3:GetObject to the support role: https://t.co/oJy7YUUAXX— Scott Piper (@0xdabbad00) December 24, 2021
Below is what I wrote in 2020, where I reference various previous blunders. A few short months later they had the ReadOnlyAccess disaster. 🧵 https://t.co/y8RwYWdXD6
That’s all folks! 👋