The day when the AWS Support got access to your S3 data
Update from: 2021-12-23
Official Security bulletin from AWS AWSSupportServiceRolePolicy Informational Update
https://aws.amazon.com/security/security-bulletins/AWS-2021-007/
You will find below details about the security incident that leads to this unattended access for millions of AWS customers.
On 22nd December 2021
, AWS deployed a new version (v20
) of AWSSupportServiceRolePolicy
used by a mandatory role: AWSServiceRoleForSupport
for AWS Support access to all AWS Accounts. In this policy, they added the action: s3:getObject
which gives access to all customer Amazon S3 data by AWS Support teams.
Immediately, @mamip_aws bot detected this change and tweeted about it.
AWSSupportServiceRolePolicy ... https://t.co/3t6zzEDrPe
— MAMIP - Monitor AWS Managed IAM Policies (@mamip_aws) December 22, 2021
Then, Scott Piper and cloudsec community of experts contacted AWS Security teams to alert them about this security incident.
I've emailed aws-security@amazon.com
— Scott Piper (@0xdabbad00) December 22, 2021
A few hours later, in fact, nearly 10 hours later. AWS reverted this change and made the v19
version the default version for everyone.
How we can audit events like this?
- You should enable CloudTrail S3 Data Events (โ it could be costly for AWS accounts with a lot of S3 object movements)
- Using CloudTrail, review when the support role was last used (
AssumeRole
) - Enable server access logs - you will not have a full picture here as some info is missing.
CloudWatch Insights query
In this example, you will search for AssumeRole
events on CloudTrail events invokedBy AWS Support teams.
fields @timestamp, @message
| filter eventSource = "sts.amazonaws.com" and eventName = "AssumeRole" and userIdentity.invokedBy = "support.amazonaws.com"
How to prevent this kind of incident?
- โ You can’t delete the Support role:
AWSServiceRoleForSupport
- โ You can’t use Service Control Policies (SCPs) on service-linked-role
- โ You can’t add inline or managed policy with
deny
statement on this IAM Policy - โ You can use KMS-CMK or client-side encryption to control access
- โ You can limit using resource policy (buckets policies)
Takeaways
- IAM is HARD, even AWS is failing at it.
- Changes made to IAM should always be peer-reviewed, manually, and using linting tools.
- Encrypt everything using your own customer-managed keys or client-side encryption.
AWS Security Digest
Don’t miss AWS Security related news with AWS Security Digest Newsletter (once a week): http://asd.zoph.io
Follow this army of AWS Twitter Bots:
- @mamip_aws: Monitor AWS Managed IAM Policies
- @mase_aws: Monitor AWS Services Endpoints and Regions
AWS Statement
To be fair, a sum-up of AWS answers since:
- Rolled back because: rollback first, investigate, explain
- AWS Support internal systems are incapable of making an s3 get on customers objects “by design” - trust us
- The first intention with this change was to give the ability to make
HeadObject
which needss3:getObject
to get S3 metadata.
Read the following thread to better understand their positions:
The tweet below is false.
— @msw@mstdn.social (@_msw_) December 24, 2021
Despite the permissions added in the AWSServiceRoleForSupport update that were subsequently rolled back, AWS Support *never* had access to customer data via S3 GetObject actions. https://t.co/2OBrh6D8CE
AWS just released a security bulletin about their addition of s3:GetObject to the support role: https://t.co/oJy7YUUAXX
— Scott Piper (@0xdabbad00) December 24, 2021
Below is what I wrote in 2020, where I reference various previous blunders. A few short months later they had the ReadOnlyAccess disaster. ๐งต https://t.co/y8RwYWdXD6
That’s all folks! ๐
zoph.