The day when the AWS Support got access to your S3 data

Update from: 2021-12-23

Official Security bulletin from AWS AWSSupportServiceRolePolicy Informational Update

https://aws.amazon.com/security/security-bulletins/AWS-2021-007/

You will find below details about the security incident that leads to this unattended access for millions of AWS customers.

On 22nd December 2021, AWS deployed a new version (v20) of AWSSupportServiceRolePolicy used by a mandatory role: AWSServiceRoleForSupport for AWS Support access to all AWS Accounts. In this policy, they added the action: s3:getObject which gives access to all customer Amazon S3 data by AWS Support teams.

Immediately, @mamip_aws bot detected this change and tweeted about it.

AWSSupportServiceRolePolicy ... https://t.co/3t6zzEDrPe

— MAMIP - Monitor AWS Managed IAM Policies (@mamip_aws) December 22, 2021

Then, Scott Piper and cloudsec community of experts contacted AWS Security teams to alert them about this security incident.

I've emailed aws-security@amazon.com

— Scott Piper (@0xdabbad00) December 22, 2021

A few hours later, in fact, nearly 10 hours later. AWS reverted this change and made the v19 version the default version for everyone.

How we can audit events like this?

1. You should enable CloudTrail S3 Data Events (❗ it could be costly for AWS accounts with a lot of S3 object movements)
2. Using CloudTrail, review when the support role was last used (AssumeRole)
3. Enable server access logs - you will not have a full picture here as some info is missing.

CloudWatch Insights query

In this example, you will search for AssumeRole events on CloudTrail events invokedBy AWS Support teams.

fields @timestamp, @message
| filter eventSource = "sts.amazonaws.com" and eventName = "AssumeRole" and userIdentity.invokedBy = "support.amazonaws.com"

How to prevent this kind of incident?

• ❌ You can’t delete the Support role: AWSServiceRoleForSupport
• ❌ You can’t use Service Control Policies (SCPs) on service-linked-role
• ❌ You can’t add inline or managed policy with deny statement on this IAM Policy
• ✅ You can use KMS-CMK or client-side encryption to control access
• ✅ You can limit using resource policy (buckets policies)

Takeaways

1. IAM is HARD, even AWS is failing at it.
2. Changes made to IAM should always be peer-reviewed, manually, and using linting tools.
3. Encrypt everything using your own customer-managed keys or client-side encryption.

AWS Security Digest

Don’t miss AWS Security related news with AWS Security Digest Newsletter (once a week): http://asd.zoph.io

Follow this army of AWS Twitter Bots:

• @mamip_aws: Monitor AWS Managed IAM Policies
• @mase_aws: Monitor AWS Services Endpoints and Regions

AWS Statement

To be fair, a sum-up of AWS answers since:

• Rolled back because: rollback first, investigate, explain
• AWS Support internal systems are incapable of making an s3 get on customers objects “by design” - trust us
• The first intention with this change was to give the ability to make HeadObject which needs s3:getObject to get S3 metadata.

Read the following thread to better understand their positions:

The tweet below is false.

Despite the permissions added in the AWSServiceRoleForSupport update that were subsequently rolled back, AWS Support *never* had access to customer data via S3 GetObject actions. https://t.co/2OBrh6D8CE

— @msw@mstdn.social (@_msw_) December 24, 2021

AWS just released a security bulletin about their addition of s3:GetObject to the support role: https://t.co/oJy7YUUAXX

Below is what I wrote in 2020, where I reference various previous blunders. A few short months later they had the ReadOnlyAccess disaster. 🧵 https://t.co/y8RwYWdXD6

— Scott Piper (@0xdabbad00) December 24, 2021

That’s all folks! 👋

zoph.