Update from: 2021-12-23

Official Security bulletin from AWS AWSSupportServiceRolePolicy Informational Update

https://aws.amazon.com/security/security-bulletins/AWS-2021-007/

You will find below details about the security incident that leads to this unattended access for millions of AWS customers.

On 22nd December 2021, AWS deployed a new version (v20) of AWSSupportServiceRolePolicy used by a mandatory role: AWSServiceRoleForSupport for AWS Support access to all AWS Accounts. In this policy, they added the action: s3:getObject which gives access to all customer Amazon S3 data by AWS Support teams.

Immediately, @mamip_aws bot detected this change and tweeted about it.

Then, Scott Piper and cloudsec community of experts contacted AWS Security teams to alert them about this security incident.

A few hours later, in fact, nearly 10 hours later. AWS reverted this change and made the v19 version the default version for everyone.

revert-to-v19-version

How we can audit events like this?

  1. You should enable CloudTrail S3 Data Events (❗ it could be costly for AWS accounts with a lot of S3 object movements)
  2. Using CloudTrail, review when the support role was last used (AssumeRole)
  3. Enable server access logs - you will not have a full picture here as some info is missing.

CloudWatch Insights query

In this example, you will search for AssumeRole events on CloudTrail events invokedBy AWS Support teams.

1
2

fields @timestamp, @message
| filter eventSource = "sts.amazonaws.com" and eventName = "AssumeRole" and userIdentity.invokedBy = "support.amazonaws.com"

CloudWatch Insights

How to prevent this kind of incident?

  • ❌ You can’t delete the Support role: AWSServiceRoleForSupport
  • ❌ You can’t use Service Control Policies (SCPs) on service-linked-role
  • ❌ You can’t add inline or managed policy with deny statement on this IAM Policy
  • ✅ You can use KMS-CMK or client-side encryption to control access
  • ✅ You can limit using resource policy (buckets policies)

Takeaways

  1. IAM is HARD, even AWS is failing at it.
  2. Changes made to IAM should always be peer-reviewed, manually, and using linting tools.
  3. Encrypt everything using your own customer-managed keys or client-side encryption.

AWS Security Digest

Don’t miss AWS Security related news with AWS Security Digest Newsletter (once a week): http://asd.zoph.io

Follow this army of AWS Twitter Bots:

  • @mamip_aws: Monitor AWS Managed IAM Policies
  • @mase_aws: Monitor AWS Services Endpoints and Regions

AWS Statement

To be fair, a sum-up of AWS answers since:

  • Rolled back because: rollback first, investigate, explain
  • AWS Support internal systems are incapable of making an s3 get on customers objects “by design” - trust us
  • The first intention with this change was to give the ability to make HeadObject which needs s3:getObject to get S3 metadata.

Read the following thread to better understand their positions:

That’s all folks! 👋

zoph.