Whitepaper 'FinOps and cost management for Kubernetes'
Please consider giving OptScale a Star on GitHub, it is 100% open source. It would increase its visibility to others and expedite product development. Thank you!
Webinar 'FinOps and cloud cost optimization for ML/AI workloads.' Register here →
Ebook 'From FinOps to proven cloud cost management & optimization strategies'
OptScale — FinOps
FinOps overview
Cost optimization:
AWS
MS Azure
Google Cloud
Alibaba Cloud
Kubernetes
OptScale — MLOps
ML/AI Profiling
ML/AI Optimization
Big Data Profiling
OPTSCALE PRICING
Acura — Cloud migration
Overview
Database replatforming
Migration to:
AWS
MS Azure
Google Cloud
Alibaba Cloud
VMWare
OpenStack
KVM
Public Cloud
Migration from:
On-premise
Acura — DR & cloud backup
Overview
Migration to:
AWS
MS Azure
Google Cloud
Alibaba Cloud
VMWare
OpenStack
KVM

The quickest way to get a list of public buckets in AWS to enhance your security

Problem description

Storing objects in AWS S3 public buckets could threaten the security of your company data. Public access is not granted to new AWS buckets and objects by default. However, in most cases public buckets are created by members of your engineering team occasionally or the reason lies in a lack of knowledge and experience of configuring AWS buckets properly. Therefore your critical data can become accessible to anyone possessing the link. The best way to prevent such unsecure cases is to avoid public buckets except special cases, when it is really needed. 

The_quickest_way_to_get_a_list_of_AWS_public_buckets

However it is a tricky task to set up centralized control under buckets with public access settings. The AWS console provides users with an opportunity to identify these buckets by filtering, but we have great doubts that your engineers will execute this procedure on a daily basis to avoid unexpected public buckets. 

A lack of notification system makes it difficult to keep the public bucket lifecycle under control.

Our recommendations will help you manage public access to Amazon S3 resources properly and ensure that all required buckets and objects have their public access blocked.

How to generate a list of bucket names which has public access in a straightforward way

Recommendation:

Find all buckets with public access using aws cli. It’s not necessary to install aws cli into your system. You can use aws cli right from aws console. Here is an instruction: https://aws.amazon.com/cloudshell/

A bucket could be marked as public in two cases:

  1. It has public policy configured.
  2. It has public acls configured.

At the same time, public access shouldn’t be blocked by a public access block, otherwise the bucket is not public.

Free cloud cost optimization. Lifetime

1. Public policy case

1.1 List all of the user’s buckets, and output the name, as text

aws s3api list-buckets --query 'Buckets[*].[Name]' --output text

1.2 Check bucket policy status

aws s3api get-bucket-policy-status --bucket "$bucket" --query 'PolicyStatus.IsPublic' --output text

1.3 Check that public access doesn’t blocked by public access block configuration

aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicPolicy' --output text

As a result one line command:

for bucket in $(aws s3api list-buckets --query 'Buckets[*].[Name]' --output text);do if [[ $(aws s3api get-bucket-policy-status --bucket "$bucket" --query 'PolicyStatus.IsPublic' --output text 2>/dev/null) == True ]] && [[ $(aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicPolicy' --output text 2>/dev/null) == False ]]; then echo "$bucket"; fi; done

2. Public acls case

2.1  List all of the user’s buckets, and output the name, as text

See point 1.1

2.2 Check bucket acls that contains AllUsers permissions

aws s3api get-bucket-acl --bucket "$bucket" --query 'Grants[?Grantee.URI==`http://acs.amazonaws.com/groups/global/AllUsers`]' --output text

2.3 Check that public acls isn’t blocked by public access block configuration

aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicAcls' --output text

As a result one line command:

for bucket in $(aws s3api list-buckets --query 'Buckets[*].[Name]' --output text);do if [[ $(aws s3api get-bucket-acl --bucket "$bucket" --query 'Grants[?Grantee.URI==`http://acs.amazonaws.com/groups/global/AllUsers`]' --output text 2> /dev/null) ]] && [[ $(aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicAcls' --output text 2>/dev/null) != True ]]; then echo "$bucket"; fi; done

Result

Both cases in one line:

for bucket in $(aws s3api list-buckets --query 'Buckets[*].[Name]' --output text); do if [[ $(aws s3api get-bucket-acl --bucket "$bucket" --query 'Grants[?Grantee.URI==`http://acs.amazonaws.com/groups/global/AllUsers`]' --output text 2> /dev/null) ]] && [[ $(aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicAcls' --output text 2>/dev/null) != True ]]; then echo "$bucket"; elif [[ $(aws s3api get-bucket-policy-status --bucket "$bucket" --query 'PolicyStatus.IsPublic' --output text 2>/dev/null) == True ]] && [[ $(aws s3api get-public-access-block --bucket "$bucket" --query 'PublicAccessBlockConfiguration.BlockPublicPolicy' --output text 2>/dev/null) == False ]]; then echo "$bucket"; fi; done

This command will generate a list of bucket names which has public access.

👆🏻 Dozens of AWS volume snapshots are usually created by your engineering team on a daily basis. But some challenges may occur while deleting instances and companies continue to pay for them. 
💡 Find the ways of identifying and deleting orphaned and unused snapshots to avoid wastage and achieve optimization goals → https://hystax.com/reduce-your-aws-bill-by-cleaning-orphaned-and-unused-disk-snapshots/

Enter your email to be notified about new and relevant content.

Thank you for joining us!

We hope you'll find it usefull

You can unsubscribe from these communications at any time. Privacy Policy

News & Reports

FinOps and MLOps

A full description of OptScale as a FinOps and MLOps open source platform to optimize cloud workload performance and infrastructure cost. Cloud cost optimization, VM rightsizing, PaaS instrumentation, S3 duplicate finder, RI/SP usage, anomaly detection, + AI developer tools for optimal cloud utilization.

FinOps, cloud cost optimization and security

Discover our best practices: 

  • How to release Elastic IPs on Amazon EC2
  • Detect incorrectly stopped MS Azure VMs
  • Reduce your AWS bill by eliminating orphaned and unused disk snapshots
  • And much more deep insights

FinOps and cloud cost optimization for ML/AI workloads

Join our live demo on 27th 
March and discover how OptScale allows running ML/AI or any type of workload with optimal performance and infrastructure cost.