A production-grade reference repository for Data Engineers, Cloud Engineers, Analytics Engineers, and DevOps Engineers working with AWS.
This repository covers AWS services commonly used in data engineering workflows through the AWS CLI v2 and Python SDK (Boto3) — with real-world examples, security guidance, troubleshooting, and best practices.
# Verify AWS CLI v2
aws --version
# Configure credentials (or use SSO / IAM roles)
aws configure
# Verify identity
aws sts get-caller-identity# Verify Boto3
import boto3
print(boto3.__version__)
session = boto3.Session()
print(session.client("sts").get_caller_identity())| Service | Description | Path |
|---|---|---|
| S3 | Object storage, data lakes, ETL staging | s3/ |
| IAM | Identity, access, roles, policies | iam/ |
| EC2 | Compute instances for pipelines | ec2/ |
| VPC | Networking, subnets, endpoints | vpc/ |
| RDS | Relational databases | rds/ |
| DynamoDB | NoSQL key-value store | dynamodb/ |
| Lambda | Serverless compute | lambda/ |
| Glue | ETL, crawlers, Data Catalog | glue/ |
| Athena | Serverless SQL on S3 | athena/ |
| Redshift | Data warehouse | redshift/ |
| CloudWatch | Monitoring, logs, alarms | cloudwatch/ |
| STS | Temporary credentials, assume role | sts/ |
| Secrets Manager | Secret rotation and retrieval | secrets_manager/ |
| KMS | Encryption keys | kms/ |
| ECR | Container registry | ecr/ |
| ECS | Container orchestration | ecs/ |
| EKS | Managed Kubernetes | eks/ |
| EMR | Big data processing (Spark, Hive) | emr/ |
| SNS | Pub/sub notifications | sns/ |
| SQS | Message queues | sqs/ |
| EventBridge | Event bus, scheduling | eventbridge/ |
| Step Functions | Workflow orchestration | step_functions/ |
| Lake Formation | Data lake governance | lake_formation/ |
| AWS Backup | Centralized backup | aws_backup/ |
| Organizations | Multi-account management | organizations/ |
| Config | Resource compliance | config/ |
| CloudFormation | Infrastructure as code | cloudformation/ |
| Boto3 | SDK patterns and utilities | boto3/ |
| Terraform | CLI + Terraform integration | terraform_integration/ |
| Troubleshooting | Common errors and fixes | troubleshooting/ |
Every service file includes:
- Service Overview — what it does, use cases, when to use it
- AWS CLI Commands — purpose, syntax, parameters, examples, sample output
- Advanced Commands — JMESPath, pagination, filtering, batch ops
- Boto3 Examples — basic, production-ready, error handling, logging
- Security Considerations — least privilege, encryption, credentials
- Troubleshooting — common errors and resolutions
- Best Practices — cost, security, performance, production tips
| Pattern | Services |
|---|---|
| Data lake ingestion | S3, Glue, Lake Formation |
| Batch ETL | Glue, EMR, Step Functions |
| Streaming pipelines | Kinesis, Lambda, SQS, EventBridge |
| Ad-hoc analytics | Athena, Redshift |
| Orchestration | Step Functions, EventBridge |
| Secrets & encryption | Secrets Manager, KMS, IAM |
| Monitoring pipelines | CloudWatch, SNS |
- AWS CLI v2 installed (installation guide)
- Python 3.9+ with
boto3andbotocore - Valid AWS credentials (IAM user, SSO, or assumed role)
- Appropriate IAM permissions for the operations documented
When adding new commands or examples:
- Use AWS CLI v2 syntax
- Include real-world data engineering context
- Add sample output where helpful
- Document required IAM permissions
- Follow the existing file structure and section headings
See LICENSE.