CertifyDeckCargando contenido…
The official AWS Certified Data Engineer - Associate syllabus, organized by domain. Check off each point as you review it.
Inicia sesión para guardar tu progreso entre sesiones.
Perform data ingestion
Amazon Redshift can read directly from Kinesis Data Streams or Amazon MSK into a materialized view without staging in S3, and AUTO REFRESH keeps it current. DynamoDB Streams keeps changes for 24 hours; Kinesis Data Streams for DynamoDB offers longer retention and multiple consumers. AWS Glue provides continuous streaming ETL jobs over Kinesis, Kafka and MSK.
Streaming ingestion to a materialized view - Amazon Redshift ↗Streaming ETL jobs in AWS Glue ↗
S3DistCp copies data from Amazon S3 to HDFS on Amazon EMR in a distributed way and can concatenate small files with --groupBy. AWS DMS offers full load, full load plus CDC or changes only, and writes CSV or Parquet to an S3 target. Redshift COPY loads multiple similarly sized compressed files in parallel when given a common prefix or a manifest.
S3DistCp (s3-dist-cp) - Amazon EMR ↗Loading data from compressed and uncompressed files - Amazon Redshift ↗
In Redshift, a manifest lists the exact files COPY must load, even across buckets, and mandatory=true makes the load fail if one is missing. In AWS Glue, groupFiles and groupSize group many small S3 files into fewer partitions. Scheduled AppFlow flows support full or incremental transfer, with an optional time offset for late source timestamps.
Using a manifest to specify data files - Amazon Redshift ↗Reading input files in larger groups - AWS Glue ↗
Amazon AppFlow transfers data from SaaS applications such as Salesforce or Zendesk to AWS services without code, on demand, on event or on a schedule. The Amazon Redshift Data API runs SQL through an HTTP endpoint without drivers or persistent connections; it is asynchronous and authenticates with Secrets Manager or temporary credentials.
What is Amazon AppFlow? - Amazon AppFlow ↗Using the Amazon Redshift Data API - Amazon Redshift ↗
AWS Glue supports scheduled triggers with cron in UTC, conditional triggers based on job or crawler states, and on-demand triggers. Amazon EventBridge Scheduler creates cron, rate or one-time schedules in a chosen time zone with daylight saving adjustment and can call AWS APIs. Amazon MWAA runs Apache Airflow DAGs without managing infrastructure.
AWS Glue triggers ↗Schedule types in EventBridge Scheduler ↗
S3 Event Notifications sends object events to SNS, standard SQS, Lambda or EventBridge; FIFO queues require EventBridge. A function that writes to the bucket that triggers it can loop, which two buckets or an incoming prefix prevent. An AWS Glue workflow can be started by EventBridge events and can batch them with an event batching condition.
Amazon S3 Event Notifications - Amazon Simple Storage Service ↗Starting an AWS Glue workflow with an Amazon EventBridge event ↗
Lambda reads Kinesis shards through an event source mapping, using a standard iterator or an enhanced fan-out consumer. BatchSize and the batching window control invocation size, ParallelizationFactor processes one shard with concurrent batches, and batch bisecting, a retry limit and an on-failure destination stop a bad record from blocking the shard.
Using Lambda to process records from Amazon Kinesis Data Streams - AWS Lambda ↗Retain discarded batch records for a Kinesis Data Streams event source in Lambda - AWS Lambda ↗
To let a partner allowlist outbound traffic from resources in private subnets, use a public NAT gateway with an Elastic IP. Customer-managed prefix lists group CIDR ranges that security groups reference and are updated in one place. Firehose delivers to Redshift in a VPC when the cluster is publicly accessible and its security group allows the Firehose CIDR block for the Region.
Consolidate and manage network CIDR blocks with managed prefix lists - Amazon VPC ↗Controlling access with Amazon Data Firehose ↗
DynamoDB returns unprocessed batch writes in UnprocessedItems, usually because of throttling, and they should be retried with exponential backoff; the SDKs already retry throttling errors. RDS Proxy pools and reuses connections to absorb surges without overwhelming the database. In Kinesis, a well-distributed partition key avoids hot shards that cause ProvisionedThroughputExceededException.
Error handling with DynamoDB - Amazon DynamoDB ↗Amazon RDS Proxy - Amazon Relational Database Service ↗
To fan out a Kinesis stream to several consumers, enhanced fan-out gives each one its own per-shard throughput with push delivery, while standard consumers and Firehose share the read limit. To distribute messages, an SNS topic with several subscribed SQS queues gives each service a persistent copy. Firehose can read from a Kinesis data stream and write to S3.
Develop enhanced fan-out consumers with dedicated throughput - Amazon Kinesis Data Streams ↗Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing ↗
A pipeline is replayable when it keeps source data long enough to reprocess it. Kinesis Data Streams lets you extend retention and read from a point in time with AT_TIMESTAMP; EventBridge archives events and replays them to the source bus; MSK tiered storage retains more data; and AWS Glue job bookmarks can be rewound to an earlier run.
Change the data retention period - Amazon Kinesis Data Streams ↗Archiving and replaying events in Amazon EventBridge ↗
Stateless processing handles each event on its own; stateful processing keeps information across events. Lambda is stateless by default, but tumbling windows keep state across invocations. Managed Service for Apache Flink uses checkpoints and snapshots to preserve state. DynamoDB TransactWriteItems applies several changes as all-or-nothing and is idempotent with a client token.
Implementing stateful Kinesis Data Streams processing in Lambda - AWS Lambda ↗Amazon DynamoDB Transactions: How it works ↗
Transform and process data
Amazon EMR on EKS runs Spark and other frameworks on an existing EKS cluster and manages their containers, sharing resources with other applications. In Amazon ECS, Service Auto Scaling adjusts tasks based on CPU, memory, schedules, historical patterns or SQS backlog per task. Fargate Spot runs interruption-tolerant tasks at a discount and sends SIGTERM before stopping them.
What is Amazon EMR on EKS? - Amazon EMR ↗Amazon ECS clusters for Fargate - Amazon ECS ↗
AWS Glue connects to databases through JDBC connections; with Amazon RDS in a VPC it needs a security group with a self-referencing inbound rule, and hashfield and hashpartitions read a JDBC table in parallel. Athena offers JDBC and ODBC drivers for BI tools and applications, and Amazon Redshift offers its own JDBC and ODBC drivers.
Reading from JDBC tables in parallel - AWS Glue ↗Setting up Amazon VPC for JDBC connections to Amazon RDS data stores from AWS Glue ↗
Athena Federated Query reads sources such as DynamoDB in place through connectors, and Redshift federated queries reach RDS and Aurora PostgreSQL or MySQL by pushing part of the work to the remote database. Zero-ETL integrations replicate Aurora, RDS, DynamoDB or SaaS applications into Redshift without a self-managed pipeline, and Amazon AppFlow moves SaaS data to S3 without code.
Use Amazon Athena Federated Query ↗Zero-ETL integrations - Amazon Redshift ↗
In AWS Glue, the flexible execution class lowers the cost of non-urgent jobs (testing, one-time loads), and Auto Scaling, from Glue 3.0, adds and removes workers based on the parallelism of each stage. In Amazon EMR, Spot Instances suit task nodes, which hold no HDFS data; primary and core nodes should be On-Demand when the cluster must not be lost.
Using auto scaling for AWS Glue ↗Configuring Amazon EMR cluster instance types and best practices for Spot instances ↗
EMR Serverless runs Spark and Hive without operating clusters; AWS Glue offers serverless Spark, Spark Streaming and Python shell jobs. Lambda fits small transformations triggered by S3 events, and in Amazon Redshift materialized views store precomputed results of joins and aggregations with automatic refresh.
What is Amazon EMR Serverless? ↗Materialized views in Amazon Redshift ↗
Parquet and ORC are columnar formats that save space and speed up queries. Athena converts tables with CTAS by setting format and write_compression, and Amazon Data Firehose converts JSON to Parquet or ORC with a deserializer (OpenX or Hive JSON SerDe), an AWS Glue Data Catalog schema and a serializer; CSV must first be turned into JSON with Lambda.
Convert input data format in Amazon Data Firehose ↗Examples of CTAS queries - Amazon Athena ↗
AWS Glue job profiling and the Spark UI help locate failures: many small files exhaust driver memory (fixed by grouping with groupFiles), a JDBC read without a fetch size exhausts one executor (dynamic frames set one by default), and coalesce(1) creates a straggler task that leaves the other executors idle.
Debugging OOM exceptions and job abnormalities ↗Debugging demanding stages and straggler tasks ↗
The Redshift Data API runs SQL over HTTP without drivers or persistent connections, using Secrets Manager or temporary credentials; BatchExecuteStatement runs several statements in one transaction. To publish data to third parties, API Gateway provides REST APIs with API keys and usage plans that throttle per client, and stage caching so the backend is not called on every request.
Using the Amazon Redshift Data API ↗Usage plans and API keys for REST APIs in API Gateway ↗
Structured data fits a schema of rows and columns; unstructured data (audio, video, documents) does not, and semi-structured data mixes both, such as videos with tags. An S3 data lake stores any variety as-is, and for high velocity with unpredictable traffic Kinesis Data Streams in on-demand mode scales without capacity planning.
Structured Data vs Unstructured Data ↗Choose the right mode to stream in - Amazon Kinesis Data Streams ↗
Amazon Redshift can call Amazon Bedrock models from SQL with CREATE EXTERNAL MODEL and MODEL_TYPE BEDROCK, which creates an inference function. For large volumes without real-time needs, Bedrock batch inference reads JSONL files from S3 with recordId and modelInput and writes the responses to S3; output order is not guaranteed, so results are joined on recordId.
CREATE EXTERNAL MODEL - Amazon Redshift ↗Format and upload your batch inference data ↗
Orchestrate data pipelines
Amazon MWAA runs Apache Airflow DAGs written in Python without managing infrastructure. AWS Glue workflows chain jobs and crawlers and can start on a schedule, on demand or from EventBridge events batched by size and window. In Step Functions, the .sync pattern waits for the integrated job to finish, while Request Response moves on as soon as it gets the response.
What Is Amazon Managed Workflows for Apache Airflow? ↗Discover service integration patterns in Step Functions ↗
In Step Functions, Retry fields with BackoffRate and JitterStrategy spread out retries and Catch routes errors to handling states; the Distributed Map state supports tolerated failure thresholds and MaxConcurrency so downstream services are not overwhelmed. In Amazon SQS, a dead-letter queue with maxReceiveCount isolates messages that always fail without stopping the rest.
Handling errors in Step Functions workflows ↗Using dead-letter queues in Amazon SQS ↗
Step Functions offers Standard workflows, long-running and exactly-once for non-idempotent actions, and Express workflows, at-least-once and built for high-volume events with idempotent actions; the type cannot be changed after creation. The Distributed Map state, available only in Standard workflows, processes large S3 datasets in parallel with ItemReader and ResultWriter.
Choosing workflow type in Step Functions ↗Using Map state in Distributed mode for large-scale parallel workloads in Step Functions ↗
Amazon SNS pushes messages to many subscribers, and with subscribed SQS queues the messages persist for later processing; filter policies are set on each subscription. AWS Glue sends job state changes (FAILED, TIMEOUT and others) to EventBridge and, with a delay threshold, events for slow runs that a rule can route to SNS.
Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing ↗Automating AWS Glue with EventBridge ↗
Apply programming concepts
To speed up Spark jobs in AWS Glue, read less data: prune partitions with push_down_predicate, select columns and filter rows before joining to benefit from pushdown on Parquet, and avoid shuffles with broadcast hash joins when one table fits in executor memory.
Reduce the amount of data scan - AWS Prescriptive Guidance ↗Optimize shuffles - AWS Prescriptive Guidance ↗
In Lambda, reserved concurrency guarantees and caps a function's capacity, while provisioned concurrency keeps pre-initialized environments to avoid cold starts. CPU grows with configured memory. With Kinesis, ParallelizationFactor processes several batches per shard while keeping per-key order, and with SQS maximum concurrency limits each event source separately.
Understanding Lambda Event Function scaling ↗Configuring scaling behavior for SQS event source mappings ↗
AWS Glue for Spark accepts ETL scripts in Python (PySpark) and Scala. In Amazon Athena, user-defined functions are written in Java, deployed as a Lambda function and called with USING EXTERNAL FUNCTION. In Amazon Redshift, PL/pgSQL stored procedures support DDL, DML, loops and conditions, unlike UDFs.
Programming Spark scripts - AWS Glue ↗Query with user defined functions - Amazon Athena ↗
Software engineering best practices also apply to data pipelines. AWS Glue Studio integrates with Git (GitHub, GitLab, Bitbucket, CodeCommit) to version jobs; the AWS Glue Docker image lets you test locally with pytest without Glue cost; glueContext.get_logger() writes to CloudWatch, and observability metrics show skew and utilization.
Using Git version control systems in AWS Glue ↗Monitoring with AWS Glue Observability metrics ↗
With AWS CloudFormation, data infrastructure is described in templates that can be versioned. Change sets preview whether a resource will be modified, replaced or deleted before execution; drift detection finds changes made outside CloudFormation; and the DeletionPolicy attribute (Delete, Retain, RetainExceptOnCreate, Snapshot) decides what happens to each resource when the stack is deleted.
Update CloudFormation stacks using change sets ↗DeletionPolicy attribute - CloudFormation ↗
AWS SAM is an IaC framework that extends CloudFormation with shorthand syntax for Lambda functions, Step Functions state machines and DynamoDB tables. sam deploy packages, uploads artifacts to S3 and deploys through a change set; AWS::Serverless::StateMachine supports DefinitionUri and DefinitionSubstitutions, and SimpleTable only creates single-attribute keys.
What is the AWS Serverless Application Model (AWS SAM)? ↗sam deploy - AWS SAM CLI command reference ↗
A Lambda function has ephemeral storage in /tmp, private to each execution environment and resizable with its own setting. For data shared across invocations, a file system is mounted on a /mnt/ path: Amazon EFS through an access point, or Amazon S3 Files, which presents a bucket as files. Both require a VPC, mount targets and NFS traffic on port 2049.
Configure ephemeral storage for Lambda functions ↗Configuring file system access for Lambda functions ↗
IaC enables repeatable deployment. CloudFormation StackSets creates or updates stacks across several accounts and Regions with one operation. The AWS CDK defines infrastructure in languages such as Python or TypeScript: cdk synth produces the CloudFormation templates, each environment needs cdk bootstrap, and constructs range from L1 (Cfn, full control) to L3 (patterns).
Managing stacks across accounts and Regions with StackSets ↗What is the AWS CDK? ↗
CI/CD automates building, testing and deploying data pipelines. AWS CodePipeline models the stages and supports manual approval actions with SNS notifications; AWS CodeBuild runs tests and creates reports from the buildspec reports section; AWS CodeDeploy shifts traffic to new Lambda versions with canary, linear or all-at-once configurations.
Distributed computing spreads data and work across several nodes that process in parallel. In Spark, each partition is one task, so an unsplittable gzip file creates a single partition until repartition is used. In Redshift, the leader node plans and compute nodes run in parallel; in EMR, core nodes hold HDFS and task nodes only add compute.
Parallelize tasks - Best practices for performance tuning AWS Glue for Apache Spark jobs ↗Understand node types in Amazon EMR: primary, core, and task nodes ↗
Graphs (nodes and edges) and trees (hierarchies) are modeled with suitable services. Amazon Neptune runs Gremlin, openCypher and SPARQL on highly connected data; Neptune Analytics loads graphs in memory and offers pathfinding, centrality and community algorithms. In DynamoDB, adjacency lists represent graphs and composite sort keys represent hierarchies queryable with begins_with.
Neptune Analytics algorithms ↗Best practices for using sort keys to organize data in DynamoDB ↗