CertifyDeck
Start for free
All cheat sheets
DEA-C01 · CHEAT SHEET

AWS Certified Data Engineer - Associate

Key concepts grouped by official exam weight.

Download as PDF (free, no sign-up)

1. Data Ingestion and Transformation

34–34%

Perform data ingestion

  • Amazon Redshift can read directly from Kinesis Data Streams or Amazon MSK into a materialized view without staging in S3, and AUTO REFRESH keeps it current. DynamoDB Streams keeps changes for 24 hours; Kinesis Data Streams for DynamoDB offers longer retention and multiple consumers. AWS Glue provides continuous streaming ETL jobs over Kinesis, Kafka and MSK.
  • S3DistCp copies data from Amazon S3 to HDFS on Amazon EMR in a distributed way and can concatenate small files with --groupBy. AWS DMS offers full load, full load plus CDC or changes only, and writes CSV or Parquet to an S3 target. Redshift COPY loads multiple similarly sized compressed files in parallel when given a common prefix or a manifest.
  • In Redshift, a manifest lists the exact files COPY must load, even across buckets, and mandatory=true makes the load fail if one is missing. In AWS Glue, groupFiles and groupSize group many small S3 files into fewer partitions. Scheduled AppFlow flows support full or incremental transfer, with an optional time offset for late source timestamps.
  • Amazon AppFlow transfers data from SaaS applications such as Salesforce or Zendesk to AWS services without code, on demand, on event or on a schedule. The Amazon Redshift Data API runs SQL through an HTTP endpoint without drivers or persistent connections; it is asynchronous and authenticates with Secrets Manager or temporary credentials.
  • AWS Glue supports scheduled triggers with cron in UTC, conditional triggers based on job or crawler states, and on-demand triggers. Amazon EventBridge Scheduler creates cron, rate or one-time schedules in a chosen time zone with daylight saving adjustment and can call AWS APIs. Amazon MWAA runs Apache Airflow DAGs without managing infrastructure.

Transform and process data

  • Amazon EMR on EKS runs Spark and other frameworks on an existing EKS cluster and manages their containers, sharing resources with other applications. In Amazon ECS, Service Auto Scaling adjusts tasks based on CPU, memory, schedules, historical patterns or SQS backlog per task. Fargate Spot runs interruption-tolerant tasks at a discount and sends SIGTERM before stopping them.
  • AWS Glue connects to databases through JDBC connections; with Amazon RDS in a VPC it needs a security group with a self-referencing inbound rule, and hashfield and hashpartitions read a JDBC table in parallel. Athena offers JDBC and ODBC drivers for BI tools and applications, and Amazon Redshift offers its own JDBC and ODBC drivers.
  • Athena Federated Query reads sources such as DynamoDB in place through connectors, and Redshift federated queries reach RDS and Aurora PostgreSQL or MySQL by pushing part of the work to the remote database. Zero-ETL integrations replicate Aurora, RDS, DynamoDB or SaaS applications into Redshift without a self-managed pipeline, and Amazon AppFlow moves SaaS data to S3 without code.
  • In AWS Glue, the flexible execution class lowers the cost of non-urgent jobs (testing, one-time loads), and Auto Scaling, from Glue 3.0, adds and removes workers based on the parallelism of each stage. In Amazon EMR, Spot Instances suit task nodes, which hold no HDFS data; primary and core nodes should be On-Demand when the cluster must not be lost.
  • EMR Serverless runs Spark and Hive without operating clusters; AWS Glue offers serverless Spark, Spark Streaming and Python shell jobs. Lambda fits small transformations triggered by S3 events, and in Amazon Redshift materialized views store precomputed results of joins and aggregations with automatic refresh.

Orchestrate data pipelines

  • Amazon MWAA runs Apache Airflow DAGs written in Python without managing infrastructure. AWS Glue workflows chain jobs and crawlers and can start on a schedule, on demand or from EventBridge events batched by size and window. In Step Functions, the .sync pattern waits for the integrated job to finish, while Request Response moves on as soon as it gets the response.
  • In Step Functions, Retry fields with BackoffRate and JitterStrategy spread out retries and Catch routes errors to handling states; the Distributed Map state supports tolerated failure thresholds and MaxConcurrency so downstream services are not overwhelmed. In Amazon SQS, a dead-letter queue with maxReceiveCount isolates messages that always fail without stopping the rest.
  • Step Functions offers Standard workflows, long-running and exactly-once for non-idempotent actions, and Express workflows, at-least-once and built for high-volume events with idempotent actions; the type cannot be changed after creation. The Distributed Map state, available only in Standard workflows, processes large S3 datasets in parallel with ItemReader and ResultWriter.
  • Amazon SNS pushes messages to many subscribers, and with subscribed SQS queues the messages persist for later processing; filter policies are set on each subscription. AWS Glue sends job state changes (FAILED, TIMEOUT and others) to EventBridge and, with a delay threshold, events for slow runs that a rule can route to SNS.

Apply programming concepts

  • To speed up Spark jobs in AWS Glue, read less data: prune partitions with push_down_predicate, select columns and filter rows before joining to benefit from pushdown on Parquet, and avoid shuffles with broadcast hash joins when one table fits in executor memory.
  • In Lambda, reserved concurrency guarantees and caps a function's capacity, while provisioned concurrency keeps pre-initialized environments to avoid cold starts. CPU grows with configured memory. With Kinesis, ParallelizationFactor processes several batches per shard while keeping per-key order, and with SQS maximum concurrency limits each event source separately.
  • AWS Glue for Spark accepts ETL scripts in Python (PySpark) and Scala. In Amazon Athena, user-defined functions are written in Java, deployed as a Lambda function and called with USING EXTERNAL FUNCTION. In Amazon Redshift, PL/pgSQL stored procedures support DDL, DML, loops and conditions, unlike UDFs.
  • Software engineering best practices also apply to data pipelines. AWS Glue Studio integrates with Git (GitHub, GitLab, Bitbucket, CodeCommit) to version jobs; the AWS Glue Docker image lets you test locally with pytest without Glue cost; glueContext.get_logger() writes to CloudWatch, and observability metrics show skew and utilization.
  • With AWS CloudFormation, data infrastructure is described in templates that can be versioned. Change sets preview whether a resource will be modified, replaced or deleted before execution; drift detection finds changes made outside CloudFormation; and the DeletionPolicy attribute (Delete, Retain, RetainExceptOnCreate, Snapshot) decides what happens to each resource when the stack is deleted.

2. Data Store Management

26–26%

Choose a data store

  • The right store depends on the required cost and performance. MSK Serverless runs Kafka without sizing the cluster, and Kinesis Data Streams offers on-demand and provisioned modes. Redshift RA3 nodes scale compute and storage separately. In DynamoDB, the Standard-IA class lowers cost for storage-dominated tables and provisioned mode suits steady traffic.
  • Each store is configured for its access pattern. In DynamoDB, global secondary indexes allow queries on attributes other than the key. In Amazon Redshift, KEY distribution collocates joined rows, EVEN suits tables without joins, and sort keys speed up range filters. In Amazon RDS, read replicas offload read-only queries.
  • Each use case has a suitable store. Aurora PostgreSQL with pgvector stores embeddings and speeds up approximate nearest neighbor search with HNSW indexes; Amazon MemoryDB is a durable in-memory database compatible with Valkey and Redis OSS with microsecond reads; Keyspaces serves Cassandra workloads and DocumentDB serves MongoDB-compatible applications.
  • AWS Transfer Family offers fully managed SFTP, FTPS, FTP and AS2 endpoints that store files in Amazon S3 or Amazon EFS without changing partner clients. Its managed workflows decrypt, tag, copy and call Lambda after each upload, and SFTP connectors retrieve files from or send files to external SFTP servers. AWS DataSync migrates data from NFS, SMB, HDFS and object storage.
  • Amazon Redshift reaches external data without migrating it: federated queries read live RDS and Aurora PostgreSQL and MySQL databases, and Redshift Spectrum queries Amazon S3 files through external schemas without loading them. Materialized views store precomputed results, can refresh automatically with AUTO REFRESH, and automatic query rewriting uses them in queries that do not reference them.

Understand data cataloging systems

  • The AWS Glue Data Catalog is a metadata repository shared by Athena, Amazon EMR, Redshift Spectrum and AWS Glue jobs. Redshift uses it through CREATE EXTERNAL SCHEMA FROM DATA CATALOG and GRANT USAGE on the schema; EMR configures it as its Hive metastore; and Glue reads tables with create_dynamic_frame_from_catalog and a push_down_predicate to filter partitions.
  • The AWS Glue Data Catalog is a persistent, Hive-compatible metastore that Amazon EMR can use instead of a local metastore, even in another account through hive.metastore.glue.catalogid. Tables should point to Amazon S3 rather than transient HDFS. Athena can read an existing Hive metastore with its external Hive metastore connector, without migrating the metadata.
  • An AWS Glue crawler classifies data (custom classifiers such as grok first, then built-in ones), groups files into tables or partitions and writes the metadata to the Data Catalog. CombineCompatibleSchemas creates a single table per S3 path when the data is compatible, and UpdateBehavior LOG or MergeNewColumns limit schema changes made by the crawler.
  • New partitions must be registered in the catalog before they can be queried: MSCK REPAIR TABLE adds Hive-style partitions but never removes them (ALTER TABLE DROP PARTITION does), incremental crawls only add new folders, and Glue jobs can register them with enableUpdateCatalog. Athena partition projection avoids registering partitions at all.
  • An AWS Glue connection is a Data Catalog object that stores the URL, credentials (preferably in AWS Secrets Manager) and VPC details, and crawlers and jobs reuse it. To reach a JDBC store in a VPC, Glue creates network interfaces that need a security group with a self-referencing rule and DNS enabled; the include path follows database/schema/table.

Manage the lifecycle of data

  • COPY loads data from Amazon S3 into Redshift in parallel when there are several files of similar size or splittable files (uncompressed CSV, Parquet, ORC); use one COPY per table and a manifest to list specific files. UNLOAD exports query results to S3 as text, JSON or Parquet, with Hive-style PARTITION BY, and auto-copy (COPY JOB) loads new files automatically.
  • S3 Lifecycle rules move objects between storage classes in a waterfall model (Standard, IA, Intelligent-Tiering, Glacier Instant Retrieval, Flexible Retrieval and Deep Archive) and can filter by prefix, tags or size. Intelligent-Tiering suits unknown access patterns with no retrieval fees; Glacier Instant Retrieval gives millisecond access, while Flexible Retrieval and Deep Archive need a restore.
  • The S3 Lifecycle Expiration action deletes objects when they reach a given age; with versioning it only adds a delete marker, and NoncurrentVersionExpiration is needed to remove old versions. ExpiredObjectDeleteMarker removes markers with no versions left, and AbortIncompleteMultipartUpload cleans up incomplete multipart uploads. A bucket policy cannot block these actions.
  • S3 Versioning keeps every version of each object: a simple DELETE only adds a delete marker, a bucket cannot return to unversioned (only be suspended) and MFA delete protects version deletions. DynamoDB TTL deletes, in the background and without consuming writes, items whose Number attribute in epoch seconds has expired, and those deletions appear in Streams.
  • To delete data on time: DynamoDB Time to Live removes expired items without consuming write throughput; in versioned S3 buckets, Expiration only adds a delete marker, and NoncurrentVersionExpiration is needed to remove old versions. Object Lock prevents deletion: a legal hold lasts until it is removed.

Design data models and schema evolution

  • In Redshift, KEY distribution on the join column collocates the rows that are joined, and the SUPER type stores semi-structured data that is queried with PartiQL. In DynamoDB, a composite sort key (for example country#region#city) lets a single query read a hierarchy at any level with begins_with.
  • When the shape of data changes: AWS Glue Schema Registry validates versions with compatibility modes (BACKWARD, FORWARD, FULL, and so on); a Glue crawler with UpdateBehavior set to LOG leaves the existing schema alone; and in Athena, Parquet (read by name) and Avro support removing and reordering columns, unlike CSV and TSV.
  • DMS Schema Conversion is the managed, web-based way to convert schemas to Aurora, RDS or Redshift: it assesses what converts automatically, converts and applies code, and adds extension packs. AWS SCT, installed locally, covers more sources and also converts ETL processes. Data is then moved with AWS DMS.
  • SageMaker ML Lineage Tracking records lineage for machine learning workflows (jobs, datasets, models and endpoints) and is queried with QueryLineage. SageMaker Catalog data lineage is OpenLineage-compatible: it is captured from Glue and Redshift connections, from Spark jobs, or through the PostLineageEvent API.
  • In Athena, partition for the most common queries, use STRING partition keys and columnar formats such as Parquet, and use partition projection on highly partitioned tables to avoid GetPartitions calls. In Redshift, ENCODE AUTO manages compression unless an encoding is set on any column.

3. Data Operations and Support

22–22%

Automate data processing by using AWS services

  • Amazon MWAA runs Apache Airflow DAGs as a managed service with worker scaling. AWS Step Functions orchestrates services through integration patterns: Request Response, Run a Job (.sync), which waits for the job to finish, and .waitForTaskToken; a Distributed Map state processes large S3 datasets with child executions.
  • To troubleshoot Amazon MWAA, enable the Airflow log groups in CloudWatch Logs (scheduler, DAG processing, task, worker and webserver); long queues are solved by staggering tasks or raising the minimum workers. In Step Functions, Retry and Catch handle errors, but States.Runtime is not caught by States.ALL.
  • With the AWS SDKs, many operations are asynchronous or paginated: Athena StartQueryExecution returns an ID and the status is read with GetQueryExecution; boto3 paginators iterate over truncated results; and ExecutionParameters with ? placeholders prevents SQL injection in Athena.
  • Each service brings processing features: AWS Glue job bookmarks prevent reprocessing old data; Amazon EMR Serverless runs Spark or Hive without managing clusters and releases resources when jobs finish; and Redshift materialized views store precomputed results with automatic refresh and query rewriting.
  • To consume data APIs without managing infrastructure: the Redshift Data API runs SQL asynchronously over HTTP without drivers or connections, using Secrets Manager or temporary credentials; Amazon AppFlow brings SaaS data into S3 or Redshift without code, and its Custom Connector SDK covers private APIs.

Analyze data by using AWS services

  • Amazon Quick Sight, the BI feature of Amazon Quick, visualizes data from sources such as Athena or Redshift through direct query or by importing it into SPICE, its in-memory engine, which speeds up queries and lets data be reused at no extra cost. SPICE supports scheduled and incremental refreshes with a look-back window. AWS Glue DataBrew produces visual profiles with distributions, correlations and duplicates.
  • To verify and clean data, Athena lets you query with SQL and find the S3 file behind each row with "$path". SageMaker Data Wrangler, now integrated into SageMaker Canvas, produces a quality report with missing, invalid and outlier values and warnings such as target leakage, and it offers transforms that impute or drop missing values and clip or remove outliers.
  • In Amazon Redshift, a standard view runs its query on every reference; WITH NO SCHEMA BINDING creates a late-binding view, required for external tables, and a materialized view stores precomputed results that are updated with REFRESH MATERIALIZED VIEW. In Athena, CREATE VIEW creates Athena-only views and CREATE PROTECTED MULTI DIALECT VIEW creates Data Catalog views shared with Redshift.
  • Amazon Athena for Apache Spark runs Spark code interactively and serverlessly, scaling automatically up to the maximums you set. It needs a workgroup whose engine is Apache Spark, which cannot be changed later; opening a notebook starts a session with parameters such as maximum concurrent DPUs or idle timeout. The %%sql magic runs SQL and displays the result with show().
  • Serverless services (Redshift Serverless, EMR Serverless, MSK Serverless) provision and scale capacity automatically and charge for use, with no node types, resizing or manual pausing. Provisioned services give more control in exchange for managing capacity. There are nuances: EMR Serverless pre-initialized capacity is paid even while idle, and MSK Serverless requires IAM and does not support Kafka ACLs.

Maintain and monitor data pipelines

  • To extract audit logs, Athena queries CloudTrail files stored in S3 with SQL, whereas event history covers only 90 days of management events. CloudWatch Logs Insights queries log groups interactively, and an export task copies a specific time range to S3, even to another account or to a bucket with Object Lock.
  • For traceability, CloudTrail records management events and, when configured, data events such as S3 GetObject, which advanced event selectors can filter; integrity validation with signed digest files proves whether a log was altered. Redshift provides connection, user and user activity logs to CloudWatch or S3; the activity log requires enable_user_activity_logging.
  • Pipeline alerts are usually sent to Amazon SNS topics. EventBridge rules capture service events, such as Glue Job State Change in FAILED or TIMEOUT, or Glue Job Run Status when a run exceeds the delay notification threshold. CloudWatch alarms notify when they change state, and composite alarms go into ALARM only when their rule is met, which reduces noise.
  • In AWS Glue, memory metrics and the Spark UI show where the bottleneck is: a driver running out of memory while reading many small files is fixed with grouping (groupFiles), and Auto Scaling adapts workers to each stage. In Athena, partition projection avoids looking up thousands of partitions in the catalog and removes partition maintenance.
  • By default CloudTrail records only management events, visible in event history for a limited period; an ongoing record needs a trail, preferably multi-Region. Data events, such as S3 DeleteObject, must be enabled explicitly. CloudTrail Insights detects unusual API call or error rates, and log file integrity validation proves that log files were not altered.

Ensure data quality

  • AWS Glue Data Quality checks data in transit inside ETL jobs with the Evaluate Data Quality transform and DQDL rules such as IsComplete (non-null values) or IsUnique. It can stop the job without loading the target when the ruleset fails, or add row-level columns (DataQualityEvaluationResult) so that bad records are routed to a quarantine location.
  • In AWS Glue DataBrew, data quality rules are grouped in rulesets that compare metrics (missing values, duplicates, ranges) with expected values and are validated in profile jobs, which produce a validation report. A rule can apply to several columns through selectors by name or regular expression, and each validation emits an EventBridge event with its result.
  • AWS Glue DataBrew profile jobs help investigate data consistency: at dataset level they compute duplicate rows and the Pearson correlation between numeric columns, and at column level they count missing values (nulls and empty strings), distinct and unique values, and detect outliers with the Z-score. The configuration selects columns by name or regular expression.
  • Sampling makes large data cheaper to analyze: DataBrew projects work on a sample of the first, last, or random rows, and profile jobs use a fixed number of rows by default unless the full dataset is chosen. In Athena, TABLESAMPLE BERNOULLI selects each row independently, while SYSTEM keeps or skips whole logical segments.
  • Data skew concentrates work on a few partitions. In Spark on AWS Glue it shows up in the Spark UI and is mitigated with Adaptive Query Execution and high-cardinality join keys. In Kinesis Data Streams, a partition key with many values spreads writes across shards, and in Redshift a high-cardinality distribution key avoids overloaded slices.

4. Data Security and Governance

18–18%

Apply authentication mechanisms

  • Security groups are stateful firewalls: allowing inbound traffic is enough for the response to leave. Using another security group as the source grants access to all its instances even when their IP addresses change. AWS Glue JDBC connections need a self-referencing inbound rule for all TCP ports, and in Amazon EMR security groups are chosen when the cluster is created, although their rules can be edited later.
  • IAM user groups apply the same policy to many users, but they cannot be nested or used as a Principal in resource-based policies. Services such as Redshift or Glue reach other services by assuming roles with temporary credentials instead of keys. S3 gateway endpoints only serve resources inside the VPC; on-premises networks need an interface endpoint.
  • AWS Secrets Manager stores encrypted credentials and rotates them in two ways: managed rotation, offered by services such as RDS, Aurora, Redshift, and DocumentDB without a Lambda function, and rotation by a Lambda function for everything else. For databases there are two strategies: single user, the simplest, and alternating users, which always keeps a valid set of credentials and suits high-availability applications.
  • Services assume IAM roles through the role's trust policy: a Lambda execution role must trust lambda.amazonaws.com and Lambda assumes it on every invocation. A CloudFormation service role fixes stack permissions regardless of the user (passing it needs iam:PassRole). In the AWS CLI, a profile with role_arn and credential_source = Ec2InstanceMetadata assumes a role without long-term keys.
  • An S3 access point policy is effective only if the bucket also allows the access; the recommended approach is to delegate control to access points with s3:DataAccessPointAccount. VPC endpoint policies control which principals and resources are reachable through the endpoint (for gateway endpoints Principal must be * and aws:PrincipalArn narrows it), and a bucket policy can require aws:SourceVpce.

Apply authorization mechanisms

  • AWS managed policies cannot be modified, AWS updates them, and they may be broader than needed; when they do not fit, create a customer managed policy (reusable across principals) or an inline policy (one-to-one, deleted with the identity). Policy variables such as ${aws:username} let one policy serve many users and require Version 2012-10-17.
  • AWS Secrets Manager stores database credentials, API keys and tokens and rotates them automatically; Amazon RDS can manage the master password in Secrets Manager with rotation and no code. Parameter Store is for configuration (String, StringList, SecureString encrypted with AWS KMS) with no credential rotation, and it can reference secrets with the /aws/reference/secretsmanager/ prefix.
  • In Amazon Redshift, GRANT gives permissions to users, groups or roles; RBAC bundles permissions, including system permissions such as CREATE USER (grantable only to roles), and roles are granted to users. GRANT supports column-level permissions, and ALTER DEFAULT PRIVILEGES applies permissions to objects created in the future, because GRANT ON ALL TABLES only affects existing ones.
  • AWS Lake Formation adds GRANT/REVOKE-style permissions on Data Catalog databases, tables and columns and on registered S3 locations, and vends temporary credentials to Athena, Amazon EMR (with runtime roles) and Redshift Spectrum. Data filters provide column, row and cell-level security and are granted only with SELECT; hybrid access mode adopts Lake Formation without cutting existing IAM access.
  • RBAC grants permissions by job function and requires policy updates as resources are added; ABAC allows actions when the resource tag matches the principal tag (aws:ResourceTag versus aws:PrincipalTag) and scales without changes, including with SAML session tags. In Lake Formation, LF-TBAC grants LF-Tag expressions on Data Catalog resources; LF-Tags are not IAM tags.

Ensure data encryption and masking

  • Amazon Redshift dynamic data masking hides, obfuscates or pseudonymizes columns at query time per user or role without altering data. The AWS Glue Detect PII transform finds sensitive entities and redacts, partially redacts or hashes them during ETL. AWS Glue DataBrew offers substitution, shuffling, deterministic or probabilistic encryption and hashing; Amazon Macie discovers sensitive data in S3.
  • AWS KMS protects root keys that never leave its HSMs and applies envelope encryption: GenerateDataKey returns a plaintext data key to encrypt outside KMS and an encrypted copy that is stored with the data and decrypted with Decrypt. Automatic rotation changes key material without changing the key ID and decrypts old data transparently; S3 Bucket Keys reduce SSE-KMS requests to KMS.
  • To encrypt data shared across accounts, use a customer managed KMS key: objects encrypted with AWS managed keys such as aws/s3 cannot be shared across accounts. The key policy must allow the external account and an IAM policy in that account must delegate the permission to its roles. In cross-account requests, specify the full key ARN, because an alias is resolved in the requester's account.
  • For encryption in transit, Amazon S3 recommends a bucket policy that denies requests where aws:SecureTransport is false, and Amazon Redshift requires SSL from every client when require_SSL is set to true in the parameter group. To encrypt before transit, use client-side encryption such as the Amazon S3 Encryption Client, which encrypts locally and hands S3 objects that are already encrypted.

Prepare logs for audit

  • By default CloudTrail records management (control plane) events; data events, such as S3 GetObject calls, and Insights events must be enabled explicitly. A trail delivers events to an S3 bucket and can be multi-Region or an organization trail to cover every account. Log file integrity validation and SSE-KMS encryption with your own key strengthen the audit trail.
  • CloudWatch Logs centralizes application logs, which the CloudWatch agent collects from EC2 instances and on-premises servers. Events are stored in log groups that never expire by default, so you set a retention period per log group. Data protection policies mask PII and financial data at ingestion, and only principals with logs:Unmask can see the values in clear text.
  • CloudTrail Lake aggregates events into immutable event data stores, even across an entire organization and several Regions, and lets you query them with SQL across multiple fields, which event history cannot do. With federation, events are queried from Athena through the Glue Data Catalog and Lake Formation, and integration channels bring in activity from outside AWS through PutAuditEvents.
  • Athena queries CloudTrail logs directly in S3, and partition projection removes the need to run ALTER TABLE ADD PARTITION. CloudWatch Logs Insights interactively analyzes data already stored, with commands such as filter and stats. For near real-time dashboards, a subscription filter streams logs to OpenSearch Service, but only from log groups in the Standard log class.
  • Large log volumes call for combining services: EMR copies its logs to S3 when a log URI (--log-uri) is set, so they survive cluster termination. CloudWatch Logs subscription filters send events to Data Firehose for continuous delivery to S3, and VPC flow logs can be written in Parquet with Hive-compatible prefixes for analysis with EMR or Athena.

Understand data privacy and governance

  • In Amazon Redshift, the producer creates a datashare, adds objects with ALTER DATASHARE and grants USAGE to a namespace or an account. Across accounts there is a two-way handshake: the producer account authorizes and the consumer account associates. INCLUDENEW shares a schema's future tables, and with VIA DATA CATALOG, Lake Formation enforces column and row-level permissions.
  • Amazon Macie detects sensitive data in S3 with managed and custom data identifiers, through continual automated discovery or targeted discovery jobs, and publishes its findings to EventBridge to automate responses such as tagging tables with LF-Tags in Lake Formation. Inside pipelines, the AWS Glue Detect PII transform detects and masks PII before the data is written.
  • To stop copies or replicas outside allowed Regions, use preventive controls such as the Control Tower Region deny control or SCPs with aws:RequestedRegion. That key limits the endpoint called, not the cross-Region impact of a call (for example PutBucketReplication), so destination-aware controls are also needed, such as backup:CopyTargets for AWS Backup copies.
  • AWS Config records every resource change as a configuration item, building a history you can view in the resource timeline or with get-resource-config-history. The configuration stream notifies changes through SNS, advanced queries filter current state, and an aggregator collects data from many accounts and Regions in a read-only view.
  • Data sovereignty combines where data is stored with who controls the keys. AWS Outposts brings AWS services into the customer's data center and Local Zones help with local residency; Control Tower data residency controls block, per OU, paths such as cross-Region peering. A KMS external key store keeps the key material outside AWS.
Practice this certification