The official Google Cloud Certified Professional Data Engineer syllabus, organized by domain. Check off each point as you review it.
Inicia sesión para guardar tu progreso entre sesiones.
Designing for security and compliance
Grant predefined roles on the lowest possible resource: BigQuery Data Viewer on the dataset and Job User on the project, and use dedicated service accounts for Dataflow workers. IAM Conditions limit access in time, and organization policies, inherited by every descendant, enforce guardrails such as domain-restricted sharing or public access prevention.
BigQuery IAM roles and permissions ↗Organization Policy overview ↗
Google encrypts data at rest by default; with CMEK you control the key encryption keys in Cloud KMS, in the same location as the dataset, and with Cloud EKM the key material stays in an external key manager. Autokey creates keys on demand, gcp.restrictNonCmekServices enforces CMEK, and rotating a key does not re-encrypt existing BigQuery tables.
Customer-managed Cloud KMS keys ↗Cloud External Key Manager ↗
Sensitive Data Protection discovery continuously locates PII and de-identification transforms it: AES-SIV and format-preserving encryption are reversible with the key, HMAC-SHA-256 is not, and date shifting keeps intervals. In BigQuery, dynamic data masking with policy tags shows masked values to Masked Readers without changing their queries.
The gcp.resourceLocations constraint limits where new resources are created, and a BigQuery dataset location cannot change after creation. Pub/Sub message storage policies, the Dataflow job region and regional log buckets keep data in one region; BigQuery Omni queries data in S3 without moving it out of AWS.
Restrict resource locations ↗Configure message storage policies ↗
For regulatory retention, Bucket Lock and Object Retention Lock in Cloud Storage and locked log buckets prevent deletion or shorter retention. Data Access audit logs are disabled by default except in BigQuery; Access Transparency and Access Approval log and approve access by Google personnel, and Assured Workloads applies control packages such as HIPAA.
Datasets are the access boundary: tables inherit their permissions, so group tables by audience. Patterns such as a central data lake with department data marts separate storage, billing and jobs; authorized datasets, row-level security, IAM tags and VPC Service Controls perimeters complete the governance design.
Introduction to the resource hierarchy ↗Authorized datasets ↗
Separate development, non-production and production into projects grouped in environment folders, with inherited policies and their own service accounts. Promote one artifact (Flex Templates in Artifact Registry, Dataform release configurations, DAGs tested in a development Managed Airflow environment) and give development tokenized data or table clones instead of real sensitive data.
Organization structure ↗Design Dataflow pipeline workflows ↗
Designing for reliability and fidelity
To clean data already loaded into BigQuery, Dataform manages SQLX transformations with Git and a dependency tree, and BigQuery data preparation applies Gemini suggestions with deduplication and validation steps that route rows to an error table. Before loading, Cloud Data Fusion Wrangler cleans visually and Dataflow sends failed rows to a dead-letter table. Gemini in BigQuery generates SQL from natural language.
Managed Airflow orchestrates Python DAGs that span clouds and on-premises sites; Workflows is the serverless option with no idle cost for a few steps, and Dataform schedules its runs with workflow configurations. In Airflow, sensors in reschedule mode free the worker. Dataflow is monitored through its job interface and Cloud Monitoring alerts on the Failed and System lag metrics.
Schedule runs (Dataform) ↗Use Cloud Monitoring for Dataflow pipelines ↗
BigQuery managed disaster recovery offers soft failover, with no data loss and both regions up, and hard failover, which can lose unreplicated data; scheduled queries must be recreated after failover. Cloud SQL combines high availability across zones with cross-region replicas that you promote, dual-region Spanner gives zero RPO within one country, and dual-region buckets with turbo replication serve data without path changes.
Managed disaster recovery (BigQuery) ↗Data availability and durability (Cloud Storage) ↗
Spanner offers atomic read-write transactions across tables with external consistency, and lock-free read-only transactions. BigQuery supports multi-statement transactions across tables, but with a table-level lock and no queuing. Bigtable is atomic only per row; with several clusters it is eventually consistent, and read-your-writes requires routing to the same cluster.
Transactions overview (Spanner) ↗Multi-statement transactions (BigQuery) ↗
Dataform validates every run with assertions (nonNull, uniqueKey, rowConditions or manual SQLX) and tests logic with unit tests over mocked inputs. Knowledge Catalog auto data quality validates tables with no infrastructure using built-in and custom SQL rules, incremental scope, quality scores, email notifications, export to BigQuery and publication of results to the catalog.
Auto data quality overview (Knowledge Catalog) ↗Test data quality (Dataform) ↗
Designing for flexibility and portability
Choose services that absorb future requirements without a redesign: Datastream to move from nightly exports to near real-time CDC, Managed Service for Apache Kafka to keep Kafka clients, serverless Spark to pay only while running, Dataflow to reuse the same logic for batch and streaming, continuous queries to react in SQL, and new Pub/Sub subscriptions to add consumers.
BigQuery Omni queries S3 or Blob Storage data with BigLake tables where it lives and can write results back to that cloud. BigLake tables delegate access and enforce row and column security for Spark too; Iceberg managed tables keep one open copy in your bucket with DML and streaming. Beam on Dataflow can change runner without rewrites, and gcp.resourceLocations limits where resources are created.
Introduction to BigQuery Omni ↗Apache Iceberg managed tables ↗
Knowledge Catalog (formerly Dataplex Universal Catalog) discovers Cloud Storage files with discovery scans that publish BigLake tables when you supply a connection, profiles tables with full or incremental data profile scans, and organizes metadata with aspects and entry types. Its search mixes natural language and keywords and respects permissions; in a data mesh each domain has raw and curated zones.
Discover and catalog Cloud Storage data ↗About data profiling ↗
Designing data migrations
Migration planning starts with an inventory of workloads, dependencies, teams and requirements; for a data warehouse, the BigQuery migration assessment analyzes metadata and query logs and proposes the target state and plan. First movers should be non-critical, loosely coupled workloads, and each workload's strategy depends on how much downtime it can tolerate.
Migration assessment ↗Migrate to Google Cloud: Assess and discover your workloads ↗
Each tool has its case: BigQuery Data Transfer Service loads warehouses such as Teradata or Redshift into BigQuery, incrementally too; Database Migration Service moves databases to Cloud SQL and AlloyDB with continuous migrations; Datastream replicates CDC changes to BigQuery over private VPN or Interconnect connectivity; Transfer Appliance covers missing bandwidth, and the validation tool compares source and target.
Migrate to Google Cloud: Transfer your large datasets ↗Database Migration Service overview ↗