Domain 1: Databricks Lakehouse Platform
- The lakehouse architecture combines the low-cost, scalable storage of a data lake with the ACID transactions, schema enforcement, and governance of a data warehouse, letting BI, data science, ML, and streaming workloads query the same tables.
- Delta Lake is the open-source storage layer that brings ACID transactions and reliable metadata handling to files on cloud object storage, and it is the technology that makes the lakehouse architecture possible.
- A plain data lake with no transactional layer can leave data in an inconsistent state after concurrent writes or failed jobs, since there is no log tracking which changes actually committed.
- Databricks separates the control plane (Databricks-managed services such as the workspace UI, notebooks, job scheduler, and cluster manager) from the data plane (compute clusters and storage, which run inside the customer's own cloud account by default).
- The customer's actual table data and cluster compute live in the data plane inside their cloud account, not in the Databricks-managed control plane, which keeps data processing inside the customer's own security boundary.
- A Databricks workspace is the environment where users access notebooks, clusters, jobs, and Databricks SQL; DBFS is a distributed file system abstraction mounted into the workspace and backed by cloud object storage.
- Clusters run on the Databricks Runtime (DBR), which bundles Apache Spark with performance and reliability optimizations such as Photon; an all-purpose cluster is shared for interactive work, while a job cluster is created for a single job run and terminates when it finishes.
- Databricks SQL provides SQL warehouses (serverless, pro, or classic) for running queries, dashboards, and alerts against lakehouse tables using familiar SQL-based tools.
- Databricks Repos integrates Git version control directly into the workspace so notebooks and code can be branched, committed, and reviewed as part of a normal software development workflow.
- The medallion architecture organizes data into bronze (raw ingested data), silver (cleaned and conformed data), and gold (aggregated, business-level data) layers as it moves through a pipeline.
- Notebooks support mixed languages: the %sql (or %python, %scala) magic command switches a single cell's language regardless of the notebook's default, and %run executes another notebook while sharing its variables and functions.
Domain 2: ELT with Spark SQL and Python
- Delta Lake is the default table format in Databricks; a CREATE TABLE statement with no USING clause creates a managed Delta table with a transaction log and ACID guarantees.
- In a managed table, Databricks manages both the metadata and the underlying data files, so DROP TABLE deletes the data; in an external (unmanaged) table, defined with a LOCATION, DROP TABLE removes only the metastore entry and leaves the files in place.
- CREATE TABLE AS SELECT (CTAS) creates and populates a new table in one statement, inferring its schema from the SELECT query, which makes it well suited to materializing the result of a join or filter as a new table.
- CREATE OR REPLACE TABLE fully redefines a table's schema and data while still recording a new Delta version, whereas TRUNCATE only removes rows and leaves the schema and table history intact.
- Constraints such as NOT NULL and CHECK are enforced at write time in Delta Lake, so any write that violates them is rejected immediately rather than silently corrupting the table.
- A temporary view's lifetime is scoped to the creating session and disappears when the session ends; a global temporary view is registered in the global_temp database and is visible across sessions on the same cluster; a persisted view is a durable metastore object with no data of its own, only a stored query.
- Files can be queried directly without first creating a table, for example SELECT * FROM json.`/path/to/data`, and COPY INTO provides an idempotent way to incrementally and repeatably load new files into a Delta table.
- Spark SQL provides functions for nested data: dot notation and colon syntax access fields inside structs, explode() turns array elements into separate rows, and higher-order functions like TRANSFORM, FILTER, and EXISTS operate on array elements without a UDF.
- A user-defined function (UDF), whether written in SQL or Python, lets you apply custom logic per row, but built-in Spark functions are generally preferred because UDFs are not optimized by Catalyst and run slower.
- PySpark DataFrame transformations such as .select(), .filter(), .withColumn(), and .groupBy() are lazily evaluated and can be freely mixed with spark.sql() calls, using a temporary view as the bridge between SQL and Python code in the same notebook.
- Deduplication can be performed with DISTINCT or dropDuplicates(), and PIVOT reshapes distinct row values in one column into multiple output columns.
Domain 3: Incremental Data Processing
- Structured Streaming models a data stream as an unbounded input table to which new rows are continually appended, and the same DataFrame operations used in batch code can be reused against it.
- spark.readStream returns a DataStreamReader that defines a streaming source, while writeStream on a DataFrame returns a DataStreamWriter that configures the sink, output mode, and trigger before .start() launches the query.
- Checkpointing (set with the checkpointLocation option) records a streaming query's progress and state, allowing it to resume correctly after a failure or restart without reprocessing or losing data.
- Append output mode writes only new rows since the last trigger, complete mode rewrites the entire result table to the sink every trigger, and update mode writes only the rows that changed since the last trigger.
- The default trigger runs micro-batches back-to-back as fast as possible; a processingTime trigger targets a fixed interval between batches; Trigger.AvailableNow() processes all currently available data across one or more micro-batches and then stops, making it well suited to a scheduled job.
- Auto Loader, invoked with the cloudFiles source format, incrementally and efficiently ingests new files as they land in cloud storage, scaling to millions of files and supporting schema inference and schema evolution.
- Auto Loader tracks schema changes in a schemaLocation and can capture unexpected columns in a rescued data column rather than failing the pipeline outright.
- Auto Loader can discover new files either by directory listing or by cloud-native file notification services, with file notification scaling better for very large or high-volume directories.
- MERGE INTO performs an upsert, updating matched rows and inserting unmatched rows, which is the standard way to apply change data capture (CDC) records into a Delta table.
- Watermarking tells Structured Streaming how late data is allowed to arrive before a windowed aggregation is finalized, and combining a watermark with dropDuplicates() bounds the state needed for streaming deduplication.
- foreachBatch exposes each micro-batch's output as a batch DataFrame, letting you apply batch-only operations, such as writing to multiple sinks or performing a MERGE, inside a streaming query.
Domain 4: Production Pipelines
- Delta Live Tables (DLT) uses a declarative model: you write SQL or Python describing the target datasets, and DLT resolves dependencies, builds a DAG, and manages the compute and orchestration needed to keep them updated.
- In SQL, a DLT dataset is declared with CREATE LIVE TABLE (or LIVE VIEW), and referencing another dataset with the LIVE. prefix registers a dependency that DLT uses to determine execution order.
- In Python, the @dlt.table decorator declares a materialized DLT table and @dlt.view declares a view that is not persisted to storage; dlt.read() performs a full batch read of an upstream dataset while dlt.read_stream() reads it incrementally.
- A STREAMING LIVE TABLE (SQL) or a table built from dlt.read_stream() (Python) is processed incrementally, so pairing it with Auto Loader's cloudFiles source is the standard pattern for bronze-layer ingestion in a DLT pipeline.
- Expectations (CONSTRAINT ... EXPECT) define data quality rules on a DLT dataset, with an ON VIOLATION action of retaining the row anyway, dropping the violating row, or failing the entire pipeline update.
- APPLY CHANGES INTO lets a DLT pipeline apply CDC records to a target table, supporting both SCD Type 1 (overwrite with the latest value) and SCD Type 2 (retain full history of changes) semantics.
- A DLT pipeline can run in triggered mode, processing available data once and then stopping, or in continuous mode, running constantly to process new data with lower latency.
- Databricks Workflows (Jobs) orchestrate one or more tasks, such as notebooks, DLT pipelines, SQL queries, or dbt tasks, into a DAG with dependencies, a cron schedule, and a choice between an all-purpose or job cluster.
- A failed Workflows job can be configured with automatic retries, and the Repair Run feature reruns only the failed and downstream tasks rather than the entire job from the start.
- Databricks SQL alerts monitor a query result against a threshold and can trigger email or webhook notifications, complementing dashboards for ongoing production monitoring.
- Development mode in DLT reuses a cluster across pipeline updates for faster iteration, while production mode terminates the cluster after each update completes to reduce cost.
Domain 5: Data Governance
- Unity Catalog is Databricks' centralized governance layer for data and AI assets, providing one place to manage permissions, auditing, lineage, and discovery consistently across every workspace attached to it.
- A Unity Catalog metastore is the top-level container for metadata and must be created and assigned to a workspace before catalogs, schemas, or tables can be created in that workspace.
- Unity Catalog uses a three-level namespace, catalog.schema.table, where the catalog is the outermost grouping (often by business unit or environment), the schema groups related tables and views (equivalent to a database), and the table is the innermost object.
- Volumes are Unity Catalog securable objects that govern access to non-tabular files in cloud storage, and registered functions and models are likewise governed with GRANT and REVOKE like tables.
- GRANT SELECT allows reading rows from a table, GRANT MODIFY allows INSERT, UPDATE, DELETE, and MERGE, and CREATE TABLE must be granted on a schema before a new table can be created inside it.
- Querying a table requires USE CATALOG and USE SCHEMA privileges on its parent containers in addition to SELECT on the table itself.
- Privileges granted at the catalog or schema level are inherited by everything beneath them, so granting USE SCHEMA plus SELECT at the schema level gives read access to every table it contains.
- Revoking a user's direct grant on an object does not remove access gained through another path, such as a grant made to a group the user belongs to.
- Dynamic views use functions such as is_member() and current_user() inside a CASE or WHERE clause to enforce row-level or column-level security within a single shared view definition.
- Delta Sharing is an open protocol for securely sharing live data across organizations and platforms without copying it or requiring the recipient to run Databricks.
- Catalog Explorer is the UI for browsing catalogs, schemas, tables, and their permissions, lineage, and metadata within Unity Catalog.
Databricks Certified Data Engineer Associate exam tips
- Know the managed vs external table DROP behavior cold: dropping a managed table deletes its underlying data files, while dropping an external table only removes the metastore entry and leaves the LOCATION files untouched.
- Distinguish OPTIMIZE (compacts small files via bin-packing) from ZORDER BY (co-locates related data within files to speed up filtering on specific columns) - they solve different problems and are often tested together.
- Compare Auto Loader (cloudFiles, incremental, schema inference/evolution, scales to millions of files) with COPY INTO (simpler, idempotent, best for smaller or less frequent batches of new files) and know when each is the right choice.
- Memorize the three ON VIOLATION behaviors for a DLT expectation: the default keeps and logs violating rows, DROP ROW discards them, and FAIL UPDATE stops the entire pipeline run on a violation.
- Unity Catalog always requires the full chain of privileges: USE CATALOG and USE SCHEMA on the parents plus SELECT (or MODIFY) on the table itself - missing any one of the three causes the query to fail even if SELECT is granted.
Study guide FAQ
What format and passing requirements does the Databricks Certified Data Engineer Associate exam have?
The exam is multiple-choice, delivered online and proctored, with about 45 questions to complete in 90 minutes. It uses a scaled score and the passing bar is roughly 70%, with no hands-on coding lab.
Are there prerequisites for the Data Engineer Associate exam?
Databricks recommends around six months of hands-on experience performing data engineering tasks on Databricks, but there is no enforced prerequisite course or certification. Familiarity with Spark SQL, basic Python, and the Databricks workspace is expected.
How is the Associate exam different from the Data Engineer Professional exam?
The Associate exam is conceptual, checking foundational knowledge of the lakehouse, ELT, incremental processing, Delta Live Tables, and Unity Catalog. The Professional exam goes deeper into performance tuning, complex streaming and DLT pipeline design, testing and deployment, and security, and assumes real production experience.
Which domain should I study hardest?
ELT with Spark SQL and Python is the largest domain at 29% of the exam, covering Delta Lake DDL, CTAS, views, and working with nested or complex data in both SQL and PySpark. Databricks Lakehouse Platform (24%) and Incremental Data Processing (22%) are the next largest, so together these three domains make up about three-quarters of the exam.
Do I need to write a lot of code for the exam?
No, the exam is entirely multiple-choice with no coding lab. Expect to read short Spark SQL, PySpark, or DLT code snippets and choose the correct output, missing piece, or best approach rather than write code from scratch.