Domain 1: Developing Code for Data Processing with Python and SQL
- A DataFrame is a plan rather than a result, so every action re-evaluates the whole lineage: two writes from one source read it twice unless the shared intermediate is persisted.
- Row-wise Python UDFs serialise each row out to a Python process and block Photon for that part of the plan; built-in functions or vectorized Pandas UDFs recover an order of magnitude.
- Window functions with a partition and an ordering are how you pick the latest row per key, which is the standard way to reduce a change feed to one row per key before a merge.
- MERGE fails deliberately when two source rows match one target row, because choosing between them is a business rule the engine will not invent for you.
- Nulls never compare equal, so an inner join silently discards rows with a null key and reports no error; decide what a null key means rather than letting the join decide.
- Structured Streaming and batch share the DataFrame API, so the same transformation function can serve both, which is what makes foreachBatch practical.
- Transformations belong in importable modules that take and return DataFrames, with reads and writes at the pipeline's edges, or the logic cannot be unit tested.
- Calling for the current timestamp inside a transformation makes the run irreproducible and the tests non-deterministic; pass the logical run time as a parameter instead.
Domain 2: Data Ingestion and Acquisition
- Auto Loader tracks which files it has already seen, so re-running it does not reprocess a directory; that state lives in the checkpoint, not in the target table.
- Directory listing costs grow with the number of files present rather than the number that are new, so a bucket accumulating millions of objects needs file notification mode.
- Notification mode depends on cloud queue and subscription resources, so its distinctive failure is arriving files that are never discovered at all.
- Schema inference plus a rescued data column keeps records whose fields do not match, which makes that column a detector for unannounced source changes.
- Per-trigger limits such as maxFilesPerTrigger convert a backlog into a series of bounded batches instead of one enormous batch that spills or fails.
- At-least-once delivery means the same record can arrive twice, so deduplication needs a key the source assigns consistently across redeliveries.
- An unpartitioned JDBC read opens one connection and one task, so a 24-worker cluster sits idle unless partitioning bounds and a column are supplied.
- Bronze is normally append-only, because the value of keeping what the source actually sent is being able to reprocess when parsing logic is corrected.
Domain 3: Data Transformation, Cleansing, and Quality
- Silver is where types, keys and naming are conformed and duplicates removed, so every downstream consumer starts from the same trustworthy detail.
- Normalising join keys once in silver beats trimming and lower-casing in every consumer, and a quality check on the normalised form stops it drifting back.
- Expectations can warn, drop or fail; choosing to warn is legitimate only if the breach is surfaced to the consumers who relied on the commitment.
- A quarantine table holds the same personal data as its source, so it inherits the same access controls, retention and erasure obligations.
- NOT NULL and CHECK constraints are enforced on write, while primary and foreign key declarations are informational and police nothing.
- Change Data Feed lets a consumer read what changed between two versions, including deletes, which a timestamp filter on the source cannot reconstruct.
- Schema enforcement failing on a new source field is the control working; the response is a deliberate evolution, not permanently enabling schema overwrite.
- A count that matches while a monetary total does not points at representation: floating point in place of decimal, or rounding applied at a different stage.
Domain 4: Data Sharing and Federation
- Delta Sharing grants access to the provider's files rather than copying data, which is why adding a recipient is instant and egress is the provider's cost.
- Sharing a view rather than a base table is how a provider narrows exposure to particular rows and columns, since the recipient receives only the view's output.
- History and change data are shared only if the provider opts in, because historical versions can contain rows that were later corrected or deleted.
- The open protocol issues a credential file, so a recipient needs no Databricks account and any compatible client can read the share.
- Rotating a recipient's credential invalidates a leaked one without disturbing the share or its other recipients.
- Lakehouse Federation governs external tables in the catalog without copying them, and query performance then depends on what the connector pushes down.
- A federated connection authenticates with a stored credential, so every query through it inherits that identity's access on the remote system.
- A clean room runs agreed computations and returns results, which is what distinguishes it from sharing when neither party may see the other's rows.
Domain 5: Monitoring and Alerting
- A job's exit status reports whether the code ran, not whether it accomplished anything, so a run that wrote zero rows succeeds unless volume is asserted.
- Failure notifications miss the run that never started, so pair them with an expectation that the output was refreshed within its window.
- Freshness is the signal consumers actually need, because it distinguishes a quiet day from a pipeline that stopped three days ago.
- A relative check against the recent norm behaves better than a fixed threshold, since legitimate volume varies by day of week and season.
- The pipeline event log holds per-expectation pass and fail counts, which is what turns quality from a run-time behaviour into a trended metric.
- Streaming progress metrics expose input rows per batch, batch duration and state size, so a query falling behind is visible before it fails.
- Alerting on every run trains people to ignore alerts; notifications should be exceptional or they stop being read.
- Billing system tables joined to compute tags are the authoritative source for what a workload cost, which estimates from runtime cannot match.
Domain 6: Cost and Performance Optimization
- Cost is capacity multiplied by time, so a job on three times the workers for a third of the duration costs about the same: that is latency bought, not a saving.
- An all-purpose cluster running all day for a nightly job spends nearly all its billed hours idle, which dwarfs any efficiency gain within the run.
- Photon is billed at a higher rate, so it only saves money where it actually accelerates the work; a plan dominated by Python UDFs pays the premium for nothing.
- File skipping compares a predicate to per-file statistics, so wrapping the column in a function or a cast silently disables it and the whole table is read.
- Partitioning suits low-cardinality columns; on a high-cardinality identifier it creates a directory per value and shatters the table into tiny files.
- Liquid clustering separates the choice of keys from the physical layout, so keys can change as query patterns do, and clustering is incremental after the first run.
- Spill means one task's working set exceeded its memory, so raise the shuffle partition count rather than the cluster size.
- Statistics are collected for a bounded set of leading columns, so a heavily filtered column at position 180 of a wide table contributes nothing to skipping.
Domain 7: Ensuring Data Security and Compliance
- A DELETE removes rows from the current version and leaves the files in place, so genuine erasure needs the retention window to elapse and VACUUM to run.
- Erasure does not propagate along lineage: silver, gold, aggregates and anything exported hold their own copies and each needs its own deletion.
- A stateful stream keys its state on whatever it deduplicates or aggregates by, so a personal identifier can sit in state and checkpoint outside any governed table.
- Column masks and row filters are enforced by the engine on the table, so every client is subject to them and no tool can opt out.
- Unity Catalog privileges are hierarchical and inherited, so a broad SELECT at catalog level silently covers every table added afterwards.
- A storage credential wraps the cloud identity and an external location binds it to a path, which replaces mounts that gave everyone on a cluster the same access.
- Audit records are the only source that says who read a table; the transaction log records writes and lineage records derivation.
- A deterministic hash of an email is pseudonymised rather than anonymous, since the same input always maps to the same value and the input space is guessable.
Domain 8: Data Governance
- The three-level name of catalog, schema and table is the identifier, and a shared metastore is what makes one qualified name mean the same table everywhere.
- A metastore is regional, so an organisation in two regions has two, joined by sharing rather than merged into one.
- Column-level lineage answers the impact question before a rename, which is what turns a breaking change into a planned migration.
- A data contract commits to structure with a breaking-change policy, and to freshness; implementation details such as compute configuration must stay free to change.
- Enabling a newer Delta table feature raises the protocol version and is hard to reverse, so the set of clients reading the table has to be known first.
- A view executes with its owner's privileges, so a consumer granted the view needs no grant on the tables behind it.
- Own production objects with a group rather than an individual, or a departure leaves an object only an administrator can administer.
- A number without its unit is not usable data: currency, timezone and measurement units are the classic omissions, and they belong in columns rather than comments.
Domain 9: Debugging and Deploying
- A checkpoint belongs to one query, so two queries sharing a location overwrite each other's progress and produce duplicated or skipped data.
- Changing a query's stateful structure invalidates its checkpoint, which is why the engine refuses to resume rather than resuming incorrectly.
- Rerun safety needs two properties: a scope fixed by a parameter rather than the clock, and a write that replaces its own slice rather than appending.
- A repair run reuses successful tasks and reruns only what failed and what depended on it, which requires tasks to write durable output.
- Code that works interactively and fails as a job is usually a permission difference, since interactive runs use the author's broader privileges.
- Asset bundles hold one definition with per-target variables, so the artefact tested in staging is the one that reaches production.
- A shallow clone gives a writable full-size copy referencing the source's files, which is the tool for testing a migration without duplicating storage.
- Measure a performance fix by bytes read and shuffled rather than duration, since duration also moves with capacity, caching and contention.
Domain 10: Data Modeling
- Type 2 gives each version of a member its own row bounded by effective dates, which is what lets a fact report against the attribute in force at its event date.
- A surrogate key identifies a version of a member; a natural key identifies only the member, which is why Type 2 history needs surrogates on facts.
- Grain is the promise a fact table makes, so a measure at a coarser level repeats across rows and any sum double counts it.
- A balance is semi-additive: it aggregates across accounts at one date and not across time, which a periodic snapshot fact models naturally.
- Late-arriving dimension members get an inferred placeholder row so the fact can load now and be enriched later, rather than being rejected or given a null key.
- Never join two fact tables on shared dimension keys: the join fans out and silently multiplies every measure. Aggregate to a common grain or drill across.
- A bridge table resolving a many-to-many relationship needs an allocation factor, or revenue is counted once per category and the total will not reconcile.
- A mini-dimension isolates rapidly changing attributes so the main dimension versions only when its stable attributes change.
Databricks Data Engineer Professional exam tips
- Almost every question is a scenario with several workable answers, so read for the constraint: a stated tolerance for longer runtime, a requirement that output must not change, or a rule that consumers cannot be disrupted is usually what eliminates three options.
- When a question asks about cost, check whether the proposed change reduces the work or merely redistributes it. A larger cluster finishing sooner is almost always the wrong answer, because cost is capacity multiplied by time.
- Know the difference between what is enforced and what is informational: NOT NULL and CHECK constraints reject writes, while primary and foreign key declarations only document intent. Several questions turn on exactly this.
- Be precise about where data persists beyond a table. Time travel keeps deleted rows until VACUUM, stateful streams keep keys in the checkpoint, and query history keeps whatever literal you typed into a predicate.
- For streaming questions, decide first whether the query is stateful. Watermarks, state size, checkpoint compatibility and deduplication windows all follow from that, and the stateless cases are far simpler than candidates expect.
- Rerun safety comes up repeatedly. A pipeline is safe to rerun when its scope comes from a parameter rather than the clock and its write replaces its own slice, through a predicate-scoped overwrite, a merge, or a transactional identifier.
Study guide FAQ
What format and passing requirements does the Databricks Certified Data Engineer Professional exam have?
The exam is multiple-choice, delivered online and proctored, with 59 scored questions to complete in 120 minutes. Databricks does not publish a fixed passing percentage, and the certification is valid for two years. There is no hands-on coding lab.
How is the Professional exam different from the Data Engineer Associate?
The Associate exam checks that you know the platform and can build a working pipeline. The Professional exam assumes that and asks about production judgement: what a change costs, whether a rerun is safe, where personal data persists after a delete, and which modelling choice makes the numbers reconcile. Almost every question is a scenario rather than a recall item.
Which domains carry the most weight?
Developing code for data processing with Python and SQL is by far the largest at 22 percent, followed by cost and performance optimization at 13 percent. Transformation and quality, monitoring and alerting, security and compliance, and debugging and deploying each carry 10 percent, so the remaining half of the exam is spread fairly evenly.
How much Spark internals knowledge does the exam expect?
Enough to reason about a query profile. You should be able to read task-level metrics to spot skew, recognise spill and know that it means one task exceeded its memory, understand why shuffles are expensive, and explain what adaptive query execution changes at runtime. It does not ask you to recite configuration parameter names.
Do I need to know Unity Catalog in detail?
Yes. Security and compliance plus data governance together are 17 percent of the exam, and Unity Catalog concepts appear in the debugging and modelling questions too. Know the three-level namespace, how privileges inherit, the difference between storage credentials and external locations, how column masks and row filters are enforced, and what managed versus external tables mean for a drop.
Related Data resources
- Databricks Data Engineer Professional practice exam
- Data practice exams
- Certification path
- CompTIA Data+ (DA0-002) study guide
- Confluent Certified Developer for Apache Kafka (CCDAK) study guide
- Databricks Certified Data Analyst Associate study guide
- Certification exam guides & tips
- Pricing & plans
- FAQ