Databricks-Certified-Professional-Data-Engineer Exam Dumps and Practice Questions

Lakeflow Declarative Pipelines provide the expect_all method for programmatically applying multiple data quality expectations at once. The documentation explains that @dlt.expect_all accepts a dictionary of expectation names mapped to SQL constraints, allowing rules to be dynamically loaded from metadata such as JSON files. This ensures that pipelines remain maintainable and scalable without needing to hardcode individual @dlt.expect decorators. The event logs will track each expectation's pass and fail counts individually, making it auditable. Other options are incorrect: invoking an external API introduces unnecessary complexity, individual decorators require hardcoding, and SQL constraints cannot dynamically reference external JSON.

Scheduling a job to execute the data processing pipeline once an hour on a new job cluster is the most cost-effective solution given the scenario. Job clusters are ephemeral in nature; they are spun up just before the job execution and terminated upon completion, which means you only incur costs for the time the cluster is active. Since the total processing time is only 10 minutes, a new job cluster created for each hourly execution minimizes the running time and thus the cost, while also fulfilling the requirement for hourly data updates for the business reporting team's dashboards.

:

Databricks documentation on jobs and job clusters: https://docs.databricks.com/jobs.html

This is the correct answer because it describes a limitation of Databricks Secrets. Databricks Secrets is a module that provides tools to store sensitive credentials and avoid accidentally displaying them in plain text. Databricks Secrets allows creating secret scopes, which are collections of secrets that can be accessed by users or groups. Databricks Secrets also allows creating and managing secrets using the Databricks CLI or the Databricks REST API. However, a limitation of Databricks Secrets is that the Databricks REST API can be used to list secrets in plain text if the personal access token has proper credentials. Therefore, users should still be careful with which credentials are stored in Databricks Secrets and which users have access to using these secrets. Verified Reference: [Databricks Certified Data Engineer Professional], under ''Databricks Workspace'' section; Databricks Documentation, under ''List secrets'' section.

In Delta Lake, the property delta.deletedFileRetentionDuration controls how long deleted data files are retained before being permanently removed during a VACUUM operation.

By default, this retention duration is set to 7 days.

To comply with stricter retention requirements, organizations can explicitly update the table property using an ALTER TABLE statement.

Option A uses the correct SQL command:

ALTER TABLE my_table SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 15 days')

This updates the Delta table metadata so that all future operations respect the 15-day retention policy for deleted files.

Why not the others?

B: This code incorrectly tries to set the property via the DeltaTable API. Delta's Python API does not expose direct attributes like deletedFileRetentionDuration; instead, properties must be set through ALTER TABLE or DataFrameWriter options.

C: VACUUM ... RETAIN specifies a one-time file cleanup action (e.g., retaining 15 hours of history), not a persistent retention policy. It cannot be used to set a continuous retention duration.

D: Setting spark.conf applies a session-level configuration and does not permanently update the table's retention metadata. Once the session ends, this configuration is lost.

Therefore, Option A is the correct and documented approach for persistently enforcing a 15-day deleted file retention period in Delta Lake.

When triggering a job run using the Databricks CLI, the run_id field in the response represents a globally unique identifier for that particular run of the job. This run_id is distinct from the job_id. While the job_id identifies the job definition and is constant across all runs of that job, the run_id is unique to each execution and is used to track and query the status of that specific job run within the Databricks environment. This distinction allows users to manage and reference individual executions of a job directly.