Your data science team needs to perform interactive SQL queries on large datasets stored in Apache Parquet format within a Cloud Storage bucket. The team is familiar with Apache Hive and wants to leverage existing HiveQL queries. You need to provide an environment for the team to run their interactive HiveQL queries directly against the data in Cloud Storage. You want to keep operational overhead to a minimum. What should you do?
Correct answer:
D
The core requirement is to run interactive HiveQL queries (not just standard SQL) while keeping operational overhead low.
Dataproc: Dataproc is the managed service for running Apache Hadoop, Spark, and Hive on Google Cloud. It allows the team to use the exact HiveQL syntax and Hive tools they are familiar with. Because it is a managed service, it significantly reduces the overhead compared to manual installation on GCE.
Cloud Storage Integration: Dataproc is designed to use Cloud Storage as the underlying 'file system' (using the gs:// connector), allowing Hive to query Parquet files directly without moving them.
Correcting other options:
A: Loading data into BigQuery is an ingestion process that adds overhead and moves the data, contradicting the 'run directly against data in Cloud Storage' and 'minimum overhead' goals.
B: Manual installation on Compute Engine has the highest operational overhead.
C: While BigQuery external tables can query Parquet on GCS, BigQuery uses GoogleSQL syntax, not HiveQL. The requirement explicitly mentions the team's familiarity and existing queries in HiveQL.
'Dataproc is a fully managed and highly scalable service for running Apache Spark, Apache Flink, Presto, and Apache Hadoop/Hive... It allows you to use your existing open-source tools and workflows on Google Cloud without the management burden of a manual cluster.' (Source: Dataproc Overview)
'You can use Dataproc to run Hive queries... By default, Dataproc uses the Cloud Storage connector to allow Hive and other components to access data directly in Cloud Storage.' (Source: Using Hive on Dataproc)