AWS Certified Data Engineer Associate · Free Practice Question Medium
Question 63
An Extract, Transform, and Load (ETL) job needs to be implemented which will process data uploaded to an Amazon S3 bucket daily. The uploaded data is in the form of .csv files, each of which is around 100 MB in size.
Which of the following represents the most cost-effective solution for the ETL job?
-
A
Configure an AWS Glue Python shell job. Use the pre-loaded Pandas library to run transformations on the data
-
B
Run the transformations on the data using AWS Glue Data Studio
-
C
Write an AWS Glue PySpark job. Use Apache Spark to transform the data
-
D
Run the transformations on the data using AWS Glue DataBrew
Reveal correct answer
Correct answer: A
Explanation
Correct option:
Configure an AWS Glue Python shell job. Use the pre-loaded Pandas library to run transformations on the data
AWS Glue ETL supports running plain non-distributed Python scripts as a shell script to run small to medium-sized generic tasks that are often part of an ETL workflow. For example, to submit SQL queries to services such as Amazon Redshift, Amazon Athena, or Amazon EMR, or run machine learning (ML) and scientific analyses.
You can run Python shell jobs using one Data Processing Unit (DPU) or 0.0625 DPU (which is 1/16 DPU), allowing you to run cost-effective small to medium jobs that do not require Spark runtime.
Compared to AWS Lambda, which has a strict 15-minute maximum timeout, AWS Glue Python Shell can be configured with a much longer timeout and higher memory, often required for data engineering jobs.
Python shell jobs in AWS Glue come pre-loaded with libraries such as Boto3, NumPy, SciPy, Pandas, and others.
Incorrect options:
Write an AWS Glue PySpark job. Use Apache Spark to transform the data - The AWS Glue PySpark job can certainly be used for the ETL job, however, it is a costlier solution than just using the AWS Glue Python shell job (with Pandas library) for the given use case.
Run the transformations on the data using AWS Glue DataBrew - AWS Glue DataBrew is a visual data preparation tool that makes it easier for data analysts and data scientists to clean and normalize data to prepare it for analytics and machine learning (ML). AWS Glue DataBrew is not an ETL solution.
Run the transformations on the data using AWS Glue Data Studio - AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor data integration jobs in AWS Glue. You can visually compose data transformation workflows and seamlessly run them on the Apache Spark-based serverless ETL engine in AWS Glue. Glue Data Studio is not an ETL solution.
References:
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
