AWS Certified Machine Learning Specialty · Free Practice Question Medium
Question 18
You want to create an AWS Glue crawler to read the transaction data dumped into an S3 based data lake in the s3://mybucket/myfolder/ location. The transaction data is in CSV format however there are some additional metadata files with .metadata extension in the same location. The metadata needs to be ignored while reading the transaction data via Athena.
How would you implement this solution?
-
A
Use exclude pattern **.metadata in the crawler definition to ignore the metadata
-
B
It is not possible to ignore the metadata in crawler. Create a daily ETL job to transfer only the transaction data specific CSV files into a new location and then read this cleansed transaction data into Athena
Reveal correct answer
Correct answer: A
Explanation
Correct option:
Use exclude pattern **.metadata in the crawler definition to ignore the metadata
AWS Glue crawler supports exclude patterns. Correct option is to use exclude pattern **.metadata in the crawler definition to ignore the metadata.
Here is a deep-dive for the exclude patterns:
https://docs.aws.amazon.com/glue/latest/dg/define-crawler.html#crawler-data-stores-exclude
Incorrect options:
It is not possible to ignore the metadata in crawler. Create a daily ETL job to transfer only the transaction data specific CSV files into a new location and then read this cleansed transaction data into Athena - This option has been added as a distractor.
Use exclude pattern .metadata/** in the crawler definition to ignore the metadata
Use exclude pattern .**metadata in the crawler definition to ignore the metadata
The correct pattern has been described in the explanation above, so both these options are incorrect.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
