Question 6

Correct answer: A

A.

AWS Glue ETL job bookmarks are designed to track the state of previously processed data, ensuring that subsequent job runs process only new or modified records. This feature records the point up to which data was successfully processed, thereby enabling incremental data loading and reducing redundant processing. By using bookmarks, companies can optimize their ETL processes and minimize resource consumption. The bookmarks are automatically managed by AWS Glue, which simplifies handling incremental data loads. This approach directly addresses the company’s need to process only the delta changes in their data.

B.

The AWS Glue Data Catalog is a central repository for storing metadata information about data sources. It provides a way to organize and search for data assets, but it does not keep track of the data processing state within ETL jobs. The Data Catalog does not record which parts of the data have been processed previously, meaning it cannot support incremental loading on its own. While it is crucial for data organization and schema management, it is not designed to enable incremental processing. Therefore, it is not the appropriate choice for ensuring that only new or updated data is processed during ETL jobs.

C.

AWS Glue Crawlers are primarily used to populate the AWS Glue Data Catalog by crawling data stores to infer schema and update metadata. While crawlers can be configured to crawl only new subfolders or limit the number of files, they do not maintain state information about processed data for ETL jobs. This means that crawlers cannot be used to track which data has already been processed for transformation purposes. Their function is to organize metadata rather than facilitate incremental data processing. Therefore, this option does not meet the company’s requirement for processing only new or updated data.

D.

AWS Glue triggers are used to schedule and orchestrate ETL jobs, determining when a job should start. Although they play a vital role in automating job execution, triggers do not have any capability to track the progress of data processing or manage incremental loads. They simply initiate the job at a specified time or event and are not involved in determining which data is new or modified. This option would help with job automation but not with incremental processing. Hence, it does not solve the problem of processing only new or updated data.