AWS Certified Data Engineer Associate · Free Practice Question Medium
Question 44
A company runs a real-time data processing application that uses Kinesis Client Library (KCL) to help consume and process data from the real-time data streams. The development team has raised a query on the viability of using the same DynamoDB table for different KCL applications.
Which of the following are correct statements for KCL while consuming Kinesis Data Streams? (Select two)
-
A
Amazon Relational Database Service (Amazon RDS) can also be used for checkpointing KCL
-
B
Multiple KCL applications can share a DynamoDB table if Amazon Amazon Simple Storage Service (Amazon S3) is configured to save transit data and metadata
-
C
Each KCL application must use its own DynamoDB table
-
D
You can only use DynamoDB for checkpointing KCL
-
E
Multiple KCL applications can share a DynamoDB table
Reveal correct answers
Correct answers: C, D
Explanation
Correct options:
Each KCL application must use its own DynamoDB table
Users can't use different KCL applications with the same DynamoDB table for the following reasons:
Scan operations are used to obtain leases from a DynamoDB table. Therefore, if a table contains leases of different KCL applications, each application could receive a lease that isn't related to the application itself.
Shard IDs in streams are used as primary keys in DynamoDB tables during checkpointing. When different KCL applications use the same DynamoDB table and the same shard IDs are used in the streams, inconsistencies in checkpoints can occur.
You can only use DynamoDB for checkpointing KCL - Users can only use DynamoDB as a checkpointing table for the KCL. A DynamoDB table is required as a checkpointing table for the KCL because the KCL behavior and implementation are interconnected with DynamoDB in the following ways:
The KCL includes
ShardSyncTask.java, which guarantees that shard leases in a stream are included in the DynamoDB table. This check is conducted periodically in the KCL.The KCL includes
DynamoDBLeaseTaker.javaandDynamoDBLeaseRenewer.java, which are components that manage and update leases in the KCL.DynamoDBLeaseTaker.javaandDynamoDBLeaseRenewer.javawork withDynamoDBLeaseRefresher.javato make frequent API requests to DynamoDB.When the KCL makes checkpoints, requests from
DynamoDBCheckpointer.javaandDynamoDBLeaseCoordinator.javaare made to DynamoDB.
Incorrect options:
Multiple KCL applications can share a DynamoDB table
Amazon Relational Database Service (Amazon RDS) can also be used for checkpointing KCL
Multiple KCL applications can share a DynamoDB table if Amazon Amazon Simple Storage Service (Amazon S3) is configured to save transit data and metadata
These three options are incorrect. Each KCL application must use its own DynamoDB table and only DynamoDB can be used for checkpointing KCL.
Reference:
https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-kcl-apps-dynamodb-table/
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
