AWS Certified Data Engineer Associate · Free Practice Question Medium
Question 11
The data engineering team at an e-commerce company processes transactions into Amazon Kinesis Data Streams using the Kinesis Producer Library (KPL). The Data Streams are managed via Auto Scaling configuration. On the other hand, the Kinesis Client Library (KCL) ingests the incoming data into the company's warehousing system to be used for downstream analytics. Lately, the data engineering team has come across issues arising out of duplicate records.
Which of the following would you identify as the most likely reason for this behavior?
-
A
The Kinesis Producer Library (KPL) is aggregating smaller records into larger records of up to 1 MB, sometimes resulting in duplicate records
-
B
If
PutRecords.Bytesmetric exceeds the provisioned write capacity, throttling for the stream kicks in, which results in record failures leading to re-writing of data by Kinesis Producer Library (KPL) -
C
The producer is experiencing network-related timeouts, forcing duplicate entries into the Kinesis Data Streams
-
D
If the
GetRecordcall fails without an acknowledgment from Amazon Kinesis Data Streams, the Kinesis Producer Library (KPL) will write the same data again
Reveal correct answer
Correct answer: C
Explanation
Correct option:
The producer is experiencing network-related timeouts, forcing duplicate entries into the Kinesis Data Stream - There are two primary reasons why records may be delivered more than once to your Amazon Kinesis Data Streams application: producer retries and consumer retries.
Consider a producer that experiences a network-related timeout after it makes a call to PutRecord, but before it can receive an acknowledgment from Amazon Kinesis Data Streams. The producer cannot be sure if the record was delivered to Kinesis Data Streams. Assuming that every record is important to the application, the producer would have been written to retry the call with the same data. If both PutRecord calls on that same data were successfully committed to Kinesis Data Streams, then there will be two Kinesis Data Streams records.
Applications that need strict guarantees should embed a primary key within the record to remove duplicates later when processing. Note that the number of duplicates due to producer retries is usually low compared to the number of duplicates due to consumer retries.
Incorrect options:
The Kinesis Producer Library (KPL) is aggregating smaller records into larger records of up to 1 MB, sometimes resulting in duplicate records - The Kinesis Producer Library (KPL) aggregates small user-formatted records into larger records up to 1 MB to make better use of Amazon Kinesis Data Streams throughput. This has no bearing on the duplicate records issue.
If the GetRecord call fails without an acknowledgment from Amazon Kinesis Data Streams, the Kinesis Producer Library (KPL) will write the same data again - It's the PutRecord call that results in writing duplicate records if the acknowledgment from Amazon Kinesis Data Streams is not received.
If PutRecords.Bytes metric exceeds the provisioned write capacity, throttling for the stream kicks in, which results in record failures leading to re-writing of data by Kinesis Producer Library (KPL) - Kinesis Data Streams sends these stream-level metrics to CloudWatch every minute. The PutRecords.Bytes` metric depicts the number of bytes put into the Kinesis stream using the PutRecords operation over the specified period. This metric has no relation to duplicate records.
References:
https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-duplicates.html
https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-cloudwatch.html
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
