AWS Certified Data Engineer Associate · Free Practice Question Medium
Question 27
As part of a workflow, the data engineering team at a company pushes data from Amazon Kinesis Data Firehose to Amazon Simple Storage Service (Amazon S3). However, the team has noticed that Kinesis Data Firehose is creating several small files in the Amazon S3 bucket, as opposed to a much lower expected number of files.
Which of the following would you attribute as the most likely cause behind this issue?
-
A
Compression is disabled on the Kinesis Data Firehose delivery stream
-
B
Data delivery to destination is lagging behind the data being written to the delivery stream
-
C
Kinesis Data Firehose delivery stream has scaled
-
D
A single delivery stream is configured to deliver data to multiple Amazon S3 buckets
Reveal correct answer
Correct answer: C
Explanation
Correct option:
Kinesis Data Firehose delivery stream has scaled
If a limit increase was requested or Kinesis Data Firehose has automatically scaled, then the Data Firehose delivery stream can scale. By default, Kinesis Data Firehose automatically scales delivery streams up to a certain limit. Amazon Kinesis' automatic scaling behavior reduces the likelihood of throttling without requiring a limit increase.
When Kinesis Data Firehose's delivery stream scales, it can cause an effect on the buffering hints?? of Data Firehose. The overall buffer size (SizeInMBs) of the delivery stream scales proportionally but inversely. For example, if the capacity of Kinesis Data Firehose increases by two times the original buffer size limit, the buffer size is halved. If Kinesis Data Firehose scales up to four times, the buffer size reduces to one-quarter of the overall buffer size.
There is also a proportional number of parallel buffering within the Kinesis Data Firehose delivery stream, where data is delivered simultaneously from all these buffers. For example, the Kinesis Data Firehose can buffer the data and create a single file based on the buffer size limit. If Kinesis Data Firehose scales to double the buffer limit, then two separate channels will create the files within the same time interval. If Kinesis Data Firehose scales up to four times, there will be four different channels creating four files in S3 during the same time interval.
Incorrect options:
Data delivery to destination is lagging behind the data being written into the delivery stream - When data delivery to destination is lagging behind the data being written into the delivery stream, Firehose raises buffer size dynamically to catch up and make sure that all data is delivered to the destination. In these circumstances, the size of delivered S3 objects might be larger than the specified buffer size.
Compression is disabled on the Kinesis Data Firehose delivery stream - This statement is incorrect. Kinesis Data Firehose delivers smaller records than specified (in the BufferingHints API) if compression is enabled on your Kinesis Data Firehose delivery stream.
A single delivery stream is configured to deliver data to multiple Amazon S3 buckets - This statement is incorrect. A single delivery stream can only deliver data to only one Amazon S3 bucket.
Reference:
https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-small-files-s3/
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
