Databricks Certified Data Engineer Associate · Free Practice Question Medium
Question 6
When defining external tables using formats CSV, JSON, TEXT, BINARY any query on the external tables caches the data and location for performance reasons, so within a given spark session any new files that may have arrived will not be available after the initial query. How can we address this limitation?
-
A
UNCACHE TABLE table_name -
B
CACHE TABLE table_name -
C
REFRESH TABLE table_name -
D
BROADCAST TABLE table_name -
E
CLEAR CACH table_name
Reveal correct answer
Correct answer: C
Explanation
The answer is REFRESH TABLE table_name
REFRESH TABLE table_name will force Spark to refresh the availability of external files and any changes.
When spark queries an external table it caches the files associated with it, so that way if the table is queried again it can use the cached files so it does not have to retrieve them again from cloud object storage, but the drawback here is that if new files are available Spark does not know until the Refresh command is ran.
Discussion
Think the marked answer is wrong, or have a better explanation? Share it below — comments appear after review.
