-
- Downloads
[SPARK-18342] Make rename failures fatal in HDFSBackedStateStore
## What changes were proposed in this pull request? If the rename operation in the state store fails (`fs.rename` returns `false`), the StateStore should throw an exception and have the task retry. Currently if renames fail, nothing happens during execution immediately. However, you will observe that snapshot operations will fail, and then any attempt at recovery (executor failure / checkpoint recovery) also fails. ## How was this patch tested? Unit test Author: Burak Yavuz <brkyvz@gmail.com> Closes #15804 from brkyvz/rename-state. (cherry picked from commit 6f7ecb0f) Signed-off-by:Tathagata Das <tathagata.das1565@gmail.com>
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala 4 additions, 2 deletions...cution/streaming/state/HDFSBackedStateStoreProvider.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala 36 additions, 5 deletions...spark/sql/execution/streaming/state/StateStoreSuite.scala
Loading
Please register or sign in to comment