-
- Downloads
[SPARK-16063][SQL] Add storageLevel to Dataset
[SPARK-11905](https://issues.apache.org/jira/browse/SPARK-11905 ) added support for `persist`/`cache` for `Dataset`. However, there is no user-facing API to check if a `Dataset` is cached and if so what the storage level is. This PR adds `getStorageLevel` to `Dataset`, analogous to `RDD.getStorageLevel`. Updated `DatasetCacheSuite`. Author: Nick Pentreath <nickp@za.ibm.com> Closes #13780 from MLnick/ds-storagelevel. Signed-off-by:Michael Armbrust <michael@databricks.com>
Showing
- python/pyspark/sql/dataframe.py 30 additions, 6 deletionspython/pyspark/sql/dataframe.py
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 12 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala 26 additions, 10 deletions...c/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
Loading
Please register or sign in to comment