-
- Downloads
You need to sign in or sign up before continuing.
[SPARK-13457][SQL] Removes DataFrame RDD operations
## What changes were proposed in this pull request? This is another try of PR #11323. This PR removes DataFrame RDD operations except for `foreach` and `foreachPartitions` (they are actions rather than transformations). Original calls are now replaced by calls to methods of `DataFrame.rdd`. PR #11323 was reverted because it introduced a regression: both `DataFrame.foreach` and `DataFrame.foreachPartitions` wrap underlying RDD operations with `withNewExecutionId` to track Spark jobs. But they are removed in #11323. ## How was the this patch tested? No extra tests are added. Existing tests should do the work. Author: Cheng Lian <lian@databricks.com> Closes #11388 from liancheng/remove-df-rdd-ops.
Showing
- examples/src/main/scala/org/apache/spark/examples/ml/DataFrameExample.scala 1 addition, 1 deletion...scala/org/apache/spark/examples/ml/DataFrameExample.scala
- examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala 4 additions, 4 deletions...la/org/apache/spark/examples/ml/DecisionTreeExample.scala
- examples/src/main/scala/org/apache/spark/examples/ml/OneVsRestExample.scala 1 addition, 1 deletion...scala/org/apache/spark/examples/ml/OneVsRestExample.scala
- examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala 1 addition, 0 deletions...in/scala/org/apache/spark/examples/mllib/LDAExample.scala
- examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/examples/sql/RDDRelation.scala
- examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala 1 addition, 1 deletion...la/org/apache/spark/examples/sql/hive/HiveFromSpark.scala
- mllib/src/main/scala/org/apache/spark/ml/Predictor.scala 4 additions, 2 deletionsmllib/src/main/scala/org/apache/spark/ml/Predictor.scala
- mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala 8 additions, 5 deletions...g/apache/spark/ml/classification/LogisticRegression.scala
- mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala 2 additions, 2 deletions...cala/org/apache/spark/ml/clustering/BisectingKMeans.scala
- mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala 3 additions, 3 deletions...rc/main/scala/org/apache/spark/ml/clustering/KMeans.scala
- mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala 1 addition, 0 deletions...b/src/main/scala/org/apache/spark/ml/clustering/LDA.scala
- mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala 4 additions, 5 deletions...e/spark/ml/evaluation/BinaryClassificationEvaluator.scala
- mllib/src/main/scala/org/apache/spark/ml/evaluation/MulticlassClassificationEvaluator.scala 3 additions, 3 deletions...ark/ml/evaluation/MulticlassClassificationEvaluator.scala
- mllib/src/main/scala/org/apache/spark/ml/evaluation/RegressionEvaluator.scala 2 additions, 1 deletion.../org/apache/spark/ml/evaluation/RegressionEvaluator.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/ml/feature/ChiSqSelector.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala 1 addition, 1 deletion...n/scala/org/apache/spark/ml/feature/CountVectorizer.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala 1 addition, 1 deletionmllib/src/main/scala/org/apache/spark/ml/feature/IDF.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala 1 addition, 1 deletion...main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala 1 addition, 1 deletion...main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/ml/feature/OneHotEncoder.scala
Loading
Please register or sign in to comment