-
- Downloads
[SPARK-15773][CORE][EXAMPLE] Avoid creating local variable `sc` in examples if possible
## What changes were proposed in this pull request? Instead of using local variable `sc` like the following example, this PR uses `spark.sparkContext`. This makes examples more concise, and also fixes some misleading, i.e., creating SparkContext from SparkSession. ``` - println("Creating SparkContext") - val sc = spark.sparkContext - println("Writing local file to DFS") val dfsFilename = dfsDirPath + "/dfs_read_write_test" - val fileRDD = sc.parallelize(fileContents) + val fileRDD = spark.sparkContext.parallelize(fileContents) ``` This will change 12 files (+30 lines, -52 lines). ## How was this patch tested? Manual. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #13520 from dongjoon-hyun/SPARK-15773.
Showing
- examples/src/main/python/pi.py 1 addition, 3 deletionsexamples/src/main/python/pi.py
- examples/src/main/python/transitive_closure.py 1 addition, 3 deletionsexamples/src/main/python/transitive_closure.py
- examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala 2 additions, 5 deletions...in/scala/org/apache/spark/examples/DFSReadWriteTest.scala
- examples/src/main/scala/org/apache/spark/examples/ExceptionHandlingTest.scala 1 addition, 2 deletions...ala/org/apache/spark/examples/ExceptionHandlingTest.scala
- examples/src/main/scala/org/apache/spark/examples/GroupByTest.scala 6 additions, 8 deletions...rc/main/scala/org/apache/spark/examples/GroupByTest.scala
- examples/src/main/scala/org/apache/spark/examples/MultiBroadcastTest.scala 3 additions, 5 deletions.../scala/org/apache/spark/examples/MultiBroadcastTest.scala
- examples/src/main/scala/org/apache/spark/examples/SimpleSkewedGroupByTest.scala 7 additions, 9 deletions...a/org/apache/spark/examples/SimpleSkewedGroupByTest.scala
- examples/src/main/scala/org/apache/spark/examples/SkewedGroupByTest.scala 5 additions, 8 deletions...n/scala/org/apache/spark/examples/SkewedGroupByTest.scala
- examples/src/main/scala/org/apache/spark/examples/SparkLR.scala 1 addition, 3 deletions...es/src/main/scala/org/apache/spark/examples/SparkLR.scala
- examples/src/main/scala/org/apache/spark/examples/SparkPi.scala 1 addition, 2 deletions...es/src/main/scala/org/apache/spark/examples/SparkPi.scala
- examples/src/main/scala/org/apache/spark/examples/SparkTC.scala 1 addition, 2 deletions...es/src/main/scala/org/apache/spark/examples/SparkTC.scala
- examples/src/main/scala/org/apache/spark/examples/sql/hive/HiveFromSpark.scala 1 addition, 2 deletions...la/org/apache/spark/examples/sql/hive/HiveFromSpark.scala
Loading
Please register or sign in to comment