Skip to content
Snippets Groups Projects
  • Dongjoon Hyun's avatar
    2022afe5
    [SPARK-15773][CORE][EXAMPLE] Avoid creating local variable `sc` in examples if possible · 2022afe5
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    Instead of using local variable `sc` like the following example, this PR uses `spark.sparkContext`. This makes examples more concise, and also fixes some misleading, i.e., creating SparkContext from SparkSession.
    ```
    -    println("Creating SparkContext")
    -    val sc = spark.sparkContext
    -
         println("Writing local file to DFS")
         val dfsFilename = dfsDirPath + "/dfs_read_write_test"
    -    val fileRDD = sc.parallelize(fileContents)
    +    val fileRDD = spark.sparkContext.parallelize(fileContents)
    ```
    
    This will change 12 files (+30 lines, -52 lines).
    
    ## How was this patch tested?
    
    Manual.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #13520 from dongjoon-hyun/SPARK-15773.
    2022afe5
    History
    [SPARK-15773][CORE][EXAMPLE] Avoid creating local variable `sc` in examples if possible
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    Instead of using local variable `sc` like the following example, this PR uses `spark.sparkContext`. This makes examples more concise, and also fixes some misleading, i.e., creating SparkContext from SparkSession.
    ```
    -    println("Creating SparkContext")
    -    val sc = spark.sparkContext
    -
         println("Writing local file to DFS")
         val dfsFilename = dfsDirPath + "/dfs_read_write_test"
    -    val fileRDD = sc.parallelize(fileContents)
    +    val fileRDD = spark.sparkContext.parallelize(fileContents)
    ```
    
    This will change 12 files (+30 lines, -52 lines).
    
    ## How was this patch tested?
    
    Manual.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #13520 from dongjoon-hyun/SPARK-15773.