Skip to content
Snippets Groups Projects
  • Wenchen Fan's avatar
    a60f9128
    [SPARK-13467] [PYSPARK] abstract python function to simplify pyspark code · a60f9128
    Wenchen Fan authored
    ## What changes were proposed in this pull request?
    
    When we pass a Python function to JVM side, we also need to send its context, e.g. `envVars`, `pythonIncludes`, `pythonExec`, etc. However, it's annoying to pass around so many parameters at many places. This PR abstract python function along with its context, to simplify some pyspark code and make the logic more clear.
    
    ## How was the this patch tested?
    
    by existing unit tests.
    
    Author: Wenchen Fan <wenchen@databricks.com>
    
    Closes #11342 from cloud-fan/python-clean.
    a60f9128
    History
    [SPARK-13467] [PYSPARK] abstract python function to simplify pyspark code
    Wenchen Fan authored
    ## What changes were proposed in this pull request?
    
    When we pass a Python function to JVM side, we also need to send its context, e.g. `envVars`, `pythonIncludes`, `pythonExec`, etc. However, it's annoying to pass around so many parameters at many places. This PR abstract python function along with its context, to simplify some pyspark code and make the logic more clear.
    
    ## How was the this patch tested?
    
    by existing unit tests.
    
    Author: Wenchen Fan <wenchen@databricks.com>
    
    Closes #11342 from cloud-fan/python-clean.