-
- Downloads
[SPARK-13467] [PYSPARK] abstract python function to simplify pyspark code
## What changes were proposed in this pull request? When we pass a Python function to JVM side, we also need to send its context, e.g. `envVars`, `pythonIncludes`, `pythonExec`, etc. However, it's annoying to pass around so many parameters at many places. This PR abstract python function along with its context, to simplify some pyspark code and make the logic more clear. ## How was the this patch tested? by existing unit tests. Author: Wenchen Fan <wenchen@databricks.com> Closes #11342 from cloud-fan/python-clean.
Showing
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 22 additions, 15 deletions...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- python/pyspark/rdd.py 14 additions, 9 deletionspython/pyspark/rdd.py
- python/pyspark/sql/context.py 1 addition, 1 deletionpython/pyspark/sql/context.py
- python/pyspark/sql/functions.py 3 additions, 5 deletionspython/pyspark/sql/functions.py
- sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala 4 additions, 4 deletions...src/main/scala/org/apache/spark/sql/UDFRegistration.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/python/BatchPythonEvaluation.scala 1 addition, 7 deletions...he/spark/sql/execution/python/BatchPythonEvaluation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDF.scala 3 additions, 10 deletions...ala/org/apache/spark/sql/execution/python/PythonUDF.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala 3 additions, 12 deletions...park/sql/execution/python/UserDefinedPythonFunction.scala
Loading
Please register or sign in to comment