-
- Downloads
[SPARK-6216] [PYSPARK] check python version of worker with driver
This PR revert #5404, change to pass the version of python in driver into JVM, check it in worker before deserializing closure, then it can works with different major version of Python. Author: Davies Liu <davies@databricks.com> Closes #6203 from davies/py_version and squashes the following commits: b8fb76e [Davies Liu] fix test 6ce5096 [Davies Liu] use string for version 47c6278 [Davies Liu] check python version of worker with driver
Showing
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 3 additions, 0 deletions...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- python/pyspark/context.py 1 addition, 0 deletionspython/pyspark/context.py
- python/pyspark/rdd.py 2 additions, 2 deletionspython/pyspark/rdd.py
- python/pyspark/sql/context.py 1 addition, 0 deletionspython/pyspark/sql/context.py
- python/pyspark/sql/functions.py 2 additions, 2 deletionspython/pyspark/sql/functions.py
- python/pyspark/tests.py 3 additions, 3 deletionspython/pyspark/tests.py
- python/pyspark/worker.py 7 additions, 5 deletionspython/pyspark/worker.py
- sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala 2 additions, 0 deletions...src/main/scala/org/apache/spark/sql/UDFRegistration.scala
- sql/core/src/main/scala/org/apache/spark/sql/UserDefinedFunction.scala 3 additions, 2 deletions...main/scala/org/apache/spark/sql/UserDefinedFunction.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala 2 additions, 0 deletions...ain/scala/org/apache/spark/sql/execution/pythonUdfs.scala
Loading
Please register or sign in to comment