-
- Downloads
[SPARK-9116] [SQL] [PYSPARK] support Python only UDT in __main__
Also we could create a Python UDT without having a Scala one, it's important for Python users. cc mengxr JoshRosen Author: Davies Liu <davies@databricks.com> Closes #7453 from davies/class_in_main and squashes the following commits: 4dfd5e1 [Davies Liu] add tests for Python and Scala UDT 793d9b2 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main dc65f19 [Davies Liu] address comment a9a3c40 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main a86e1fc [Davies Liu] fix serialization ad528ba [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main 63f52ef [Davies Liu] fix pylint check 655b8a9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into class_in_main 316a394 [Davies Liu] support Python UDT with UTF 0bcb3ef [Davies Liu] fix bug in mllib de986d6 [Davies Liu] fix test 83d65ac [Davies Liu] fix bug in StructType 55bb86e [Davies Liu] support Python UDT in __main__ (without Scala one)
Showing
- pylintrc 1 addition, 1 deletionpylintrc
- python/pyspark/cloudpickle.py 37 additions, 1 deletionpython/pyspark/cloudpickle.py
- python/pyspark/shuffle.py 1 addition, 1 deletionpython/pyspark/shuffle.py
- python/pyspark/sql/context.py 67 additions, 41 deletionspython/pyspark/sql/context.py
- python/pyspark/sql/tests.py 103 additions, 9 deletionspython/pyspark/sql/tests.py
- python/pyspark/sql/types.py 39 additions, 39 deletionspython/pyspark/sql/types.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataType.scala 9 additions, 0 deletions.../src/main/scala/org/apache/spark/sql/types/DataType.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/UserDefinedType.scala 29 additions, 0 deletions...in/scala/org/apache/spark/sql/types/UserDefinedType.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUDFs.scala 0 additions, 1 deletion...ain/scala/org/apache/spark/sql/execution/pythonUDFs.scala
Loading
Please register or sign in to comment