-
- Downloads
[SPARK-4192][SQL] Internal API for Python UDT
Following #2919, this PR adds Python UDT (for internal use only) with tests under "pyspark.tests". Before `SQLContext.applySchema`, we check whether we need to convert user-type instances into SQL recognizable data. In the current implementation, a Python UDT must be paired with a Scala UDT for serialization on the JVM side. A following PR will add VectorUDT in MLlib for both Scala and Python. marmbrus jkbradley davies Author: Xiangrui Meng <meng@databricks.com> Closes #3068 from mengxr/SPARK-4192-sql and squashes the following commits: acff637 [Xiangrui Meng] merge master dba5ea7 [Xiangrui Meng] only use pyClass for Python UDT output sqlType as well 2c9d7e4 [Xiangrui Meng] move import to global setup; update needsConversion 7c4a6a9 [Xiangrui Meng] address comments 75223db [Xiangrui Meng] minor update f740379 [Xiangrui Meng] remove UDT from default imports e98d9d0 [Xiangrui Meng] fix py style 4e84fce [Xiangrui Meng] remove local hive tests and add more tests 39f19e0 [Xiangrui Meng] add tests b7f666d [Xiangrui Meng] add Python UDT
Showing
- python/pyspark/sql.py 204 additions, 2 deletionspython/pyspark/sql.py
- python/pyspark/tests.py 92 additions, 1 deletionpython/pyspark/tests.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/dataTypes.scala 8 additions, 1 deletion...scala/org/apache/spark/sql/catalyst/types/dataTypes.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 2 additions, 0 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala 5 additions, 0 deletions...ain/scala/org/apache/spark/sql/execution/pythonUdfs.scala
- sql/core/src/main/scala/org/apache/spark/sql/test/ExamplePointUDT.scala 64 additions, 0 deletions...ain/scala/org/apache/spark/sql/test/ExamplePointUDT.scala
- sql/core/src/main/scala/org/apache/spark/sql/types/util/DataTypeConversions.scala 0 additions, 1 deletion...org/apache/spark/sql/types/util/DataTypeConversions.scala
Loading
Please register or sign in to comment