-
- Downloads
[SPARK-11775][PYSPARK][SQL] Allow PySpark to register Java UDF
Currently pyspark can only call the builtin java UDF, but can not call custom java UDF. It would be better to allow that. 2 benefits: * Leverage the power of rich third party java library * Improve the performance. Because if we use python UDF, python daemons will be started on worker which will affect the performance. Author: Jeff Zhang <zjffdu@apache.org> Closes #9766 from zjffdu/SPARK-11775.
Showing
- python/pyspark/sql/context.py 27 additions, 1 deletionpython/pyspark/sql/context.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/JavaTypeInference.scala
- sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala 73 additions, 2 deletions...src/main/scala/org/apache/spark/sql/UDFRegistration.scala
- sql/core/src/test/java/test/org/apache/spark/sql/JavaStringLength.java 30 additions, 0 deletions...test/java/test/org/apache/spark/sql/JavaStringLength.java
- sql/core/src/test/java/test/org/apache/spark/sql/JavaUDFSuite.java 21 additions, 0 deletions...src/test/java/test/org/apache/spark/sql/JavaUDFSuite.java
Loading
Please register or sign in to comment