-
- Downloads
[SPARK-13049] Add First/last with ignore nulls to functions.scala
This PR adds the ability to specify the ```ignoreNulls``` option to the functions dsl, e.g: ```df.select($"id", last($"value", ignoreNulls = true).over(Window.partitionBy($"id").orderBy($"other"))``` This PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6. cc yhuai Author: Herman van Hovell <hvanhovell@questtec.nl> Closes #10957 from hvanhovell/SPARK-13049.
Showing
- python/pyspark/sql/functions.py 24 additions, 2 deletionspython/pyspark/sql/functions.py
- python/pyspark/sql/tests.py 10 additions, 0 deletionspython/pyspark/sql/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 91 additions, 27 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameWindowSuite.scala 32 additions, 0 deletions...est/scala/org/apache/spark/sql/DataFrameWindowSuite.scala
Loading
Please register or sign in to comment