Skip to content
Snippets Groups Projects
  • Herman van Hovell's avatar
    5a8b978f
    [SPARK-13049] Add First/last with ignore nulls to functions.scala · 5a8b978f
    Herman van Hovell authored
    This PR adds the ability to specify the ```ignoreNulls``` option to the functions dsl, e.g:
    ```df.select($"id", last($"value", ignoreNulls = true).over(Window.partitionBy($"id").orderBy($"other"))```
    
    This PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6.
    
    cc yhuai
    
    Author: Herman van Hovell <hvanhovell@questtec.nl>
    
    Closes #10957 from hvanhovell/SPARK-13049.
    5a8b978f
    History
    [SPARK-13049] Add First/last with ignore nulls to functions.scala
    Herman van Hovell authored
    This PR adds the ability to specify the ```ignoreNulls``` option to the functions dsl, e.g:
    ```df.select($"id", last($"value", ignoreNulls = true).over(Window.partitionBy($"id").orderBy($"other"))```
    
    This PR is some where between a bug fix (see the JIRA) and a new feature. I am not sure if we should backport to 1.6.
    
    cc yhuai
    
    Author: Herman van Hovell <hvanhovell@questtec.nl>
    
    Closes #10957 from hvanhovell/SPARK-13049.