Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    6183b5e2
    [SPARK-6911] [SQL] improve accessor for nested types · 6183b5e2
    Davies Liu authored
    Support access columns by index in Python:
    ```
    >>> df[df[0] > 3].collect()
    [Row(age=5, name=u'Bob')]
    ```
    
    Access items in ArrayType or MapType
    ```
    >>> df.select(df.l.getItem(0), df.d.getItem("key")).show()
    >>> df.select(df.l[0], df.d["key"]).show()
    ```
    
    Access field in StructType
    ```
    >>> df.select(df.r.getField("b")).show()
    >>> df.select(df.r.a).show()
    ```
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #5513 from davies/access and squashes the following commits:
    
    e04d5a0 [Davies Liu] Update run-tests-jenkins
    7ada9eb [Davies Liu] update timeout
    d125ac4 [Davies Liu] check column name, improve scala tests
    6b62540 [Davies Liu] fix test
    db15b42 [Davies Liu] Merge branch 'master' of github.com:apache/spark into access
    6c32e79 [Davies Liu] add scala tests
    11f1df3 [Davies Liu] improve accessor for nested types
    6183b5e2
    History
    [SPARK-6911] [SQL] improve accessor for nested types
    Davies Liu authored
    Support access columns by index in Python:
    ```
    >>> df[df[0] > 3].collect()
    [Row(age=5, name=u'Bob')]
    ```
    
    Access items in ArrayType or MapType
    ```
    >>> df.select(df.l.getItem(0), df.d.getItem("key")).show()
    >>> df.select(df.l[0], df.d["key"]).show()
    ```
    
    Access field in StructType
    ```
    >>> df.select(df.r.getField("b")).show()
    >>> df.select(df.r.a).show()
    ```
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #5513 from davies/access and squashes the following commits:
    
    e04d5a0 [Davies Liu] Update run-tests-jenkins
    7ada9eb [Davies Liu] update timeout
    d125ac4 [Davies Liu] check column name, improve scala tests
    6b62540 [Davies Liu] fix test
    db15b42 [Davies Liu] Merge branch 'master' of github.com:apache/spark into access
    6c32e79 [Davies Liu] add scala tests
    11f1df3 [Davies Liu] improve accessor for nested types