-
- Downloads
[SPARK-6911] [SQL] improve accessor for nested types
Support access columns by index in Python: ``` >>> df[df[0] > 3].collect() [Row(age=5, name=u'Bob')] ``` Access items in ArrayType or MapType ``` >>> df.select(df.l.getItem(0), df.d.getItem("key")).show() >>> df.select(df.l[0], df.d["key"]).show() ``` Access field in StructType ``` >>> df.select(df.r.getField("b")).show() >>> df.select(df.r.a).show() ``` Author: Davies Liu <davies@databricks.com> Closes #5513 from davies/access and squashes the following commits: e04d5a0 [Davies Liu] Update run-tests-jenkins 7ada9eb [Davies Liu] update timeout d125ac4 [Davies Liu] check column name, improve scala tests 6b62540 [Davies Liu] fix test db15b42 [Davies Liu] Merge branch 'master' of github.com:apache/spark into access 6c32e79 [Davies Liu] add scala tests 11f1df3 [Davies Liu] improve accessor for nested types
Showing
- python/pyspark/sql/dataframe.py 44 additions, 5 deletionspython/pyspark/sql/dataframe.py
- python/pyspark/sql/tests.py 18 additions, 0 deletionspython/pyspark/sql/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 4 additions, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 6 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/TestData.scala 4 additions, 5 deletionssql/core/src/test/scala/org/apache/spark/sql/TestData.scala
Loading
Please register or sign in to comment