-
- Downloads
[SPARK-17764][SQL] Add `to_json` supporting to convert nested struct column to JSON string
## What changes were proposed in this pull request? This PR proposes to add `to_json` function in contrast with `from_json` in Scala, Java and Python. It'd be useful if we can convert a same column from/to json. Also, some datasources do not support nested types. If we are forced to save a dataframe into those data sources, we might be able to work around by this function. The usage is as below: ``` scala val df = Seq(Tuple1(Tuple1(1))).toDF("a") df.select(to_json($"a").as("json")).show() ``` ``` bash +--------+ | json| +--------+ |{"_1":1}| +--------+ ``` ## How was this patch tested? Unit tests in `JsonFunctionsSuite` and `JsonExpressionsSuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #15354 from HyukjinKwon/SPARK-17764.
Showing
- python/pyspark/sql/functions.py 23 additions, 0 deletionspython/pyspark/sql/functions.py
- python/pyspark/sql/readwriter.py 1 addition, 1 deletionpython/pyspark/sql/readwriter.py
- python/pyspark/sql/streaming.py 1 addition, 1 deletionpython/pyspark/sql/streaming.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala 46 additions, 2 deletions...ache/spark/sql/catalyst/expressions/jsonExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala 2 additions, 3 deletions...org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala 26 additions, 0 deletions...ala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/JsonExpressionsSuite.scala 9 additions, 0 deletions...spark/sql/catalyst/expressions/JsonExpressionsSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 1 addition, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala 1 addition, 1 deletion...spark/sql/execution/datasources/json/JsonFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 43 additions, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala 24 additions, 6 deletions.../test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
Loading
Please register or sign in to comment