-
- Downloads
[SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with...
[SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with documentation improvement ## What changes were proposed in this pull request? This PR proposes three things as below: - Use casting rules to a timestamp in `to_timestamp` by default (it was `yyyy-MM-dd HH:mm:ss`). - Support single argument for `to_timestamp` similarly with APIs in other languages. For example, the one below works ``` import org.apache.spark.sql.functions._ Seq("2016-12-31 00:12:00.00").toDF("a").select(to_timestamp(col("a"))).show() ``` prints ``` +----------------------------------------+ |to_timestamp(`a`, 'yyyy-MM-dd HH:mm:ss')| +----------------------------------------+ | 2016-12-31 00:12:00| +----------------------------------------+ ``` whereas this does not work in SQL. **Before** ``` spark-sql> SELECT to_timestamp('2016-12-31 00:12:00'); Error in query: Invalid number of arguments for function to_timestamp; line 1 pos 7 ``` **After** ``` spark-sql> SELECT to_timestamp('2016-12-31 00:12:00'); 2016-12-31 00:12:00 ``` - Related document improvement for SQL function descriptions and other API descriptions accordingly. **Before** ``` spark-sql> DESCRIBE FUNCTION extended to_date; ... Usage: to_date(date_str, fmt) - Parses the `left` expression with the `fmt` expression. Returns null with invalid input. Extended Usage: Examples: > SELECT to_date('2016-12-31', 'yyyy-MM-dd'); 2016-12-31 ``` ``` spark-sql> DESCRIBE FUNCTION extended to_timestamp; ... Usage: to_timestamp(timestamp, fmt) - Parses the `left` expression with the `format` expression to a timestamp. Returns null with invalid input. Extended Usage: Examples: > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd'); 2016-12-31 00:00:00.0 ``` **After** ``` spark-sql> DESCRIBE FUNCTION extended to_date; ... Usage: to_date(date_str[, fmt]) - Parses the `date_str` expression with the `fmt` expression to a date. Returns null with invalid input. By default, it follows casting rules to a date if the `fmt` is omitted. Extended Usage: Examples: > SELECT to_date('2009-07-30 04:17:52'); 2009-07-30 > SELECT to_date('2016-12-31', 'yyyy-MM-dd'); 2016-12-31 ``` ``` spark-sql> DESCRIBE FUNCTION extended to_timestamp; ... Usage: to_timestamp(timestamp[, fmt]) - Parses the `timestamp` expression with the `fmt` expression to a timestamp. Returns null with invalid input. By default, it follows casting rules to a timestamp if the `fmt` is omitted. Extended Usage: Examples: > SELECT to_timestamp('2016-12-31 00:12:00'); 2016-12-31 00:12:00 > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd'); 2016-12-31 00:00:00 ``` ## How was this patch tested? Added tests in `datetime.sql`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #17901 from HyukjinKwon/to_timestamp_arg.
Showing
- R/pkg/R/functions.R 5 additions, 3 deletionsR/pkg/R/functions.R
- python/pyspark/sql/functions.py 6 additions, 10 deletionspython/pyspark/sql/functions.py
- python/pyspark/sql/tests.py 1 addition, 1 deletionpython/pyspark/sql/tests.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala 27 additions, 38 deletions.../spark/sql/catalyst/expressions/datetimeExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala 1 addition, 1 deletion...la/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala 0 additions, 8 deletions...spark/sql/catalyst/expressions/DateExpressionsSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 5 additions, 6 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/test/resources/sql-tests/inputs/datetime.sql 4 additions, 0 deletionssql/core/src/test/resources/sql-tests/inputs/datetime.sql
- sql/core/src/test/resources/sql-tests/results/datetime.sql.out 17 additions, 1 deletion...ore/src/test/resources/sql-tests/results/datetime.sql.out
- sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala 2 additions, 2 deletions.../test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
Loading
Please register or sign in to comment