Skip to content
  • hyukjinkwon's avatar
    720708cc
    [SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with... · 720708cc
    hyukjinkwon authored
    [SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with documentation improvement
    
    ## What changes were proposed in this pull request?
    
    This PR proposes three things as below:
    
    - Use casting rules to a timestamp in `to_timestamp` by default (it was `yyyy-MM-dd HH:mm:ss`).
    
    - Support single argument for `to_timestamp` similarly with APIs in other languages.
    
      For example, the one below works
    
      ```
      import org.apache.spark.sql.functions._
      Seq("2016-12-31 00:12:00.00").toDF("a").select(to_timestamp(col("a"))).show()
      ```
    
      prints
    
      ```
      +----------------------------------------+
      |to_timestamp(`a`, 'yyyy-MM-dd HH:mm:ss')|
      +----------------------------------------+
      |                     2016-12-31 00:12:00|
      +----------------------------------------+
      ```
    
      whereas this does not work in SQL.
    
      **Before**
    
      ```
      spark-sql> SELECT to_timestamp('2016-12-31 00:12:00');
      Error in query: Invalid number of arguments for function to_timestamp; line 1 pos 7
      ```
    
      **After**
    
      ```
      spark-sql> SELECT to_timestamp('2016-12-31 00:12:00');
      2016-12-31 00:12:00
      ```
    
    - Related document improvement for SQL function descriptions and other API descriptions accordingly.
    
      **Before**
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_date;
      ...
      Usage: to_date(date_str, fmt) - Parses the `left` expression with the `fmt` expression. Returns null with invalid input.
      Extended Usage:
          Examples:
            > SELECT to_date('2016-12-31', 'yyyy-MM-dd');
             2016-12-31
      ```
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_timestamp;
      ...
      Usage: to_timestamp(timestamp, fmt) - Parses the `left` expression with the `format` expression to a timestamp. Returns null with invalid input.
      Extended Usage:
          Examples:
            > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
             2016-12-31 00:00:00.0
      ```
    
      **After**
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_date;
      ...
      Usage:
          to_date(date_str[, fmt]) - Parses the `date_str` expression with the `fmt` expression to
            a date. Returns null with invalid input. By default, it follows casting rules to a date if
            the `fmt` is omitted.
    
      Extended Usage:
          Examples:
            > SELECT to_date('2009-07-30 04:17:52');
             2009-07-30
            > SELECT to_date('2016-12-31', 'yyyy-MM-dd');
             2016-12-31
      ```
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_timestamp;
      ...
       Usage:
          to_timestamp(timestamp[, fmt]) - Parses the `timestamp` expression with the `fmt` expression to
            a timestamp. Returns null with invalid input. By default, it follows casting rules to
            a timestamp if the `fmt` is omitted.
    
      Extended Usage:
          Examples:
            > SELECT to_timestamp('2016-12-31 00:12:00');
             2016-12-31 00:12:00
            > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
             2016-12-31 00:00:00
      ```
    
    ## How was this patch tested?
    
    Added tests in `datetime.sql`.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #17901 from HyukjinKwon/to_timestamp_arg.
    720708cc
    [SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with...
    hyukjinkwon authored
    [SPARK-20639][SQL] Add single argument support for to_timestamp in SQL with documentation improvement
    
    ## What changes were proposed in this pull request?
    
    This PR proposes three things as below:
    
    - Use casting rules to a timestamp in `to_timestamp` by default (it was `yyyy-MM-dd HH:mm:ss`).
    
    - Support single argument for `to_timestamp` similarly with APIs in other languages.
    
      For example, the one below works
    
      ```
      import org.apache.spark.sql.functions._
      Seq("2016-12-31 00:12:00.00").toDF("a").select(to_timestamp(col("a"))).show()
      ```
    
      prints
    
      ```
      +----------------------------------------+
      |to_timestamp(`a`, 'yyyy-MM-dd HH:mm:ss')|
      +----------------------------------------+
      |                     2016-12-31 00:12:00|
      +----------------------------------------+
      ```
    
      whereas this does not work in SQL.
    
      **Before**
    
      ```
      spark-sql> SELECT to_timestamp('2016-12-31 00:12:00');
      Error in query: Invalid number of arguments for function to_timestamp; line 1 pos 7
      ```
    
      **After**
    
      ```
      spark-sql> SELECT to_timestamp('2016-12-31 00:12:00');
      2016-12-31 00:12:00
      ```
    
    - Related document improvement for SQL function descriptions and other API descriptions accordingly.
    
      **Before**
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_date;
      ...
      Usage: to_date(date_str, fmt) - Parses the `left` expression with the `fmt` expression. Returns null with invalid input.
      Extended Usage:
          Examples:
            > SELECT to_date('2016-12-31', 'yyyy-MM-dd');
             2016-12-31
      ```
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_timestamp;
      ...
      Usage: to_timestamp(timestamp, fmt) - Parses the `left` expression with the `format` expression to a timestamp. Returns null with invalid input.
      Extended Usage:
          Examples:
            > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
             2016-12-31 00:00:00.0
      ```
    
      **After**
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_date;
      ...
      Usage:
          to_date(date_str[, fmt]) - Parses the `date_str` expression with the `fmt` expression to
            a date. Returns null with invalid input. By default, it follows casting rules to a date if
            the `fmt` is omitted.
    
      Extended Usage:
          Examples:
            > SELECT to_date('2009-07-30 04:17:52');
             2009-07-30
            > SELECT to_date('2016-12-31', 'yyyy-MM-dd');
             2016-12-31
      ```
    
      ```
      spark-sql> DESCRIBE FUNCTION extended to_timestamp;
      ...
       Usage:
          to_timestamp(timestamp[, fmt]) - Parses the `timestamp` expression with the `fmt` expression to
            a timestamp. Returns null with invalid input. By default, it follows casting rules to
            a timestamp if the `fmt` is omitted.
    
      Extended Usage:
          Examples:
            > SELECT to_timestamp('2016-12-31 00:12:00');
             2016-12-31 00:12:00
            > SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd');
             2016-12-31 00:00:00
      ```
    
    ## How was this patch tested?
    
    Added tests in `datetime.sql`.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #17901 from HyukjinKwon/to_timestamp_arg.
Loading