-
- Downloads
[SPARK-10177] [SQL] fix reading Timestamp in parquet from Hive
We misunderstood the Julian days and nanoseconds of the day in parquet (as TimestampType) from Hive/Impala, they are overlapped, so can't be added together directly. In order to avoid the confusing rounding when do the converting, we use `2440588` as the Julian Day of epoch of unix timestamp (which should be 2440587.5). Author: Davies Liu <davies@databricks.com> Author: Cheng Lian <lian@databricks.com> Closes #8400 from davies/timestamp_parquet.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala 4 additions, 3 deletions...la/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala 9 additions, 4 deletions...g/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala 1 addition, 1 deletion...apache/spark/sql/hive/ParquetHiveCompatibilitySuite.scala
Loading
Please register or sign in to comment