-
- Downloads
[SPARK-19727][SQL] Fix for round function that modifies original column
## What changes were proposed in this pull request? Fix for SQL round function that modifies original column when underlying data frame is created from a local product. import org.apache.spark.sql.functions._ case class NumericRow(value: BigDecimal) val df = spark.createDataFrame(Seq(NumericRow(BigDecimal("1.23456789")))) df.show() +--------------------+ | value| +--------------------+ |1.234567890000000000| +--------------------+ df.withColumn("value_rounded", round('value)).show() // before +--------------------+-------------+ | value|value_rounded| +--------------------+-------------+ |1.000000000000000000| 1| +--------------------+-------------+ // after +--------------------+-------------+ | value|value_rounded| +--------------------+-------------+ |1.234567890000000000| 1| +--------------------+-------------+ ## How was this patch tested? New unit test added to existing suite `org.apache.spark.sql.MathFunctionsSuite` Author: Wojtek Szymanski <wk.szymanski@gmail.com> Closes #17075 from wojtek-szymanski/SPARK-19727.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala 1 addition, 5 deletions...rg/apache/spark/sql/catalyst/CatalystTypeConverters.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala 11 additions, 2 deletions...cala/org/apache/spark/sql/catalyst/expressions/Cast.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/decimalExpressions.scala 2 additions, 8 deletions...e/spark/sql/catalyst/expressions/decimalExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala 1 addition, 1 deletion...ache/spark/sql/catalyst/expressions/mathExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala 20 additions, 8 deletions...t/src/main/scala/org/apache/spark/sql/types/Decimal.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala 7 additions, 1 deletion.../test/scala/org/apache/spark/sql/types/DecimalSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/MathFunctionsSuite.scala 12 additions, 0 deletions.../test/scala/org/apache/spark/sql/MathFunctionsSuite.scala
Loading
Please register or sign in to comment