-
- Downloads
[SPARK-21174][SQL] Validate sampling fraction in logical operator level
## What changes were proposed in this pull request? Currently the validation of sampling fraction in dataset is incomplete. As an improvement, validate sampling fraction in logical operator level: 1) if with replacement: fraction should be nonnegative 2) else: fraction should be on interval [0, 1] Also add test cases for the validation. ## How was this patch tested? integration tests gatorsmile cloud-fan Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Wang Gengliang <ltnwgl@gmail.com> Closes #18387 from gengliangwang/sample_ratio_validate.
Showing
- sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 1 addition, 1 deletion...in/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 2 additions, 1 deletion...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala 13 additions, 0 deletions...rk/sql/catalyst/plans/logical/basicLogicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 0 additions, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/test/resources/sql-tests/inputs/tablesample-negative.sql 14 additions, 0 deletions.../test/resources/sql-tests/inputs/tablesample-negative.sql
- sql/core/src/test/resources/sql-tests/results/operators.sql.out 4 additions, 4 deletions...re/src/test/resources/sql-tests/results/operators.sql.out
- sql/core/src/test/resources/sql-tests/results/tablesample-negative.sql.out 62 additions, 0 deletions.../resources/sql-tests/results/tablesample-negative.sql.out
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 28 additions, 0 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment