-
- Downloads
[SPARK-19488][SQL] fix csv infer schema when the field is Nan/Inf etc
## What changes were proposed in this pull request? when csv infer schema, it does not use user defined csvoptions to parse the field, such as `inf`, `-inf` which are should be parsed to DoubleType this pr add `options.nanValue`, `options.negativeInf`, `options.positiveIn` to check if the field is a DoubleType ## How was this patch tested? unit test added Author: windpiger <songjun@outlook.com> Closes #16834 from windpiger/fixinferInfSchemaCsv.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala 5 additions, 1 deletion.../spark/sql/execution/datasources/csv/CSVInferSchema.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchemaSuite.scala 8 additions, 0 deletions...k/sql/execution/datasources/csv/CSVInferSchemaSuite.scala
Loading
Please register or sign in to comment