Skip to content
Snippets Groups Projects
  • Cheolsoo Park's avatar
    ac2e17b0
    [SPARK-8355] [SQL] Python DataFrameReader/Writer should mirror Scala · ac2e17b0
    Cheolsoo Park authored
    I compared PySpark DataFrameReader/Writer against Scala ones. `Option` function is missing in both reader and writer, but the rest seems to all match.
    
    I added `Option` to reader and writer and updated the `pyspark-sql` test.
    
    Author: Cheolsoo Park <cheolsoop@netflix.com>
    
    Closes #7078 from piaozhexiu/SPARK-8355 and squashes the following commits:
    
    c63d419 [Cheolsoo Park] Fix version
    524e0aa [Cheolsoo Park] Add option function to df reader and writer
    ac2e17b0
    History
    [SPARK-8355] [SQL] Python DataFrameReader/Writer should mirror Scala
    Cheolsoo Park authored
    I compared PySpark DataFrameReader/Writer against Scala ones. `Option` function is missing in both reader and writer, but the rest seems to all match.
    
    I added `Option` to reader and writer and updated the `pyspark-sql` test.
    
    Author: Cheolsoo Park <cheolsoop@netflix.com>
    
    Closes #7078 from piaozhexiu/SPARK-8355 and squashes the following commits:
    
    c63d419 [Cheolsoo Park] Fix version
    524e0aa [Cheolsoo Park] Add option function to df reader and writer