Skip to content
Snippets Groups Projects
  • Jason Lee's avatar
    edd47375
    [SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with... · edd47375
    Jason Lee authored
    [SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with `None` triggers cryptic failure
    
    The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.
    
    Author: Jason Lee <cjlee@us.ibm.com>
    
    Closes #8969 from jasoncl/SPARK-10847.
    edd47375
    History
    [SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with...
    Jason Lee authored
    [SPARK-10847][SQL][PYSPARK] Pyspark - DataFrame - Optional Metadata with `None` triggers cryptic failure
    
    The error message is now changed from "Do not support type class scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be more informative about what is not supported. Also, StructType metadata now handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to tests.py to show the fix works.
    
    Author: Jason Lee <cjlee@us.ibm.com>
    
    Closes #8969 from jasoncl/SPARK-10847.