Skip to content
  • Holden Karau's avatar
    d1ca634d
    [SPARK-12300] [SQL] [PYSPARK] fix schema inferance on local collections · d1ca634d
    Holden Karau authored
    Current schema inference for local python collections halts as soon as there are no NullTypes. This is different than when we specify a sampling ratio of 1.0 on a distributed collection. This could result in incomplete schema information.
    
    Author: Holden Karau <holden@us.ibm.com>
    
    Closes #10275 from holdenk/SPARK-12300-fix-schmea-inferance-on-local-collections.
    d1ca634d
    [SPARK-12300] [SQL] [PYSPARK] fix schema inferance on local collections
    Holden Karau authored
    Current schema inference for local python collections halts as soon as there are no NullTypes. This is different than when we specify a sampling ratio of 1.0 on a distributed collection. This could result in incomplete schema information.
    
    Author: Holden Karau <holden@us.ibm.com>
    
    Closes #10275 from holdenk/SPARK-12300-fix-schmea-inferance-on-local-collections.
Loading