Skip to content
  • Davies Liu's avatar
    885d1621
    [SPARK-3500] [SQL] use JavaSchemaRDD as SchemaRDD._jschema_rdd · 885d1621
    Davies Liu authored
    Currently, SchemaRDD._jschema_rdd is SchemaRDD, the Scala API (coalesce(), repartition()) can not been called in Python easily, there is no way to specify the implicit parameter `ord`. The _jrdd is an JavaRDD, so _jschema_rdd should also be JavaSchemaRDD.
    
    In this patch, change _schema_rdd to JavaSchemaRDD, also added an assert for it. If some methods are missing from JavaSchemaRDD, then it's called by _schema_rdd.baseSchemaRDD().xxx().
    
    BTW, Do we need JavaSQLContext?
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2369 from davies/fix_schemardd and squashes the following commits:
    
    abee159 [Davies Liu] use JavaSchemaRDD as SchemaRDD._jschema_rdd
    885d1621
    [SPARK-3500] [SQL] use JavaSchemaRDD as SchemaRDD._jschema_rdd
    Davies Liu authored
    Currently, SchemaRDD._jschema_rdd is SchemaRDD, the Scala API (coalesce(), repartition()) can not been called in Python easily, there is no way to specify the implicit parameter `ord`. The _jrdd is an JavaRDD, so _jschema_rdd should also be JavaSchemaRDD.
    
    In this patch, change _schema_rdd to JavaSchemaRDD, also added an assert for it. If some methods are missing from JavaSchemaRDD, then it's called by _schema_rdd.baseSchemaRDD().xxx().
    
    BTW, Do we need JavaSQLContext?
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2369 from davies/fix_schemardd and squashes the following commits:
    
    abee159 [Davies Liu] use JavaSchemaRDD as SchemaRDD._jschema_rdd
Loading