-
- Downloads
[SPARK-2736] PySpark converter and example script for reading Avro files
JIRA: https://issues.apache.org/jira/browse/SPARK-2736 This patch includes: 1. An Avro converter that converts Avro data types to Python. It handles all 3 Avro data mappings (Generic, Specific and Reflect). 2. An example Python script for reading Avro files using AvroKeyInputFormat and the converter. 3. Fixing a classloading issue. cc @MLnick @JoshRosen @mateiz Author: Kan Zhang <kzhang@apache.org> Closes #1916 from kanzhang/SPARK-2736 and squashes the following commits: 02443f8 [Kan Zhang] [SPARK-2736] Adding .avsc files to .rat-excludes f74e9a9 [Kan Zhang] [SPARK-2736] nit: clazz -> className 82cc505 [Kan Zhang] [SPARK-2736] Update data sample 0be7761 [Kan Zhang] [SPARK-2736] Example pyspark script and data files c8e5881 [Kan Zhang] [SPARK-2736] Trying to work with all 3 Avro data models 2271a5b [Kan Zhang] [SPARK-2736] Using the right class loader to find Avro classes 536876b [Kan Zhang] [SPARK-2736] Adding Avro to Java converter
Showing
- .rat-excludes 1 addition, 0 deletions.rat-excludes
- core/src/main/scala/org/apache/spark/api/python/PythonHadoopUtil.scala 2 additions, 1 deletion.../scala/org/apache/spark/api/python/PythonHadoopUtil.scala
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 12 additions, 12 deletions...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- examples/src/main/python/avro_inputformat.py 75 additions, 0 deletionsexamples/src/main/python/avro_inputformat.py
- examples/src/main/resources/user.avsc 8 additions, 0 deletionsexamples/src/main/resources/user.avsc
- examples/src/main/resources/users.avro 0 additions, 0 deletionsexamples/src/main/resources/users.avro
- examples/src/main/scala/org/apache/spark/examples/pythonconverters/AvroConverters.scala 130 additions, 0 deletions...ache/spark/examples/pythonconverters/AvroConverters.scala
Loading
Please register or sign in to comment