-
- Downloads
[SPARK-15364][ML][PYSPARK] Implement PySpark picklers for ml.Vector and...
[SPARK-15364][ML][PYSPARK] Implement PySpark picklers for ml.Vector and ml.Matrix under spark.ml.python ## What changes were proposed in this pull request? Now we have PySpark picklers for new and old vector/matrix, individually. However, they are all implemented under `PythonMLlibAPI`. To separate spark.mllib from spark.ml, we should implement the picklers of new vector/matrix under `spark.ml.python` instead. ## How was this patch tested? Existing tests. Author: Liang-Chi Hsieh <simonh@tw.ibm.com> Closes #13219 from viirya/pyspark-pickler-ml.
Showing
- mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala 224 additions, 0 deletions...b/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala
- mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala 68 additions, 241 deletions...la/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala
- mllib/src/test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala 72 additions, 0 deletions.../test/scala/org/apache/spark/ml/python/MLSerDeSuite.scala
- python/pyspark/java_gateway.py 1 addition, 0 deletionspython/pyspark/java_gateway.py
- python/pyspark/ml/base.py 1 addition, 1 deletionpython/pyspark/ml/base.py
- python/pyspark/ml/classification.py 1 addition, 1 deletionpython/pyspark/ml/classification.py
- python/pyspark/ml/clustering.py 1 addition, 1 deletionpython/pyspark/ml/clustering.py
- python/pyspark/ml/common.py 137 additions, 0 deletionspython/pyspark/ml/common.py
- python/pyspark/ml/evaluation.py 1 addition, 1 deletionpython/pyspark/ml/evaluation.py
- python/pyspark/ml/feature.py 1 addition, 1 deletionpython/pyspark/ml/feature.py
- python/pyspark/ml/pipeline.py 1 addition, 1 deletionpython/pyspark/ml/pipeline.py
- python/pyspark/ml/recommendation.py 1 addition, 1 deletionpython/pyspark/ml/recommendation.py
- python/pyspark/ml/regression.py 1 addition, 1 deletionpython/pyspark/ml/regression.py
- python/pyspark/ml/tests.py 5 additions, 5 deletionspython/pyspark/ml/tests.py
- python/pyspark/ml/tuning.py 1 addition, 1 deletionpython/pyspark/ml/tuning.py
- python/pyspark/ml/util.py 1 addition, 1 deletionpython/pyspark/ml/util.py
- python/pyspark/ml/wrapper.py 1 addition, 1 deletionpython/pyspark/ml/wrapper.py
Loading
Please register or sign in to comment