Skip to content

GitLab

Explore

Sign in

Primary navigation

Project

S

spark
- Activity
- Members
- Labels
- Issues
- Issue boards
- Milestones
- Wiki
- Releases
- Model registry
- Environments
- Incidents

Snippets Groups Projects

7e4b2666

Commit 7e4b2666 authored 11 years ago by Matei Zaharia

Downloads
- Plain Diff

Merge pull request #563 from jey/python-optimization

Optimize PySpark worker invocation

parents b350f347 c75bed0e

No related branches found

No related tags found

No related merge requests found

Changes 7

Hide whitespace changes

Inline Side-by-side

Showing

core/src/main/scala/spark/SparkEnv.scala 12 additions, 0 deletions

core/src/main/scala/spark/SparkEnv.scala
core/src/main/scala/spark/api/python/PythonRDD.scala 36 additions, 40 deletions

core/src/main/scala/spark/api/python/PythonRDD.scala
core/src/main/scala/spark/api/python/PythonWorkerFactory.scala 95 additions, 0 deletions

...src/main/scala/spark/api/python/PythonWorkerFactory.scala
python/pyspark/daemon.py 158 additions, 0 deletions

python/pyspark/daemon.py
python/pyspark/serializers.py 4 additions, 0 deletions

python/pyspark/serializers.py
python/pyspark/tests.py 43 additions, 0 deletions

python/pyspark/tests.py
python/pyspark/worker.py 34 additions, 21 deletions

python/pyspark/worker.py

with 382 additions and 61 deletions

Loading

0% Loading or .

You are about to add 0 people to the discussion. Proceed with caution.

Finish editing this message first!

Please register or sign in to comment