-
- Downloads
[SPARK-3463] [PySpark] aggregate and show spilled bytes in Python
Aggregate the number of bytes spilled into disks during aggregation or sorting, show them in Web UI.  This patch is blocked by SPARK-3465. (It includes a fix for that). Author: Davies Liu <davies.liu@gmail.com> Closes #2336 from davies/metrics and squashes the following commits: e37df38 [Davies Liu] remove outdated comments 1245eb7 [Davies Liu] remove the temporary fix ebd2f43 [Davies Liu] Merge branch 'master' into metrics 7e4ad04 [Davies Liu] Merge branch 'master' into metrics fbe9029 [Davies Liu] show spilled bytes in Python in web ui
Showing
- core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala 4 additions, 0 deletions...rc/main/scala/org/apache/spark/api/python/PythonRDD.scala
- python/pyspark/shuffle.py 16 additions, 3 deletionspython/pyspark/shuffle.py
- python/pyspark/tests.py 8 additions, 7 deletionspython/pyspark/tests.py
- python/pyspark/worker.py 10 additions, 4 deletionspython/pyspark/worker.py
Loading
Please register or sign in to comment