Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    e34f38ff
    [SPARK-4017] show progress bar in console · e34f38ff
    Davies Liu authored
    The progress bar will look like this:
    
    ![1___spark_job__85_250_finished__4_are_running___java_](https://cloud.githubusercontent.com/assets/40902/4854813/a02f44ac-6099-11e4-9060-7c73a73151d6.png)
    
    In the right corner, the numbers are: finished tasks, running tasks, total tasks.
    
    After the stage has finished, it will disappear.
    
    The progress bar is only showed if logging level is WARN or higher (but progress in title is still showed), it can be turned off by spark.driver.showConsoleProgress.
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #3029 from davies/progress and squashes the following commits:
    
    95336d5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    fc49ac8 [Davies Liu] address commentse
    2e90f75 [Davies Liu] show multiple stages in same time
    0081bcc [Davies Liu] address comments
    38c42f1 [Davies Liu] fix tests
    ab87958 [Davies Liu] disable progress bar during tests
    30ac852 [Davies Liu] re-implement progress bar
    b3f34e5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    6fd30ff [Davies Liu] show progress bar if no task finished in 500ms
    e4e7344 [Davies Liu] refactor
    e1f524d [Davies Liu] revert unnecessary change
    a60477c [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    5cae3f2 [Davies Liu] fix style
    ea49fe0 [Davies Liu] address comments
    bc53d99 [Davies Liu] refactor
    e6bb189 [Davies Liu] fix logging in sparkshell
    7e7d4e7 [Davies Liu] address commments
    5df26bb [Davies Liu] fix style
    9e42208 [Davies Liu] show progress bar in console and title
    e34f38ff
    History
    [SPARK-4017] show progress bar in console
    Davies Liu authored
    The progress bar will look like this:
    
    ![1___spark_job__85_250_finished__4_are_running___java_](https://cloud.githubusercontent.com/assets/40902/4854813/a02f44ac-6099-11e4-9060-7c73a73151d6.png)
    
    In the right corner, the numbers are: finished tasks, running tasks, total tasks.
    
    After the stage has finished, it will disappear.
    
    The progress bar is only showed if logging level is WARN or higher (but progress in title is still showed), it can be turned off by spark.driver.showConsoleProgress.
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #3029 from davies/progress and squashes the following commits:
    
    95336d5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    fc49ac8 [Davies Liu] address commentse
    2e90f75 [Davies Liu] show multiple stages in same time
    0081bcc [Davies Liu] address comments
    38c42f1 [Davies Liu] fix tests
    ab87958 [Davies Liu] disable progress bar during tests
    30ac852 [Davies Liu] re-implement progress bar
    b3f34e5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    6fd30ff [Davies Liu] show progress bar if no task finished in 500ms
    e4e7344 [Davies Liu] refactor
    e1f524d [Davies Liu] revert unnecessary change
    a60477c [Davies Liu] Merge branch 'master' of github.com:apache/spark into progress
    5cae3f2 [Davies Liu] fix style
    ea49fe0 [Davies Liu] address comments
    bc53d99 [Davies Liu] refactor
    e6bb189 [Davies Liu] fix logging in sparkshell
    7e7d4e7 [Davies Liu] address commments
    5df26bb [Davies Liu] fix style
    9e42208 [Davies Liu] show progress bar in console and title