Skip to content
  • Kousuke Saruta's avatar
    7fe0f3f2
    [SPARK-3468] [WEBUI] Timeline-View feature · 7fe0f3f2
    Kousuke Saruta authored
    I sometimes trouble-shoot and analyse the cause of long time spending job.
    
    At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.
    
    Another case, I find executors which spends long time for running a task and analyse the details of a task.
    
    In such situation, I think it's helpful to visualize timeline  for each application, job, task and the details of proportion of activity for each task.
    
    I added 3 timeline features to existing Web UI.
    
    [Application Timeline View]
    This view shows following things.
    
    * When each executor was added/removed and the reason why it's removed.
    * When each job  was started/finished.
    * Status of each job.
    
    ![screenshot from 2015-04-01 16 49 25](https://cloud.githubusercontent.com/assets/4736016/6936886/e35fd582-d891-11e4-980d-8de13f50e442.png)
    
    [Stage Timeline View]
    Similar to Application Timeline View, this view shows following things.
    
    * When each executor was added/removed and the reason why it's removed.
    * When each job was started/finished.
    * Status of each stage.
    
    ![screenshot from 2015-04-01 16 50 59](https://cloud.githubusercontent.com/assets/4736016/6936900/0dca6526-d892-11e4-84a8-efd9037af444.png)
    
    [Task Assignment Timeline View]
    This view shows following things.
    
    * When each task started/finished
    * How long each task spent and the proportion.
    * Status of each task.
    * Where each task ran on.
    
    ![screenshot from 2015-04-01 16 51 54](https://cloud.githubusercontent.com/assets/4736016/6936910/20fd5acc-d892-11e4-9018-80e463881fc2.png)
    
    All the view above is zoomable by mouse wheel action and scrollable by drag action.
    
    Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
    
    Closes #2342 from sarutak/timeline-viewer-feature and squashes the following commits:
    
    11fe67d [Kousuke Saruta] Fixed conflict
    79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    a91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
    ef34a5b [Kousuke Saruta] Implement tooltip using bootstrap
    b09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css
    d3c63c8 [Kousuke Saruta] Fixed a little bit bugs
    a36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
    28714b6 [Kousuke Saruta] Fixed highlight issue
    0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks
    8110acf [Kousuke Saruta] Added scroll limit to Job timeline
    974a64a [Kousuke Saruta] Removed unused function
    ee7a7f0 [Kousuke Saruta] Refactored
    6a91872 [Kousuke Saruta] Temporary commit
    6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click
    8f88222 [Kousuke Saruta] Added job/stage description
    aeed4b1 [Kousuke Saruta] Removed stage timeline
    fc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
    999ccd4 [Kousuke Saruta] Improved scalability
    0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
    52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    dec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    fcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    dab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    09cce97 [Kousuke Saruta] Cleanuped
    16f82cf [Kousuke Saruta] Cleanuped
    9fb522e [Kousuke Saruta] Cleanuped
    d05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    e85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala
    a76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css
    5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude
    082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages
    7fe0f3f2
    [SPARK-3468] [WEBUI] Timeline-View feature
    Kousuke Saruta authored
    I sometimes trouble-shoot and analyse the cause of long time spending job.
    
    At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task.
    
    Another case, I find executors which spends long time for running a task and analyse the details of a task.
    
    In such situation, I think it's helpful to visualize timeline  for each application, job, task and the details of proportion of activity for each task.
    
    I added 3 timeline features to existing Web UI.
    
    [Application Timeline View]
    This view shows following things.
    
    * When each executor was added/removed and the reason why it's removed.
    * When each job  was started/finished.
    * Status of each job.
    
    ![screenshot from 2015-04-01 16 49 25](https://cloud.githubusercontent.com/assets/4736016/6936886/e35fd582-d891-11e4-980d-8de13f50e442.png)
    
    [Stage Timeline View]
    Similar to Application Timeline View, this view shows following things.
    
    * When each executor was added/removed and the reason why it's removed.
    * When each job was started/finished.
    * Status of each stage.
    
    ![screenshot from 2015-04-01 16 50 59](https://cloud.githubusercontent.com/assets/4736016/6936900/0dca6526-d892-11e4-84a8-efd9037af444.png)
    
    [Task Assignment Timeline View]
    This view shows following things.
    
    * When each task started/finished
    * How long each task spent and the proportion.
    * Status of each task.
    * Where each task ran on.
    
    ![screenshot from 2015-04-01 16 51 54](https://cloud.githubusercontent.com/assets/4736016/6936910/20fd5acc-d892-11e4-9018-80e463881fc2.png)
    
    All the view above is zoomable by mouse wheel action and scrollable by drag action.
    
    Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp>
    
    Closes #2342 from sarutak/timeline-viewer-feature and squashes the following commits:
    
    11fe67d [Kousuke Saruta] Fixed conflict
    79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    a91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
    ef34a5b [Kousuke Saruta] Implement tooltip using bootstrap
    b09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css
    d3c63c8 [Kousuke Saruta] Fixed a little bit bugs
    a36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature
    28714b6 [Kousuke Saruta] Fixed highlight issue
    0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks
    8110acf [Kousuke Saruta] Added scroll limit to Job timeline
    974a64a [Kousuke Saruta] Removed unused function
    ee7a7f0 [Kousuke Saruta] Refactored
    6a91872 [Kousuke Saruta] Temporary commit
    6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click
    8f88222 [Kousuke Saruta] Added job/stage description
    aeed4b1 [Kousuke Saruta] Removed stage timeline
    fc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
    999ccd4 [Kousuke Saruta] Improved scalability
    0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature
    52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    dec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    fcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    dab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    09cce97 [Kousuke Saruta] Cleanuped
    16f82cf [Kousuke Saruta] Cleanuped
    9fb522e [Kousuke Saruta] Cleanuped
    d05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature
    e85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala
    a76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css
    5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude
    082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages
Loading