Skip to content
Snippets Groups Projects
  • Philipp Hoffmann's avatar
    0869b3a5
    [SPARK-15271][MESOS] Allow force pulling executor docker images · 0869b3a5
    Philipp Hoffmann authored
    ## What changes were proposed in this pull request?
    
    (Please fill in changes proposed in this fix)
    
    ## How was this patch tested?
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    Mesos agents by default will not pull docker images which are cached
    locally already. In order to run Spark executors from mutable tags like
    `:latest` this commit introduces a Spark setting
    (`spark.mesos.executor.docker.forcePullImage`). Setting this flag to
    true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous
    implementation and Mesos' default
    behaviour).
    
    Author: Philipp Hoffmann <mail@philipphoffmann.de>
    
    Closes #14348 from philipphoffmann/force-pull-image.
    0869b3a5
    History
    [SPARK-15271][MESOS] Allow force pulling executor docker images
    Philipp Hoffmann authored
    ## What changes were proposed in this pull request?
    
    (Please fill in changes proposed in this fix)
    
    ## How was this patch tested?
    
    (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
    
    (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
    
    Mesos agents by default will not pull docker images which are cached
    locally already. In order to run Spark executors from mutable tags like
    `:latest` this commit introduces a Spark setting
    (`spark.mesos.executor.docker.forcePullImage`). Setting this flag to
    true will tell the Mesos agent to force pull the docker image (default is `false` which is consistent with the previous
    implementation and Mesos' default
    behaviour).
    
    Author: Philipp Hoffmann <mail@philipphoffmann.de>
    
    Closes #14348 from philipphoffmann/force-pull-image.