Skip to content
Snippets Groups Projects
  • foxish's avatar
    7ab165b7
    [SPARK-22648][K8S] Spark on Kubernetes - Documentation · 7ab165b7
    foxish authored
    What changes were proposed in this pull request?
    
    This PR contains documentation on the usage of Kubernetes scheduler in Spark 2.3, and a shell script to make it easier to build docker images required to use the integration. The changes detailed here are covered by https://github.com/apache/spark/pull/19717 and https://github.com/apache/spark/pull/19468 which have merged already.
    
    How was this patch tested?
    The script has been in use for releases on our fork. Rest is documentation.
    
    cc rxin mateiz (shepherd)
    k8s-big-data SIG members & contributors: foxish ash211 mccheah liyinan926 erikerlandson ssuchter varunkatta kimoonkim tnachen ifilonenko
    reviewers: vanzin felixcheung jiangxb1987 mridulm
    
    TODO:
    - [x] Add dockerfiles directory to built distribution. (https://github.com/apache/spark/pull/20007)
    - [x] Change references to docker to instead say "container" (https://github.com/apache/spark/pull/19995)
    - [x] Update configuration table.
    - [x] Modify spark.kubernetes.allocation.batch.delay to take time instead of int (#20032)
    
    Author: foxish <ramanathana@google.com>
    
    Closes #19946 from foxish/update-k8s-docs.
    7ab165b7
    History
    [SPARK-22648][K8S] Spark on Kubernetes - Documentation
    foxish authored
    What changes were proposed in this pull request?
    
    This PR contains documentation on the usage of Kubernetes scheduler in Spark 2.3, and a shell script to make it easier to build docker images required to use the integration. The changes detailed here are covered by https://github.com/apache/spark/pull/19717 and https://github.com/apache/spark/pull/19468 which have merged already.
    
    How was this patch tested?
    The script has been in use for releases on our fork. Rest is documentation.
    
    cc rxin mateiz (shepherd)
    k8s-big-data SIG members & contributors: foxish ash211 mccheah liyinan926 erikerlandson ssuchter varunkatta kimoonkim tnachen ifilonenko
    reviewers: vanzin felixcheung jiangxb1987 mridulm
    
    TODO:
    - [x] Add dockerfiles directory to built distribution. (https://github.com/apache/spark/pull/20007)
    - [x] Change references to docker to instead say "container" (https://github.com/apache/spark/pull/19995)
    - [x] Update configuration table.
    - [x] Modify spark.kubernetes.allocation.batch.delay to take time instead of int (#20032)
    
    Author: foxish <ramanathana@google.com>
    
    Closes #19946 from foxish/update-k8s-docs.