Skip to content
Snippets Groups Projects
  • Nicholas Chammas's avatar
    5912ca67
    [SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states · 5912ca67
    Nicholas Chammas authored
    Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets `spark-ec2` explicitly wait for a cluster to reach a desired state.
    
    This is useful in a couple of situations:
    * The cluster is launching and you want to wait until SSH is available before installing stuff.
    * The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.
    
    This patch removes the need for the `--wait` option and removes some of the time-based retry logic that was being used.
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    
    Closes #2339 from nchammas/spark-ec2-wait-properly and squashes the following commits:
    
    43a69f0 [Nicholas Chammas] short-circuit SSH check; linear backoff
    9a9e035 [Nicholas Chammas] remove extraneous comment
    26c5ed0 [Nicholas Chammas] replace print with write()
    bb67c06 [Nicholas Chammas] deprecate wait option; remove dead code
    7969265 [Nicholas Chammas] fix long line (PEP 8)
    126e4cf [Nicholas Chammas] wait for specific cluster states
    5912ca67
    History
    [SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states
    Nicholas Chammas authored
    Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets `spark-ec2` explicitly wait for a cluster to reach a desired state.
    
    This is useful in a couple of situations:
    * The cluster is launching and you want to wait until SSH is available before installing stuff.
    * The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.
    
    This patch removes the need for the `--wait` option and removes some of the time-based retry logic that was being used.
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    
    Closes #2339 from nchammas/spark-ec2-wait-properly and squashes the following commits:
    
    43a69f0 [Nicholas Chammas] short-circuit SSH check; linear backoff
    9a9e035 [Nicholas Chammas] remove extraneous comment
    26c5ed0 [Nicholas Chammas] replace print with write()
    bb67c06 [Nicholas Chammas] deprecate wait option; remove dead code
    7969265 [Nicholas Chammas] fix long line (PEP 8)
    126e4cf [Nicholas Chammas] wait for specific cluster states