Skip to content
Snippets Groups Projects
  • Xiangrui Meng's avatar
    7eba0fbe
    [Spark-4509] Revert EC2 tag-based cluster membership patch · 7eba0fbe
    Xiangrui Meng authored
    This PR reverts changes related to tag-based cluster membership. As discussed in SPARK-3332, we didn't figure out a safe strategy to use tags to determine cluster membership, because tagging is not atomic. The following changes are reverted:
    
    SPARK-2333: 94053a7b
    SPARK-3213: 7faf755a
    SPARK-3608: 78d4220f.
    
    I tested launch, login, and destroy. It is easy to check the diff by comparing it to Josh's patch for branch-1.1:
    
    https://github.com/apache/spark/pull/2225/files
    
    JoshRosen I sent the PR to master. It might be easier for us to keep master and branch-1.2 the same at this time. We can always re-apply the patch once we figure out a stable solution.
    
    Author: Xiangrui Meng <meng@databricks.com>
    
    Closes #3453 from mengxr/SPARK-4509 and squashes the following commits:
    
    f0b708b [Xiangrui Meng] revert 94053a7b
    4298ea5 [Xiangrui Meng] revert 7faf755a
    35963a1 [Xiangrui Meng] Revert "SPARK-3608 Break if the instance tag naming succeeds"
    7eba0fbe
    History
    [Spark-4509] Revert EC2 tag-based cluster membership patch
    Xiangrui Meng authored
    This PR reverts changes related to tag-based cluster membership. As discussed in SPARK-3332, we didn't figure out a safe strategy to use tags to determine cluster membership, because tagging is not atomic. The following changes are reverted:
    
    SPARK-2333: 94053a7b
    SPARK-3213: 7faf755a
    SPARK-3608: 78d4220f.
    
    I tested launch, login, and destroy. It is easy to check the diff by comparing it to Josh's patch for branch-1.1:
    
    https://github.com/apache/spark/pull/2225/files
    
    JoshRosen I sent the PR to master. It might be easier for us to keep master and branch-1.2 the same at this time. We can always re-apply the patch once we figure out a stable solution.
    
    Author: Xiangrui Meng <meng@databricks.com>
    
    Closes #3453 from mengxr/SPARK-4509 and squashes the following commits:
    
    f0b708b [Xiangrui Meng] revert 94053a7b
    4298ea5 [Xiangrui Meng] revert 7faf755a
    35963a1 [Xiangrui Meng] Revert "SPARK-3608 Break if the instance tag naming succeeds"