Skip to content
Snippets Groups Projects
  • Josh Rosen's avatar
    ee74498d
    [SPARK-8725][PROJECT-INFRA] Test modules in topologically-sorted order in dev/run-tests · ee74498d
    Josh Rosen authored
    This patch improves our `dev/run-tests` script to test modules in a topologically-sorted order based on modules' dependencies.  This will help to ensure that bugs in upstream projects are not misattributed to downstream projects because those projects' tests were the first ones to exhibit the failure
    
    Topological sorting is also useful for shortening the feedback loop when testing pull requests: if I make a change in SQL then the SQL tests should run before MLlib, not after.
    
    In addition, this patch also updates our test module definitions to split `sql` into `catalyst`, `sql`, and `hive` in order to allow more tests to be skipped when changing only `hive/` files.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #10885 from JoshRosen/SPARK-8725.
    ee74498d
    History
    [SPARK-8725][PROJECT-INFRA] Test modules in topologically-sorted order in dev/run-tests
    Josh Rosen authored
    This patch improves our `dev/run-tests` script to test modules in a topologically-sorted order based on modules' dependencies.  This will help to ensure that bugs in upstream projects are not misattributed to downstream projects because those projects' tests were the first ones to exhibit the failure
    
    Topological sorting is also useful for shortening the feedback loop when testing pull requests: if I make a change in SQL then the SQL tests should run before MLlib, not after.
    
    In addition, this patch also updates our test module definitions to split `sql` into `catalyst`, `sql`, and `hive` in order to allow more tests to be skipped when changing only `hive/` files.
    
    Author: Josh Rosen <joshrosen@databricks.com>
    
    Closes #10885 from JoshRosen/SPARK-8725.