Skip to content
  • Andrew Or's avatar
    9eb222c1
    [SPARK-7558] Demarcate tests in unit-tests.log · 9eb222c1
    Andrew Or authored
    Right now `unit-tests.log` are not of much value because we can't tell where the test boundaries are easily. This patch adds log statements before and after each test to outline the test boundaries, e.g.:
    
    ```
    ===== TEST OUTPUT FOR o.a.s.serializer.KryoSerializerSuite: 'kryo with parallelize for primitive arrays' =====
    
    15/05/27 12:36:39.596 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO SparkContext: Starting job: count at KryoSerializerSuite.scala:230
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Got job 3 (count at KryoSerializerSuite.scala:230) with 4 output partitions (allowLocal=false)
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 3(count at KryoSerializerSuite.scala:230)
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List()
    15/05/27 12:36:39.597 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List()
    15/05/27 12:36:39.597 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 3 (ParallelCollectionRDD[5] at parallelize at KryoSerializerSuite.scala:230), which has no missing parents
    
    ...
    
    15/05/27 12:36:39.624 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO DAGScheduler: Job 3 finished: count at KryoSerializerSuite.scala:230, took 0.028563 s
    15/05/27 12:36:39.625 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO KryoSerializerSuite:
    
    ***** FINISHED o.a.s.serializer.KryoSerializerSuite: 'kryo with parallelize for primitive arrays' *****
    
    ...
    ```
    
    Author: Andrew Or <andrew@databricks.com>
    
    Closes #6441 from andrewor14/demarcate-tests and squashes the following commits:
    
    879b060 [Andrew Or] Fix compile after rebase
    d622af7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    017c8ba [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    7790b6c [Andrew Or] Fix tests after logical merge conflict
    c7460c0 [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    c43ffc4 [Andrew Or] Fix tests?
    8882581 [Andrew Or] Fix tests
    ee22cda [Andrew Or] Fix log message
    fa9450e [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    12d1e1b [Andrew Or] Various whitespace changes (minor)
    69cbb24 [Andrew Or] Make all test suites extend SparkFunSuite instead of FunSuite
    bbce12e [Andrew Or] Fix manual things that cannot be covered through automation
    da0b12f [Andrew Or] Add core tests as dependencies in all modules
    f7d29ce [Andrew Or] Introduce base abstract class for all test suites
    9eb222c1
    [SPARK-7558] Demarcate tests in unit-tests.log
    Andrew Or authored
    Right now `unit-tests.log` are not of much value because we can't tell where the test boundaries are easily. This patch adds log statements before and after each test to outline the test boundaries, e.g.:
    
    ```
    ===== TEST OUTPUT FOR o.a.s.serializer.KryoSerializerSuite: 'kryo with parallelize for primitive arrays' =====
    
    15/05/27 12:36:39.596 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO SparkContext: Starting job: count at KryoSerializerSuite.scala:230
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Got job 3 (count at KryoSerializerSuite.scala:230) with 4 output partitions (allowLocal=false)
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Final stage: ResultStage 3(count at KryoSerializerSuite.scala:230)
    15/05/27 12:36:39.596 dag-scheduler-event-loop INFO DAGScheduler: Parents of final stage: List()
    15/05/27 12:36:39.597 dag-scheduler-event-loop INFO DAGScheduler: Missing parents: List()
    15/05/27 12:36:39.597 dag-scheduler-event-loop INFO DAGScheduler: Submitting ResultStage 3 (ParallelCollectionRDD[5] at parallelize at KryoSerializerSuite.scala:230), which has no missing parents
    
    ...
    
    15/05/27 12:36:39.624 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO DAGScheduler: Job 3 finished: count at KryoSerializerSuite.scala:230, took 0.028563 s
    15/05/27 12:36:39.625 pool-1-thread-1-ScalaTest-running-KryoSerializerSuite INFO KryoSerializerSuite:
    
    ***** FINISHED o.a.s.serializer.KryoSerializerSuite: 'kryo with parallelize for primitive arrays' *****
    
    ...
    ```
    
    Author: Andrew Or <andrew@databricks.com>
    
    Closes #6441 from andrewor14/demarcate-tests and squashes the following commits:
    
    879b060 [Andrew Or] Fix compile after rebase
    d622af7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    017c8ba [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    7790b6c [Andrew Or] Fix tests after logical merge conflict
    c7460c0 [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    c43ffc4 [Andrew Or] Fix tests?
    8882581 [Andrew Or] Fix tests
    ee22cda [Andrew Or] Fix log message
    fa9450e [Andrew Or] Merge branch 'master' of github.com:apache/spark into demarcate-tests
    12d1e1b [Andrew Or] Various whitespace changes (minor)
    69cbb24 [Andrew Or] Make all test suites extend SparkFunSuite instead of FunSuite
    bbce12e [Andrew Or] Fix manual things that cannot be covered through automation
    da0b12f [Andrew Or] Add core tests as dependencies in all modules
    f7d29ce [Andrew Or] Introduce base abstract class for all test suites
Loading