-
- Downloads
[SPARK-17480][SQL][FOLLOWUP] Fix more instances which calls List.length/size which is O(n)
## What changes were proposed in this pull request? This PR fixes all the instances which was fixed in the previous PR. To make sure, I manually debugged and also checked the Scala source. `length` in [LinearSeqOptimized.scala#L49-L57](https://github.com/scala/scala/blob/2.11.x/src/library/scala/collection/LinearSeqOptimized.scala#L49-L57) is O(n). Also, `size` calls `length` via [SeqLike.scala#L106](https://github.com/scala/scala/blob/2.11.x/src/library/scala/collection/SeqLike.scala#L106). For debugging, I have created these as below: ```scala ArrayBuffer(1, 2, 3) Array(1, 2, 3) List(1, 2, 3) Seq(1, 2, 3) ``` and then called `size` and `length` for each to debug. ## How was this patch tested? I ran the bash as below on Mac ```bash find . -name *.scala -type f -exec grep -il "while (.*\\.length)" {} \; | grep "src/main" find . -name *.scala -type f -exec grep -il "while (.*\\.size)" {} \; | grep "src/main" ``` and then checked each. Author: hyukjinkwon <gurwls223@gmail.com> Closes #15093 from HyukjinKwon/SPARK-17480-followup.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 9 additions, 19 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala 2 additions, 1 deletion...ark/sql/catalyst/expressions/conditionalExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala 2 additions, 1 deletion.../org/apache/spark/sql/catalyst/expressions/ordering.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/QuantileSummaries.scala 5 additions, 5 deletions...rg/apache/spark/sql/catalyst/util/QuantileSummaries.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala 1 addition, 1 deletion...ache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala 4 additions, 2 deletions...main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/sql/hive/TableReader.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala 2 additions, 1 deletion...e/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 4 additions, 2 deletions...n/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
Loading
Please register or sign in to comment