-
- Downloads
[SPARK-16671][CORE][SQL] Consolidate code to do variable substitution.
Both core and sql have slightly different code that does variable substitution of config values. This change refactors that code and encapsulates the logic of reading config values and expading variables in a new helper class, which can be configured so that both core and sql can use it without losing existing functionality, and allows for easier testing and makes it easier to add more features in the future. Tested with existing and new unit tests, and by running spark-shell with some configs referencing variables and making sure it behaved as expected. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #14468 from vanzin/SPARK-16671.
Showing
- core/src/main/scala/org/apache/spark/SparkConf.scala 7 additions, 2 deletionscore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala 14 additions, 78 deletions.../scala/org/apache/spark/internal/config/ConfigEntry.scala
- core/src/main/scala/org/apache/spark/internal/config/ConfigProvider.scala 74 additions, 0 deletions...ala/org/apache/spark/internal/config/ConfigProvider.scala
- core/src/main/scala/org/apache/spark/internal/config/ConfigReader.scala 106 additions, 0 deletions...scala/org/apache/spark/internal/config/ConfigReader.scala
- core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala 31 additions, 47 deletions...a/org/apache/spark/internal/config/ConfigEntrySuite.scala
- core/src/test/scala/org/apache/spark/internal/config/ConfigReaderSuite.scala 62 additions, 0 deletions.../org/apache/spark/internal/config/ConfigReaderSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 6 additions, 3 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/VariableSubstitution.scala 12 additions, 80 deletions.../org/apache/spark/sql/internal/VariableSubstitution.scala
- sql/core/src/test/scala/org/apache/spark/sql/internal/VariableSubstitutionSuite.scala 0 additions, 18 deletions...apache/spark/sql/internal/VariableSubstitutionSuite.scala
Loading
Please register or sign in to comment