-
- Downloads
[SPARK-20641][CORE] Add key-value store abstraction and LevelDB implementation.
This change adds an abstraction and LevelDB implementation for a key-value store that will be used to store UI and SHS data. The interface is described in KVStore.java (see javadoc). Specifics of the LevelDB implementation are discussed in the javadocs of both LevelDB.java and LevelDBTypeInfo.java. Included also are a few small benchmarks just to get some idea of latency. Because they're too slow for regular unit test runs, they're disabled by default. Tested with the included unit tests, and also as part of the overall feature implementation (including running SHS with hundreds of apps). Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #17902 from vanzin/shs-ng/M1.
Showing
- common/kvstore/pom.xml 101 additions, 0 deletionscommon/kvstore/pom.xml
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVIndex.java 82 additions, 0 deletions...store/src/main/java/org/apache/spark/kvstore/KVIndex.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVStore.java 129 additions, 0 deletions...store/src/main/java/org/apache/spark/kvstore/KVStore.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreIterator.java 47 additions, 0 deletions...c/main/java/org/apache/spark/kvstore/KVStoreIterator.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreSerializer.java 86 additions, 0 deletions...main/java/org/apache/spark/kvstore/KVStoreSerializer.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVStoreView.java 126 additions, 0 deletions...e/src/main/java/org/apache/spark/kvstore/KVStoreView.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/KVTypeInfo.java 156 additions, 0 deletions...re/src/main/java/org/apache/spark/kvstore/KVTypeInfo.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDB.java 308 additions, 0 deletions...store/src/main/java/org/apache/spark/kvstore/LevelDB.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDBIterator.java 278 additions, 0 deletions...c/main/java/org/apache/spark/kvstore/LevelDBIterator.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/LevelDBTypeInfo.java 516 additions, 0 deletions...c/main/java/org/apache/spark/kvstore/LevelDBTypeInfo.java
- common/kvstore/src/main/java/org/apache/spark/kvstore/UnsupportedStoreVersionException.java 27 additions, 0 deletions...pache/spark/kvstore/UnsupportedStoreVersionException.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/CustomType1.java 63 additions, 0 deletions...e/src/test/java/org/apache/spark/kvstore/CustomType1.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/DBIteratorSuite.java 506 additions, 0 deletions...c/test/java/org/apache/spark/kvstore/DBIteratorSuite.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBBenchmark.java 280 additions, 0 deletions.../test/java/org/apache/spark/kvstore/LevelDBBenchmark.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBIteratorSuite.java 48 additions, 0 deletions...t/java/org/apache/spark/kvstore/LevelDBIteratorSuite.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBSuite.java 312 additions, 0 deletions.../src/test/java/org/apache/spark/kvstore/LevelDBSuite.java
- common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBTypeInfoSuite.java 207 additions, 0 deletions...t/java/org/apache/spark/kvstore/LevelDBTypeInfoSuite.java
- common/kvstore/src/test/resources/log4j.properties 27 additions, 0 deletionscommon/kvstore/src/test/resources/log4j.properties
- pom.xml 11 additions, 0 deletionspom.xml
- project/SparkBuild.scala 3 additions, 3 deletionsproject/SparkBuild.scala
Loading
Please register or sign in to comment