Skip to content
Snippets Groups Projects
Commit 4633a87b authored by Pei-Lun Lee's avatar Pei-Lun Lee Committed by Cheng Lian
Browse files

[SPARK-6330] [SQL] Add a test case for SPARK-6330

When getting file statuses, create file system from each path instead of a single one from hadoop configuration.

Author: Pei-Lun Lee <pllee@appier.com>

Closes #5039 from ypcat/spark-6351 and squashes the following commits:

a19a3fe [Pei-Lun Lee] [SPARK-6330] [SQL] fix test
506f5a0 [Pei-Lun Lee] [SPARK-6351] [SQL] fix test
fa2290e [Pei-Lun Lee] [SPARK-6330] [SQL] Rename test case and add comment
606c967 [Pei-Lun Lee] Merge branch 'master' of https://github.com/apache/spark into spark-6351
896e80a [Pei-Lun Lee] [SPARK-6351] [SQL] Add test case
2ae0916 [Pei-Lun Lee] [SPARK-6351] [SQL] ParquetRelation2 supporting multiple file systems
parent c94d0626
No related branches found
No related tags found
No related merge requests found
......@@ -329,6 +329,7 @@ class ParquetIOSuiteBase extends QueryTest with ParquetTest {
checkAnswer(parquetFile(file), (data ++ newData).map(Row.fromTuple))
}
}
}
class ParquetDataSourceOnIOSuite extends ParquetIOSuiteBase with BeforeAndAfterAll {
......@@ -341,6 +342,18 @@ class ParquetDataSourceOnIOSuite extends ParquetIOSuiteBase with BeforeAndAfterA
override protected def afterAll(): Unit = {
sqlContext.setConf(SQLConf.PARQUET_USE_DATA_SOURCE_API, originalConf.toString)
}
test("SPARK-6330 regression test") {
// In 1.3.0, save to fs other than file: without configuring core-site.xml would get:
// IllegalArgumentException: Wrong FS: hdfs://..., expected: file:///
intercept[java.io.FileNotFoundException] {
sqlContext.parquetFile("file:///nonexistent")
}
val errorMessage = intercept[Throwable] {
sqlContext.parquetFile("hdfs://nonexistent")
}.toString
assert(errorMessage.contains("UnknownHostException"))
}
}
class ParquetDataSourceOffIOSuite extends ParquetIOSuiteBase with BeforeAndAfterAll {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment