Skip to content
Snippets Groups Projects
Commit d6a972b3 authored by Kostas Sakellis's avatar Kostas Sakellis Committed by Michael Armbrust
Browse files

[SPARK-4774] [SQL] Makes HiveFromSpark more portable

HiveFromSpark read the kv1.txt file from SPARK_HOME/examples/src/main/resources/kv1.txt which assumed
you had a source tree checked out. Now we copy the kv1.txt file to a temporary file and delete it when
the jvm shuts down. This allows us to run this example outside of a spark source tree.

Author: Kostas Sakellis <kostas@cloudera.com>

Closes #3628 from ksakellis/kostas-spark-4774 and squashes the following commits:

6770f83 [Kostas Sakellis] [SPARK-4774] [SQL] Makes HiveFromSpark more portable
parent ab2abcb5
No related branches found
No related tags found
No related merge requests found
......@@ -17,6 +17,10 @@
package org.apache.spark.examples.sql.hive
import com.google.common.io.{ByteStreams, Files}
import java.io.File
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql._
import org.apache.spark.sql.hive.HiveContext
......@@ -24,10 +28,15 @@ import org.apache.spark.sql.hive.HiveContext
object HiveFromSpark {
case class Record(key: Int, value: String)
// Copy kv1.txt file from classpath to temporary directory
val kv1Stream = HiveFromSpark.getClass.getResourceAsStream("/kv1.txt")
val kv1File = File.createTempFile("kv1", "txt")
kv1File.deleteOnExit()
ByteStreams.copy(kv1Stream, Files.newOutputStreamSupplier(kv1File))
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("HiveFromSpark")
val sc = new SparkContext(sparkConf)
val path = s"${System.getenv("SPARK_HOME")}/examples/src/main/resources/kv1.txt"
// A hive context adds support for finding tables in the MetaStore and writing queries
// using HiveQL. Users who do not have an existing Hive deployment can still create a
......@@ -37,7 +46,7 @@ object HiveFromSpark {
import hiveContext._
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
sql(s"LOAD DATA LOCAL INPATH '$path' INTO TABLE src")
sql(s"LOAD DATA LOCAL INPATH '${kv1File.getAbsolutePath}' INTO TABLE src")
// Queries are expressed in HiveQL
println("Result of 'SELECT *': ")
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment