Skip to content
Snippets Groups Projects
  • Sandeep Singh's avatar
    de9c85cc
    [SPARK-15270] [SQL] Use SparkSession Builder to build a session with HiveSupport · de9c85cc
    Sandeep Singh authored
    ## What changes were proposed in this pull request?
    Before:
    Creating a hiveContext was failing
    ```python
    from pyspark.sql import HiveContext
    hc = HiveContext(sc)
    ```
    with
    ```
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "spark-2.0/python/pyspark/sql/context.py", line 458, in __init__
        sparkSession = SparkSession.withHiveSupport(sparkContext)
      File "spark-2.0/python/pyspark/sql/session.py", line 192, in withHiveSupport
        jsparkSession = sparkContext._jvm.SparkSession.withHiveSupport(sparkContext._jsc.sc())
      File "spark-2.0/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py", line 1048, in __getattr__
    py4j.protocol.Py4JError: org.apache.spark.sql.SparkSession.withHiveSupport does not exist in the JVM
    ```
    
    Now:
    ```python
    >>> from pyspark.sql import HiveContext
    >>> hc = HiveContext(sc)
    >>> hc.range(0, 100)
    DataFrame[id: bigint]
    >>> hc.range(0, 100).count()
    100
    ```
    ## How was this patch tested?
    Existing Tests, tested manually in python shell
    
    Author: Sandeep Singh <sandeep@techaddict.me>
    
    Closes #13056 from techaddict/SPARK-15270.
    de9c85cc
    History
    [SPARK-15270] [SQL] Use SparkSession Builder to build a session with HiveSupport
    Sandeep Singh authored
    ## What changes were proposed in this pull request?
    Before:
    Creating a hiveContext was failing
    ```python
    from pyspark.sql import HiveContext
    hc = HiveContext(sc)
    ```
    with
    ```
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "spark-2.0/python/pyspark/sql/context.py", line 458, in __init__
        sparkSession = SparkSession.withHiveSupport(sparkContext)
      File "spark-2.0/python/pyspark/sql/session.py", line 192, in withHiveSupport
        jsparkSession = sparkContext._jvm.SparkSession.withHiveSupport(sparkContext._jsc.sc())
      File "spark-2.0/python/lib/py4j-0.9.2-src.zip/py4j/java_gateway.py", line 1048, in __getattr__
    py4j.protocol.Py4JError: org.apache.spark.sql.SparkSession.withHiveSupport does not exist in the JVM
    ```
    
    Now:
    ```python
    >>> from pyspark.sql import HiveContext
    >>> hc = HiveContext(sc)
    >>> hc.range(0, 100)
    DataFrame[id: bigint]
    >>> hc.range(0, 100).count()
    100
    ```
    ## How was this patch tested?
    Existing Tests, tested manually in python shell
    
    Author: Sandeep Singh <sandeep@techaddict.me>
    
    Closes #13056 from techaddict/SPARK-15270.