Skip to content
Snippets Groups Projects
  • Nicholas Chammas's avatar
    5d16d5bb
    [SPARK-2470] PEP8 fixes to PySpark · 5d16d5bb
    Nicholas Chammas authored
    This pull request aims to resolve all outstanding PEP8 violations in PySpark.
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    Author: nchammas <nicholas.chammas@gmail.com>
    
    Closes #1505 from nchammas/master and squashes the following commits:
    
    98171af [Nicholas Chammas] [SPARK-2470] revert PEP 8 fixes to cloudpickle
    cba7768 [Nicholas Chammas] [SPARK-2470] wrap expression list in parentheses
    e178dbe [Nicholas Chammas] [SPARK-2470] style - change position of line break
    9127d2b [Nicholas Chammas] [SPARK-2470] wrap expression lists in parentheses
    22132a4 [Nicholas Chammas] [SPARK-2470] wrap conditionals in parentheses
    24639bc [Nicholas Chammas] [SPARK-2470] fix whitespace for doctest
    7d557b7 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to tests.py
    8f8e4c0 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to storagelevel.py
    b3b96cf [Nicholas Chammas] [SPARK-2470] PEP8 fixes to statcounter.py
    d644477 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to worker.py
    aa3a7b6 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to sql.py
    1916859 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to shell.py
    95d1d95 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to serializers.py
    a0fec2e [Nicholas Chammas] [SPARK-2470] PEP8 fixes to mllib
    c85e1e5 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to join.py
    d14f2f1 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to __init__.py
    81fcb20 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to resultiterable.py
    1bde265 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to java_gateway.py
    7fc849c [Nicholas Chammas] [SPARK-2470] PEP8 fixes to daemon.py
    ca2d28b [Nicholas Chammas] [SPARK-2470] PEP8 fixes to context.py
    f4e0039 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to conf.py
    a6d5e4b [Nicholas Chammas] [SPARK-2470] PEP8 fixes to cloudpickle.py
    f0a7ebf [Nicholas Chammas] [SPARK-2470] PEP8 fixes to rddsampler.py
    4dd148f [nchammas] Merge pull request #5 from apache/master
    f7e4581 [Nicholas Chammas] unrelated pep8 fix
    a36eed0 [Nicholas Chammas] name ec2 instances and security groups consistently
    de7292a [nchammas] Merge pull request #4 from apache/master
    2e4fe00 [nchammas] Merge pull request #3 from apache/master
    89fde08 [nchammas] Merge pull request #2 from apache/master
    69f6e22 [Nicholas Chammas] PEP8 fixes
    2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
    6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
    69da6cf [nchammas] Merge pull request #1 from apache/master
    5d16d5bb
    History
    [SPARK-2470] PEP8 fixes to PySpark
    Nicholas Chammas authored
    This pull request aims to resolve all outstanding PEP8 violations in PySpark.
    
    Author: Nicholas Chammas <nicholas.chammas@gmail.com>
    Author: nchammas <nicholas.chammas@gmail.com>
    
    Closes #1505 from nchammas/master and squashes the following commits:
    
    98171af [Nicholas Chammas] [SPARK-2470] revert PEP 8 fixes to cloudpickle
    cba7768 [Nicholas Chammas] [SPARK-2470] wrap expression list in parentheses
    e178dbe [Nicholas Chammas] [SPARK-2470] style - change position of line break
    9127d2b [Nicholas Chammas] [SPARK-2470] wrap expression lists in parentheses
    22132a4 [Nicholas Chammas] [SPARK-2470] wrap conditionals in parentheses
    24639bc [Nicholas Chammas] [SPARK-2470] fix whitespace for doctest
    7d557b7 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to tests.py
    8f8e4c0 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to storagelevel.py
    b3b96cf [Nicholas Chammas] [SPARK-2470] PEP8 fixes to statcounter.py
    d644477 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to worker.py
    aa3a7b6 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to sql.py
    1916859 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to shell.py
    95d1d95 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to serializers.py
    a0fec2e [Nicholas Chammas] [SPARK-2470] PEP8 fixes to mllib
    c85e1e5 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to join.py
    d14f2f1 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to __init__.py
    81fcb20 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to resultiterable.py
    1bde265 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to java_gateway.py
    7fc849c [Nicholas Chammas] [SPARK-2470] PEP8 fixes to daemon.py
    ca2d28b [Nicholas Chammas] [SPARK-2470] PEP8 fixes to context.py
    f4e0039 [Nicholas Chammas] [SPARK-2470] PEP8 fixes to conf.py
    a6d5e4b [Nicholas Chammas] [SPARK-2470] PEP8 fixes to cloudpickle.py
    f0a7ebf [Nicholas Chammas] [SPARK-2470] PEP8 fixes to rddsampler.py
    4dd148f [nchammas] Merge pull request #5 from apache/master
    f7e4581 [Nicholas Chammas] unrelated pep8 fix
    a36eed0 [Nicholas Chammas] name ec2 instances and security groups consistently
    de7292a [nchammas] Merge pull request #4 from apache/master
    2e4fe00 [nchammas] Merge pull request #3 from apache/master
    89fde08 [nchammas] Merge pull request #2 from apache/master
    69f6e22 [Nicholas Chammas] PEP8 fixes
    2627247 [Nicholas Chammas] broke up lines before they hit 100 chars
    6544b7e [Nicholas Chammas] [SPARK-2065] give launched instances names
    69da6cf [nchammas] Merge pull request #1 from apache/master
__init__.py 2.39 KiB
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

"""
PySpark is the Python API for Spark.

Public classes:

  - L{SparkContext<pyspark.context.SparkContext>}
      Main entry point for Spark functionality.
  - L{RDD<pyspark.rdd.RDD>}
      A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
  - L{Broadcast<pyspark.broadcast.Broadcast>}
      A broadcast variable that gets reused across tasks.
  - L{Accumulator<pyspark.accumulators.Accumulator>}
      An "add-only" shared variable that tasks can only add values to.
  - L{SparkConf<pyspark.conf.SparkConf>}
      For configuring Spark.
  - L{SparkFiles<pyspark.files.SparkFiles>}
      Access files shipped with jobs.
  - L{StorageLevel<pyspark.storagelevel.StorageLevel>}
      Finer-grained cache persistence levels.

Spark SQL:
  - L{SQLContext<pyspark.sql.SQLContext>}
      Main entry point for SQL functionality.
  - L{SchemaRDD<pyspark.sql.SchemaRDD>}
      A Resilient Distributed Dataset (RDD) with Schema information for the data contained. In
      addition to normal RDD operations, SchemaRDDs also support SQL.
  - L{Row<pyspark.sql.Row>}
      A Row of data returned by a Spark SQL query.

Hive:
  - L{HiveContext<pyspark.context.HiveContext>}
      Main entry point for accessing data stored in Apache Hive..
"""

from pyspark.conf import SparkConf
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
from pyspark.rdd import RDD
from pyspark.sql import SchemaRDD
from pyspark.sql import Row
from pyspark.files import SparkFiles
from pyspark.storagelevel import StorageLevel


__all__ = ["SparkConf", "SparkContext", "SQLContext", "RDD", "SchemaRDD",
           "SparkFiles", "StorageLevel", "Row"]