Skip to content
Snippets Groups Projects
  1. Oct 19, 2016
  2. Oct 18, 2016
    • Reynold Xin's avatar
      Revert "[SPARK-17985][CORE] Bump commons-lang3 version to 3.5." · cd662bc7
      Reynold Xin authored
      This reverts commit bfe7885a.
      
      The commit caused build failures on Hadoop 2.2 profile:
      
      ```
      [error] /scratch/rxin/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1489: value read is not a member of object org.apache.commons.io.IOUtils
      [error]       var numBytes = IOUtils.read(gzInputStream, buf)
      [error]                              ^
      [error] /scratch/rxin/spark/core/src/main/scala/org/apache/spark/util/Utils.scala:1492: value read is not a member of object org.apache.commons.io.IOUtils
      [error]         numBytes = IOUtils.read(gzInputStream, buf)
      [error]                            ^
      ```
      cd662bc7
    • Takuya UESHIN's avatar
      [SPARK-17985][CORE] Bump commons-lang3 version to 3.5. · bfe7885a
      Takuya UESHIN authored
      ## What changes were proposed in this pull request?
      
      `SerializationUtils.clone()` of commons-lang3 (<3.5) has a bug that breaks thread safety, which gets stack sometimes caused by race condition of initializing hash map.
      See https://issues.apache.org/jira/browse/LANG-1251.
      
      ## How was this patch tested?
      
      Existing tests.
      
      Author: Takuya UESHIN <ueshin@happy-camper.st>
      
      Closes #15525 from ueshin/issues/SPARK-17985.
      bfe7885a
  3. Aug 13, 2016
    • Jagadeesan's avatar
      [SPARK-12370][DOCUMENTATION] Documentation should link to examples … · e46cb78b
      Jagadeesan authored
      ## What changes were proposed in this pull request?
      
      When documentation is built is should reference examples from the same build. There are times when the docs have links that point to files in the GitHub head which may not be valid on the current release. Changed that in URLs to make them point to the right tag in git using ```SPARK_VERSION_SHORT```
      
      …from its own release version] [Streaming programming guide]
      
      Author: Jagadeesan <as2@us.ibm.com>
      
      Closes #14596 from jagadeesanas2/SPARK-12370.
      e46cb78b
  4. Feb 22, 2016
  5. Jan 18, 2016
  6. Sep 05, 2015
  7. Jul 01, 2015
    • zsxwing's avatar
      [SPARK-8378] [STREAMING] Add the Python API for Flume · 75b9fe4c
      zsxwing authored
      Author: zsxwing <zsxwing@gmail.com>
      
      Closes #6830 from zsxwing/flume-python and squashes the following commits:
      
      78dfdac [zsxwing] Fix the compile error in the test code
      f1bf3c0 [zsxwing] Address TD's comments
      0449723 [zsxwing] Add sbt goal streaming-flume-assembly/assembly
      e93736b [zsxwing] Fix the test case for determine_modules_to_test
      9d5821e [zsxwing] Fix pyspark_core dependencies
      f9ee681 [zsxwing] Merge branch 'master' into flume-python
      7a55837 [zsxwing] Add streaming_flume_assembly to run-tests.py
      b96b0de [zsxwing] Merge branch 'master' into flume-python
      ce85e83 [zsxwing] Fix incompatible issues for Python 3
      01cbb3d [zsxwing] Add import sys
      152364c [zsxwing] Fix the issue that StringIO doesn't work in Python 3
      14ba0ff [zsxwing] Add flume-assembly for sbt building
      b8d5551 [zsxwing] Merge branch 'master' into flume-python
      4762c34 [zsxwing] Fix the doc
      0336579 [zsxwing] Refactor Flume unit tests and also add tests for Python API
      9f33873 [zsxwing] Add the Python API for Flume
      75b9fe4c
  8. Jun 18, 2015
    • zsxwing's avatar
      [SPARK-8376] [DOCS] Add common lang3 to the Spark Flume Sink doc · 24e53793
      zsxwing authored
      Commons Lang 3 has been added as one of the dependencies of Spark Flume Sink since #5703. This PR updates the doc for it.
      
      Author: zsxwing <zsxwing@gmail.com>
      
      Closes #6829 from zsxwing/flume-sink-dep and squashes the following commits:
      
      f8617f0 [zsxwing] Add common lang3 to the Spark Flume Sink doc
      24e53793
  9. Mar 11, 2015
    • Tathagata Das's avatar
      [SPARK-6128][Streaming][Documentation] Updates to Spark Streaming Programming Guide · cd3b68d9
      Tathagata Das authored
      Updates to the documentation are as follows:
      
      - Added information on Kafka Direct API and Kafka Python API
      - Added joins to the main streaming guide
      - Improved details on the fault-tolerance semantics
      
      Generated docs located here
      http://people.apache.org/~tdas/spark-1.3.0-temp-docs/streaming-programming-guide.html#fault-tolerance-semantics
      
      More things to add:
      - Configuration for Kafka receive rate
      - May be add concurrentJobs
      
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      
      Closes #4956 from tdas/streaming-guide-update-1.3 and squashes the following commits:
      
      819408c [Tathagata Das] Minor fixes.
      debe484 [Tathagata Das] Added DataFrames and MLlib
      380cf8d [Tathagata Das] Fix link
      04167a6 [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into streaming-guide-update-1.3
      0b77486 [Tathagata Das] Updates based on Josh's comments.
      86c4c2a [Tathagata Das] Updated streaming guides
      82de92a [Tathagata Das] Add Kafka to Python api docs
      cd3b68d9
  10. Feb 16, 2015
  11. Dec 11, 2014
    • Tathagata Das's avatar
      [SPARK-4806] Streaming doc update for 1.2 · b004150a
      Tathagata Das authored
      Important updates to the streaming programming guide
      - Make the fault-tolerance properties easier to understand, with information about write ahead logs
      - Update the information about deploying the spark streaming app with information about Driver HA
      - Update Receiver guide to discuss reliable vs unreliable receivers.
      
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Author: Josh Rosen <joshrosen@databricks.com>
      Author: Josh Rosen <rosenville@gmail.com>
      
      Closes #3653 from tdas/streaming-doc-update-1.2 and squashes the following commits:
      
      f53154a [Tathagata Das] Addressed Josh's comments.
      ce299e4 [Tathagata Das] Minor update.
      ca19078 [Tathagata Das] Minor change
      f746951 [Tathagata Das] Mentioned performance problem with WAL
      7787209 [Tathagata Das] Merge branch 'streaming-doc-update-1.2' of github.com:tdas/spark into streaming-doc-update-1.2
      2184729 [Tathagata Das] Updated Kafka and Flume guides with reliability information.
      2f3178c [Tathagata Das] Added more information about writing reliable receivers in the custom receiver guide.
      91aa5aa [Tathagata Das] Improved API Docs menu
      5707581 [Tathagata Das] Added Pythn API badge
      b9c8c24 [Tathagata Das] Merge pull request #26 from JoshRosen/streaming-programming-guide
      b8c8382 [Josh Rosen] minor fixes
      a4ef126 [Josh Rosen] Restructure parts of the fault-tolerance section to read a bit nicer when skipping over the headings
      65f66cd [Josh Rosen] Fix broken link to fault-tolerance semantics section.
      f015397 [Josh Rosen] Minor grammar / pluralization fixes.
      3019f3a [Josh Rosen] Fix minor Markdown formatting issues
      aa8bb87 [Tathagata Das] Small update.
      195852c [Tathagata Das] Updated based on Josh's comments, updated receiver reliability and deploying section, and also updated configuration.
      17b99fb [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into streaming-doc-update-1.2
      a0217c0 [Tathagata Das] Changed Deploying menu layout
      67fcffc [Tathagata Das] Added cluster mode + supervise example to submitting application guide.
      e45453b [Tathagata Das] Update streaming guide, added deploying section.
      192c7a7 [Tathagata Das] Added more info about Python API, and rewrote the checkpointing section.
      b004150a
  12. Sep 03, 2014
    • Tathagata Das's avatar
      [SPARK-2419][Streaming][Docs] Updates to the streaming programming guide · a5224079
      Tathagata Das authored
      Updated the main streaming programming guide, and also added source-specific guides for Kafka, Flume, Kinesis.
      
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Author: Jacek Laskowski <jacek@japila.pl>
      
      Closes #2254 from tdas/streaming-doc-fix and squashes the following commits:
      
      e45c6d7 [Jacek Laskowski] More fixes from an old PR
      5125316 [Tathagata Das] Fixed links
      dc02f26 [Tathagata Das] Refactored streaming kinesis guide and made many other changes.
      acbc3e3 [Tathagata Das] Fixed links between streaming guides.
      cb7007f [Tathagata Das] Added Streaming + Flume integration guide.
      9bd9407 [Tathagata Das] Updated streaming programming guide with additional information from SPARK-2419.
      a5224079
Loading