Skip to content
Snippets Groups Projects
  1. Dec 16, 2015
  2. Nov 22, 2015
  3. Oct 27, 2015
  4. Jul 02, 2015
    • Deron Eriksson's avatar
      [SPARK-1564] [DOCS] Added Javascript to Javadocs to create badges for tags like :: Experimental :: · fcbcba66
      Deron Eriksson authored
      Modified copy_api_dirs.rb and created api-javadocs.js and api-javadocs.css files in order to add badges to javadoc files for :: Experimental ::, :: DeveloperApi ::, and :: AlphaComponent :: tags
      
      Author: Deron Eriksson <deron@us.ibm.com>
      
      Closes #7169 from deroneriksson/SPARK-1564_JavaDocs_badges and squashes the following commits:
      
      a8353db [Deron Eriksson] added license headers to api-docs.css and api-javadocs.css
      07feb07 [Deron Eriksson] added linebreaks to make jquery more readable when adding html badge tags
      65b4930 [Deron Eriksson] Modified copy_api_dirs.rb and created api-javadocs.js and api-javadocs.css files in order to add badges to javadoc files for :: Experimental ::, :: DeveloperApi ::, and :: AlphaComponent :: tags
      fcbcba66
  5. Jun 18, 2015
  6. May 30, 2014
    • Matei Zaharia's avatar
      [SPARK-1566] consolidate programming guide, and general doc updates · c8bf4131
      Matei Zaharia authored
      This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
      
      * A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
      * New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
      * Spark-submit guide moved to a separate page and expanded slightly
      * Various cleanups of the menu system, security docs, and others
      * Updated look of title bar to differentiate the docs from previous Spark versions
      
      You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
      
      Author: Matei Zaharia <matei@databricks.com>
      
      Closes #896 from mateiz/1.0-docs and squashes the following commits:
      
      03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
      0779508 [Matei Zaharia] tweak
      ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
      1bf4112 [Matei Zaharia] Review comments
      4414f88 [Matei Zaharia] tweaks
      d04e979 [Matei Zaharia] Fix some old links to Java guide
      a34ed33 [Matei Zaharia] tweak
      541bb3b [Matei Zaharia] miscellaneous changes
      fcefdec [Matei Zaharia] Moved submitting apps to separate doc
      61d72b4 [Matei Zaharia] stuff
      181f217 [Matei Zaharia] migration guide, remove old language guides
      e11a0da [Matei Zaharia] Add more API functions
      6a030a9 [Matei Zaharia] tweaks
      8db0ae3 [Matei Zaharia] Added key-value pairs section
      318d2c9 [Matei Zaharia] tweaks
      1c81477 [Matei Zaharia] New section on basics and function syntax
      e38f559 [Matei Zaharia] Actually added programming guide to Git
      a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
      3b6a876 [Matei Zaharia] More CSS tweaks
      01ec8bf [Matei Zaharia] More CSS tweaks
      e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
      c8bf4131
  7. Apr 09, 2014
    • Patrick Wendell's avatar
      SPARK-1093: Annotate developer and experimental API's · 87bd1f9e
      Patrick Wendell authored
      This patch marks some existing classes as private[spark] and adds two types of API annotations:
      - `EXPERIMENTAL API` = experimental user-facing module
      - `DEVELOPER API - UNSTABLE` = developer-facing API that might change
      
      There is some discussion of the different mechanisms for doing this here:
      https://issues.apache.org/jira/browse/SPARK-1081
      
      I was pretty aggressive with marking things private. Keep in mind that if we want to open something up in the future we can, but we can never reduce visibility.
      
      A few notes here:
      - In the past we've been inconsistent with the visiblity of the X-RDD classes. This patch marks them private whenever there is an existing function in RDD that can directly creat them (e.g. CoalescedRDD and rdd.coalesce()). One trade-off here is users can't subclass them.
      - Noted that compression and serialization formats don't have to be wire compatible across versions.
      - Compression codecs and serialization formats are semi-private as users typically don't instantiate them directly.
      - Metrics sources are made private - user only interacts with them through Spark's reflection
      
      Author: Patrick Wendell <pwendell@gmail.com>
      Author: Andrew Or <andrewor14@gmail.com>
      
      Closes #274 from pwendell/private-apis and squashes the following commits:
      
      44179e4 [Patrick Wendell] Merge remote-tracking branch 'apache-github/master' into private-apis
      042c803 [Patrick Wendell] spark.annotations -> spark.annotation
      bfe7b52 [Patrick Wendell] Adding experimental for approximate counts
      8d0c873 [Patrick Wendell] Warning in SparkEnv
      99b223a [Patrick Wendell] Cleaning up annotations
      e849f64 [Patrick Wendell] Merge pull request #2 from andrewor14/annotations
      982a473 [Andrew Or] Generalize jQuery matching for non Spark-core API docs
      a01c076 [Patrick Wendell] Merge pull request #1 from andrewor14/annotations
      c1bcb41 [Andrew Or] DeveloperAPI -> DeveloperApi
      0d48908 [Andrew Or] Comments and new lines (minor)
      f3954e0 [Andrew Or] Add identifier tags in comments to work around scaladocs bug
      99192ef [Andrew Or] Dynamically add badges based on annotations
      824011b [Andrew Or] Add support for injecting arbitrary JavaScript to API docs
      037755c [Patrick Wendell] Some changes after working with andrew or
      f7d124f [Patrick Wendell] Small fixes
      c318b24 [Patrick Wendell] Use CSS styles
      e4c76b9 [Patrick Wendell] Logging
      f390b13 [Patrick Wendell] Better visibility for workaround constructors
      d6b0afd [Patrick Wendell] Small chang to existing constructor
      403ba52 [Patrick Wendell] Style fix
      870a7ba [Patrick Wendell] Work around for SI-8479
      7fb13b2 [Patrick Wendell] Changes to UnionRDD and EmptyRDD
      4a9e90c [Patrick Wendell] EXPERIMENTAL API --> EXPERIMENTAL
      c581dce [Patrick Wendell] Changes after building against Shark.
      8452309 [Patrick Wendell] Style fixes
      1ed27d2 [Patrick Wendell] Formatting and coloring of badges
      cd7a465 [Patrick Wendell] Code review feedback
      2f706f1 [Patrick Wendell] Don't use floats
      542a736 [Patrick Wendell] Small fixes
      cf23ec6 [Patrick Wendell] Marking GraphX as alpha
      d86818e [Patrick Wendell] Another naming change
      5a76ed6 [Patrick Wendell] More visiblity clean-up
      42c1f09 [Patrick Wendell] Using better labels
      9d48cbf [Patrick Wendell] Initial pass
      87bd1f9e
  8. Feb 08, 2014
    • Martin Jaggi's avatar
      Merge pull request #552 from martinjaggi/master. Closes #552. · fabf1749
      Martin Jaggi authored
      tex formulas in the documentation
      
      using mathjax.
      and spliting the MLlib documentation by techniques
      
      see jira
      https://spark-project.atlassian.net/browse/MLLIB-19
      and
      https://github.com/shivaram/spark/compare/mathjax
      
      Author: Martin Jaggi <m.jaggi@gmail.com>
      
      == Merge branch commits ==
      
      commit 0364bfabbfc347f917216057a20c39b631842481
      Author: Martin Jaggi <m.jaggi@gmail.com>
      Date:   Fri Feb 7 03:19:38 2014 +0100
      
          minor polishing, as suggested by @pwendell
      
      commit dcd2142c164b2f602bf472bb152ad55bae82d31a
      Author: Martin Jaggi <m.jaggi@gmail.com>
      Date:   Thu Feb 6 18:04:26 2014 +0100
      
          enabling inline latex formulas with $.$
      
          same mathjax configuration as used in math.stackexchange.com
      
          sample usage in the linear algebra (SVD) documentation
      
      commit bbafafd2b497a5acaa03a140bb9de1fbb7d67ffa
      Author: Martin Jaggi <m.jaggi@gmail.com>
      Date:   Thu Feb 6 17:31:29 2014 +0100
      
          split MLlib documentation by techniques
      
          and linked from the main mllib-guide.md site
      
      commit d1c5212b93c67436543c2d8ddbbf610fdf0a26eb
      Author: Martin Jaggi <m.jaggi@gmail.com>
      Date:   Thu Feb 6 16:59:43 2014 +0100
      
          enable mathjax formula in the .md documentation files
      
          code by @shivaram
      
      commit d73948db0d9bc36296054e79fec5b1a657b4eab4
      Author: Martin Jaggi <m.jaggi@gmail.com>
      Date:   Thu Feb 6 16:57:23 2014 +0100
      
          minor update on how to compile the documentation
      fabf1749
  9. Jan 28, 2014
    • Tathagata Das's avatar
      Merge pull request #497 from tdas/docs-update · 79302096
      Tathagata Das authored
      Updated Spark Streaming Programming Guide
      
      Here is the updated version of the Spark Streaming Programming Guide. This is still a work in progress, but the major changes are in place. So feedback is most welcome.
      
      In general, I have tried to make the guide to easier to understand even if the reader does not know much about Spark. The updated website is hosted here -
      
      http://www.eecs.berkeley.edu/~tdas/spark_docs/streaming-programming-guide.html
      
      The major changes are:
      - Overview illustrates the usecases of Spark Streaming - various input sources and various output sources
      - An example right after overview to quickly give an idea of what Spark Streaming program looks like
      - Made Java API and examples a first class citizen like Scala by using tabs to show both Scala and Java examples (similar to AMPCamp tutorial's code tabs)
      - Highlighted the DStream operations updateStateByKey and transform because of their powerful nature
      - Updated driver node failure recovery text to highlight automatic recovery in Spark standalone mode
      - Added information about linking and using the external input sources like Kafka and Flume
      - In general, reorganized the sections to better show the Basic section and the more advanced sections like Tuning and Recovery.
      
      Todos:
      - Links to the docs of external Kafka, Flume, etc
      - Illustrate window operation with figure as well as example.
      
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      
      == Merge branch commits ==
      
      commit 18ff10556570b39d672beeb0a32075215cfcc944
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Tue Jan 28 21:49:30 2014 -0800
      
          Fixed a lot of broken links.
      
      commit 34a5a6008dac2e107624c7ff0db0824ee5bae45f
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Tue Jan 28 18:02:28 2014 -0800
      
          Updated github url to use SPARK_GITHUB_URL variable.
      
      commit f338a60ae8069e0a382d2cb170227e5757cc0b7a
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Mon Jan 27 22:42:42 2014 -0800
      
          More updates based on Patrick and Harvey's comments.
      
      commit 89a81ff25726bf6d26163e0dd938290a79582c0f
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Mon Jan 27 13:08:34 2014 -0800
      
          Updated docs based on Patricks PR comments.
      
      commit d5b6196b532b5746e019b959a79ea0cc013a8fc3
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Sun Jan 26 20:15:58 2014 -0800
      
          Added spark.streaming.unpersist config and info on StreamingListener interface.
      
      commit e3dcb46ab83d7071f611d9b5008ba6bc16c9f951
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Sun Jan 26 18:41:12 2014 -0800
      
          Fixed docs on StreamingContext.getOrCreate.
      
      commit 6c29524639463f11eec721e4d17a9d7159f2944b
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Thu Jan 23 18:49:39 2014 -0800
      
          Added example and figure for window operations, and links to Kafka and Flume API docs.
      
      commit f06b964a51bb3b21cde2ff8bdea7d9785f6ce3a9
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Wed Jan 22 22:49:12 2014 -0800
      
          Fixed missing endhighlight tag in the MLlib guide.
      
      commit 036a7d46187ea3f2a0fb8349ef78f10d6c0b43a9
      Merge: eab351d a1cd1851
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Wed Jan 22 22:17:42 2014 -0800
      
          Merge remote-tracking branch 'apache/master' into docs-update
      
      commit eab351d05c0baef1d4b549e1581310087158d78d
      Author: Tathagata Das <tathagata.das1565@gmail.com>
      Date:   Wed Jan 22 22:17:15 2014 -0800
      
          Update Spark Streaming Programming Guide.
      79302096
  10. Jan 07, 2014
  11. Dec 30, 2013
  12. Sep 02, 2013
  13. Sep 01, 2013
  14. Aug 30, 2013
  15. Aug 08, 2013
  16. Aug 07, 2013
  17. Aug 01, 2013
  18. Jul 31, 2013
  19. Feb 25, 2013
  20. Oct 13, 2012
  21. Oct 09, 2012
    • Matei Zaharia's avatar
      Updates to documentation: · bc0bc672
      Matei Zaharia authored
      - Edited quick start and tuning guide to simplify them a little
      - Simplified top menu bar
      - Made private a SparkContext constructor parameter that was left as
        public
      - Various small fixes
      bc0bc672
  22. Oct 08, 2012
  23. Sep 26, 2012
  24. Sep 25, 2012
  25. Sep 13, 2012
  26. Sep 12, 2012
Loading