Skip to content
Snippets Groups Projects
  • Patrick Wendell's avatar
    87bd1f9e
    SPARK-1093: Annotate developer and experimental API's · 87bd1f9e
    Patrick Wendell authored
    This patch marks some existing classes as private[spark] and adds two types of API annotations:
    - `EXPERIMENTAL API` = experimental user-facing module
    - `DEVELOPER API - UNSTABLE` = developer-facing API that might change
    
    There is some discussion of the different mechanisms for doing this here:
    https://issues.apache.org/jira/browse/SPARK-1081
    
    I was pretty aggressive with marking things private. Keep in mind that if we want to open something up in the future we can, but we can never reduce visibility.
    
    A few notes here:
    - In the past we've been inconsistent with the visiblity of the X-RDD classes. This patch marks them private whenever there is an existing function in RDD that can directly creat them (e.g. CoalescedRDD and rdd.coalesce()). One trade-off here is users can't subclass them.
    - Noted that compression and serialization formats don't have to be wire compatible across versions.
    - Compression codecs and serialization formats are semi-private as users typically don't instantiate them directly.
    - Metrics sources are made private - user only interacts with them through Spark's reflection
    
    Author: Patrick Wendell <pwendell@gmail.com>
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #274 from pwendell/private-apis and squashes the following commits:
    
    44179e4 [Patrick Wendell] Merge remote-tracking branch 'apache-github/master' into private-apis
    042c803 [Patrick Wendell] spark.annotations -> spark.annotation
    bfe7b52 [Patrick Wendell] Adding experimental for approximate counts
    8d0c873 [Patrick Wendell] Warning in SparkEnv
    99b223a [Patrick Wendell] Cleaning up annotations
    e849f64 [Patrick Wendell] Merge pull request #2 from andrewor14/annotations
    982a473 [Andrew Or] Generalize jQuery matching for non Spark-core API docs
    a01c076 [Patrick Wendell] Merge pull request #1 from andrewor14/annotations
    c1bcb41 [Andrew Or] DeveloperAPI -> DeveloperApi
    0d48908 [Andrew Or] Comments and new lines (minor)
    f3954e0 [Andrew Or] Add identifier tags in comments to work around scaladocs bug
    99192ef [Andrew Or] Dynamically add badges based on annotations
    824011b [Andrew Or] Add support for injecting arbitrary JavaScript to API docs
    037755c [Patrick Wendell] Some changes after working with andrew or
    f7d124f [Patrick Wendell] Small fixes
    c318b24 [Patrick Wendell] Use CSS styles
    e4c76b9 [Patrick Wendell] Logging
    f390b13 [Patrick Wendell] Better visibility for workaround constructors
    d6b0afd [Patrick Wendell] Small chang to existing constructor
    403ba52 [Patrick Wendell] Style fix
    870a7ba [Patrick Wendell] Work around for SI-8479
    7fb13b2 [Patrick Wendell] Changes to UnionRDD and EmptyRDD
    4a9e90c [Patrick Wendell] EXPERIMENTAL API --> EXPERIMENTAL
    c581dce [Patrick Wendell] Changes after building against Shark.
    8452309 [Patrick Wendell] Style fixes
    1ed27d2 [Patrick Wendell] Formatting and coloring of badges
    cd7a465 [Patrick Wendell] Code review feedback
    2f706f1 [Patrick Wendell] Don't use floats
    542a736 [Patrick Wendell] Small fixes
    cf23ec6 [Patrick Wendell] Marking GraphX as alpha
    d86818e [Patrick Wendell] Another naming change
    5a76ed6 [Patrick Wendell] More visiblity clean-up
    42c1f09 [Patrick Wendell] Using better labels
    9d48cbf [Patrick Wendell] Initial pass
    87bd1f9e
    History
    SPARK-1093: Annotate developer and experimental API's
    Patrick Wendell authored
    This patch marks some existing classes as private[spark] and adds two types of API annotations:
    - `EXPERIMENTAL API` = experimental user-facing module
    - `DEVELOPER API - UNSTABLE` = developer-facing API that might change
    
    There is some discussion of the different mechanisms for doing this here:
    https://issues.apache.org/jira/browse/SPARK-1081
    
    I was pretty aggressive with marking things private. Keep in mind that if we want to open something up in the future we can, but we can never reduce visibility.
    
    A few notes here:
    - In the past we've been inconsistent with the visiblity of the X-RDD classes. This patch marks them private whenever there is an existing function in RDD that can directly creat them (e.g. CoalescedRDD and rdd.coalesce()). One trade-off here is users can't subclass them.
    - Noted that compression and serialization formats don't have to be wire compatible across versions.
    - Compression codecs and serialization formats are semi-private as users typically don't instantiate them directly.
    - Metrics sources are made private - user only interacts with them through Spark's reflection
    
    Author: Patrick Wendell <pwendell@gmail.com>
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #274 from pwendell/private-apis and squashes the following commits:
    
    44179e4 [Patrick Wendell] Merge remote-tracking branch 'apache-github/master' into private-apis
    042c803 [Patrick Wendell] spark.annotations -> spark.annotation
    bfe7b52 [Patrick Wendell] Adding experimental for approximate counts
    8d0c873 [Patrick Wendell] Warning in SparkEnv
    99b223a [Patrick Wendell] Cleaning up annotations
    e849f64 [Patrick Wendell] Merge pull request #2 from andrewor14/annotations
    982a473 [Andrew Or] Generalize jQuery matching for non Spark-core API docs
    a01c076 [Patrick Wendell] Merge pull request #1 from andrewor14/annotations
    c1bcb41 [Andrew Or] DeveloperAPI -> DeveloperApi
    0d48908 [Andrew Or] Comments and new lines (minor)
    f3954e0 [Andrew Or] Add identifier tags in comments to work around scaladocs bug
    99192ef [Andrew Or] Dynamically add badges based on annotations
    824011b [Andrew Or] Add support for injecting arbitrary JavaScript to API docs
    037755c [Patrick Wendell] Some changes after working with andrew or
    f7d124f [Patrick Wendell] Small fixes
    c318b24 [Patrick Wendell] Use CSS styles
    e4c76b9 [Patrick Wendell] Logging
    f390b13 [Patrick Wendell] Better visibility for workaround constructors
    d6b0afd [Patrick Wendell] Small chang to existing constructor
    403ba52 [Patrick Wendell] Style fix
    870a7ba [Patrick Wendell] Work around for SI-8479
    7fb13b2 [Patrick Wendell] Changes to UnionRDD and EmptyRDD
    4a9e90c [Patrick Wendell] EXPERIMENTAL API --> EXPERIMENTAL
    c581dce [Patrick Wendell] Changes after building against Shark.
    8452309 [Patrick Wendell] Style fixes
    1ed27d2 [Patrick Wendell] Formatting and coloring of badges
    cd7a465 [Patrick Wendell] Code review feedback
    2f706f1 [Patrick Wendell] Don't use floats
    542a736 [Patrick Wendell] Small fixes
    cf23ec6 [Patrick Wendell] Marking GraphX as alpha
    d86818e [Patrick Wendell] Another naming change
    5a76ed6 [Patrick Wendell] More visiblity clean-up
    42c1f09 [Patrick Wendell] Using better labels
    9d48cbf [Patrick Wendell] Initial pass
api-docs.js 1.10 KiB