Skip to content
  • Jeremy Freeman's avatar
    f6a18993
    Streaming mllib [SPARK-2438][MLLIB] · f6a18993
    Jeremy Freeman authored
    This PR implements a streaming linear regression analysis, in which a linear regression model is trained online as new data arrive. The design is based on discussions with tdas and mengxr, in which we determined how to add this functionality in a general way, with minimal changes to existing libraries.
    
    __Summary of additions:__
    
    _StreamingLinearAlgorithm_
    - An abstract class for fitting generalized linear models online to streaming data, including training on (and updating) a model, and making predictions.
    
    _StreamingLinearRegressionWithSGD_
    - Class and companion object for running streaming linear regression
    
    _StreamingLinearRegressionTestSuite_
    - Unit tests
    
    _StreamingLinearRegression_
    - Example use case: fitting a model online to data from one stream, and making predictions on other data
    
    __Notes__
    - If this looks good, I can use the StreamingLinearAlgorithm class to easily implement other analyses that follow the same logic (Ridge, Lasso, Logistic, SVM).
    
    Author: Jeremy Freeman <the.freeman.lab@gmail.com>
    Author: freeman <the.freeman.lab@gmail.com>
    
    Closes #1361 from freeman-lab/streaming-mllib and squashes the following commits:
    
    775ea29 [Jeremy Freeman] Throw error if user doesn't initialize weights
    4086fee [Jeremy Freeman] Fixed current weight formatting
    8b95b27 [Jeremy Freeman] Restored broadcasting
    29f27ec [Jeremy Freeman] Formatting
    8711c41 [Jeremy Freeman] Used return to avoid indentation
    777b596 [Jeremy Freeman] Restored treeAggregate
    74cf440 [Jeremy Freeman] Removed static methods
    d28cf9a [Jeremy Freeman] Added usage notes
    c3326e7 [Jeremy Freeman] Improved documentation
    9541a41 [Jeremy Freeman] Merge remote-tracking branch 'upstream/master' into streaming-mllib
    66eba5e [Jeremy Freeman] Fixed line lengths
    2fe0720 [Jeremy Freeman] Minor cleanup
    7d51378 [Jeremy Freeman] Moved streaming loader to MLUtils
    b9b69f6 [Jeremy Freeman] Added setter methods
    c3f8b5a [Jeremy Freeman] Modified logging
    00aafdc [Jeremy Freeman] Add modifiers
    14b801e [Jeremy Freeman] Name changes
    c7d38a3 [Jeremy Freeman] Move check for empty data to GradientDescent
    4b0a5d3 [Jeremy Freeman] Cleaned up tests
    74188d6 [Jeremy Freeman] Eliminate dependency on commons
    50dd237 [Jeremy Freeman] Removed experimental tag
    6bfe1e6 [Jeremy Freeman] Fixed imports
    a2a63ad [freeman] Makes convergence test more robust
    86220bc [freeman] Streaming linear regression unit tests
    fb4683a [freeman] Minor changes for scalastyle consistency
    fd31e03 [freeman] Changed logging behavior
    453974e [freeman] Fixed indentation
    c4b1143 [freeman] Streaming linear regression
    604f4d7 [freeman] Expanded private class to include mllib
    d99aa85 [freeman] Helper methods for streaming MLlib apps
    0898add [freeman] Added dependency on streaming
    f6a18993
    Streaming mllib [SPARK-2438][MLLIB]
    Jeremy Freeman authored
    This PR implements a streaming linear regression analysis, in which a linear regression model is trained online as new data arrive. The design is based on discussions with tdas and mengxr, in which we determined how to add this functionality in a general way, with minimal changes to existing libraries.
    
    __Summary of additions:__
    
    _StreamingLinearAlgorithm_
    - An abstract class for fitting generalized linear models online to streaming data, including training on (and updating) a model, and making predictions.
    
    _StreamingLinearRegressionWithSGD_
    - Class and companion object for running streaming linear regression
    
    _StreamingLinearRegressionTestSuite_
    - Unit tests
    
    _StreamingLinearRegression_
    - Example use case: fitting a model online to data from one stream, and making predictions on other data
    
    __Notes__
    - If this looks good, I can use the StreamingLinearAlgorithm class to easily implement other analyses that follow the same logic (Ridge, Lasso, Logistic, SVM).
    
    Author: Jeremy Freeman <the.freeman.lab@gmail.com>
    Author: freeman <the.freeman.lab@gmail.com>
    
    Closes #1361 from freeman-lab/streaming-mllib and squashes the following commits:
    
    775ea29 [Jeremy Freeman] Throw error if user doesn't initialize weights
    4086fee [Jeremy Freeman] Fixed current weight formatting
    8b95b27 [Jeremy Freeman] Restored broadcasting
    29f27ec [Jeremy Freeman] Formatting
    8711c41 [Jeremy Freeman] Used return to avoid indentation
    777b596 [Jeremy Freeman] Restored treeAggregate
    74cf440 [Jeremy Freeman] Removed static methods
    d28cf9a [Jeremy Freeman] Added usage notes
    c3326e7 [Jeremy Freeman] Improved documentation
    9541a41 [Jeremy Freeman] Merge remote-tracking branch 'upstream/master' into streaming-mllib
    66eba5e [Jeremy Freeman] Fixed line lengths
    2fe0720 [Jeremy Freeman] Minor cleanup
    7d51378 [Jeremy Freeman] Moved streaming loader to MLUtils
    b9b69f6 [Jeremy Freeman] Added setter methods
    c3f8b5a [Jeremy Freeman] Modified logging
    00aafdc [Jeremy Freeman] Add modifiers
    14b801e [Jeremy Freeman] Name changes
    c7d38a3 [Jeremy Freeman] Move check for empty data to GradientDescent
    4b0a5d3 [Jeremy Freeman] Cleaned up tests
    74188d6 [Jeremy Freeman] Eliminate dependency on commons
    50dd237 [Jeremy Freeman] Removed experimental tag
    6bfe1e6 [Jeremy Freeman] Fixed imports
    a2a63ad [freeman] Makes convergence test more robust
    86220bc [freeman] Streaming linear regression unit tests
    fb4683a [freeman] Minor changes for scalastyle consistency
    fd31e03 [freeman] Changed logging behavior
    453974e [freeman] Fixed indentation
    c4b1143 [freeman] Streaming linear regression
    604f4d7 [freeman] Expanded private class to include mllib
    d99aa85 [freeman] Helper methods for streaming MLlib apps
    0898add [freeman] Added dependency on streaming
Loading