Skip to content
  • DB Tsai's avatar
    c1080b6f
    [SPARK-7568] [ML] ml.LogisticRegression doesn't output the right prediction · c1080b6f
    DB Tsai authored
    The difference is because we previously don't fit the intercept in Spark 1.3. Here, we change the input `String` so that the probability of instance 6 can be classified as `1.0` without any ambiguity.
    
    with lambda = 0.001 in current LOR implementation, the prediction is
    ```
    (4, spark i j k) --> prob=[0.1596407738787411,0.8403592261212589], prediction=1.0
    (5, l m n) --> prob=[0.8378325685476612,0.16216743145233883], prediction=0.0
    (6, spark hadoop spark) --> prob=[0.0692663313297627,0.9307336686702373], prediction=1.0
    (7, apache hadoop) --> prob=[0.9821575333444208,0.01784246665557917], prediction=0.0
    ```
    and the training accuracy is
    ```
    (0, a b c d e spark) --> prob=[0.0021342419881406746,0.9978657580118594], prediction=1.0
    (1, b d) --> prob=[0.9959176174854043,0.004082382514595685], prediction=0.0
    (2, spark f g h) --> prob=[0.0014541569986711233,0.9985458430013289], prediction=1.0
    (3, hadoop mapreduce) --> prob=[0.9982978367343561,0.0017021632656438518], prediction=0.0
    ```
    
    Author: DB Tsai <dbt@netflix.com>
    
    Closes #6109 from dbtsai/lor-example and squashes the following commits:
    
    ac63ce4 [DB Tsai] first commit
    c1080b6f
    [SPARK-7568] [ML] ml.LogisticRegression doesn't output the right prediction
    DB Tsai authored
    The difference is because we previously don't fit the intercept in Spark 1.3. Here, we change the input `String` so that the probability of instance 6 can be classified as `1.0` without any ambiguity.
    
    with lambda = 0.001 in current LOR implementation, the prediction is
    ```
    (4, spark i j k) --> prob=[0.1596407738787411,0.8403592261212589], prediction=1.0
    (5, l m n) --> prob=[0.8378325685476612,0.16216743145233883], prediction=0.0
    (6, spark hadoop spark) --> prob=[0.0692663313297627,0.9307336686702373], prediction=1.0
    (7, apache hadoop) --> prob=[0.9821575333444208,0.01784246665557917], prediction=0.0
    ```
    and the training accuracy is
    ```
    (0, a b c d e spark) --> prob=[0.0021342419881406746,0.9978657580118594], prediction=1.0
    (1, b d) --> prob=[0.9959176174854043,0.004082382514595685], prediction=0.0
    (2, spark f g h) --> prob=[0.0014541569986711233,0.9985458430013289], prediction=1.0
    (3, hadoop mapreduce) --> prob=[0.9982978367343561,0.0017021632656438518], prediction=0.0
    ```
    
    Author: DB Tsai <dbt@netflix.com>
    
    Closes #6109 from dbtsai/lor-example and squashes the following commits:
    
    ac63ce4 [DB Tsai] first commit
Loading