Skip to content
Snippets Groups Projects
  • 郭小龙 10207633's avatar
    4d99b95a
    [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl'... · 4d99b95a
    郭小龙 10207633 authored
    [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md
    
    ## What changes were proposed in this pull request?
    
    Currently, our project needs to be set to clean up the worker directory cleanup cycle is three days.
    When I follow http://spark.apache.org/docs/latest/spark-standalone.html, configure the 'spark.worker.cleanup.appDataTtl' parameter, I configured to 3 * 24 * 3600.
    When I start the spark service, the startup fails, and the worker log displays the error log as follows:
    
    2017-04-28 15:02:03,306 INFO Utils: Successfully started service 'sparkWorker' on port 48728.
    Exception in thread "main" java.lang.NumberFormatException: For input string: "3 * 24 * 3600"
    	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    	at java.lang.Long.parseLong(Long.java:430)
    	at java.lang.Long.parseLong(Long.java:483)
    	at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:276)
    	at scala.collection.immutable.StringOps.toLong(StringOps.scala:29)
    	at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
    	at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
    	at scala.Option.map(Option.scala:146)
    	at org.apache.spark.SparkConf.getLong(SparkConf.scala:380)
    	at org.apache.spark.deploy.worker.Worker.<init>(Worker.scala:100)
    	at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:730)
    	at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:709)
    	at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
    
    **Because we put 7 * 24 * 3600 as a string, forced to convert to the dragon type,  will lead to problems in the program.**
    
    **So I think the default value of the current configuration should be a specific long value, rather than 7 * 24 * 3600,should be 604800. Because it would mislead users for similar configurations, resulting in spark start failure.**
    
    ## How was this patch tested?
    manual tests
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.
    
    Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn>
    Author: guoxiaolong <guo.xiaolong1@zte.com.cn>
    Author: guoxiaolongzte <guo.xiaolong1@zte.com.cn>
    
    Closes #17798 from guoxiaolongzte/SPARK-20521.
    4d99b95a
    History
    [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl'...
    郭小龙 10207633 authored
    [SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md
    
    ## What changes were proposed in this pull request?
    
    Currently, our project needs to be set to clean up the worker directory cleanup cycle is three days.
    When I follow http://spark.apache.org/docs/latest/spark-standalone.html, configure the 'spark.worker.cleanup.appDataTtl' parameter, I configured to 3 * 24 * 3600.
    When I start the spark service, the startup fails, and the worker log displays the error log as follows:
    
    2017-04-28 15:02:03,306 INFO Utils: Successfully started service 'sparkWorker' on port 48728.
    Exception in thread "main" java.lang.NumberFormatException: For input string: "3 * 24 * 3600"
    	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    	at java.lang.Long.parseLong(Long.java:430)
    	at java.lang.Long.parseLong(Long.java:483)
    	at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:276)
    	at scala.collection.immutable.StringOps.toLong(StringOps.scala:29)
    	at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
    	at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
    	at scala.Option.map(Option.scala:146)
    	at org.apache.spark.SparkConf.getLong(SparkConf.scala:380)
    	at org.apache.spark.deploy.worker.Worker.<init>(Worker.scala:100)
    	at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:730)
    	at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:709)
    	at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
    
    **Because we put 7 * 24 * 3600 as a string, forced to convert to the dragon type,  will lead to problems in the program.**
    
    **So I think the default value of the current configuration should be a specific long value, rather than 7 * 24 * 3600,should be 604800. Because it would mislead users for similar configurations, resulting in spark start failure.**
    
    ## How was this patch tested?
    manual tests
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.
    
    Author: 郭小龙 10207633 <guo.xiaolong1@zte.com.cn>
    Author: guoxiaolong <guo.xiaolong1@zte.com.cn>
    Author: guoxiaolongzte <guo.xiaolong1@zte.com.cn>
    
    Closes #17798 from guoxiaolongzte/SPARK-20521.