Skip to content
Snippets Groups Projects
  • Punya Biswal's avatar
    f6a9a57a
    [SPARK-6952] Handle long args when detecting PID reuse · f6a9a57a
    Punya Biswal authored
    sbin/spark-daemon.sh used
    
        ps -p "$TARGET_PID" -o args=
    
    to figure out whether the process running with the expected PID is actually a Spark
    daemon. When running with a large classpath, the output of ps gets
    truncated and the check fails spuriously.
    
    This weakens the check to see if it's a java command (which is something
    we do in other parts of the script) rather than looking for the specific
    main class name. This means that SPARK-4832 might happen under a
    slightly broader range of circumstances (a java program happened to
    reuse the same PID), but it seems worthwhile compared to failing
    consistently with a large classpath.
    
    Author: Punya Biswal <pbiswal@palantir.com>
    
    Closes #5535 from punya/feature/SPARK-6952 and squashes the following commits:
    
    7ea12d1 [Punya Biswal] Handle long args when detecting PID reuse
    f6a9a57a
    History
    [SPARK-6952] Handle long args when detecting PID reuse
    Punya Biswal authored
    sbin/spark-daemon.sh used
    
        ps -p "$TARGET_PID" -o args=
    
    to figure out whether the process running with the expected PID is actually a Spark
    daemon. When running with a large classpath, the output of ps gets
    truncated and the check fails spuriously.
    
    This weakens the check to see if it's a java command (which is something
    we do in other parts of the script) rather than looking for the specific
    main class name. This means that SPARK-4832 might happen under a
    slightly broader range of circumstances (a java program happened to
    reuse the same PID), but it seems worthwhile compared to failing
    consistently with a large classpath.
    
    Author: Punya Biswal <pbiswal@palantir.com>
    
    Closes #5535 from punya/feature/SPARK-6952 and squashes the following commits:
    
    7ea12d1 [Punya Biswal] Handle long args when detecting PID reuse
spark-daemon.sh 5.06 KiB