[SPARK-19617][SS] Fix the race condition when starting and stopping a query quickly (branch-2.1)

## What changes were proposed in this pull request?

Backport #16947 to branch 2.1. Note: we still need to support old Hadoop versions in 2.1.*.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #16979 from zsxwing/SPARK-19617-branch-2.1.