[HOT FIX] [YARN] Check whether `/lib` exists before listing its files

This is caused by a975dc32

Author: Andrew Or <andrew@databricks.com>

Closes #3589 from andrewor14/yarn-hot-fix and squashes the following commits:

a4fad5f [Andrew Or] Check whether lib directory exists before listing its files