[SPARK-5096] Use sbt tasks instead of vals to get hadoop version

This makes it possible to compile spark as an external `ProjectRef` where as now we throw a `FileNotFoundException`

Author: Michael Armbrust <michael@databricks.com>

Closes #3905 from marmbrus/effectivePom and squashes the following commits:

fd63aae [Michael Armbrust] Use sbt tasks instead of vals to get hadoop version.