Approximate distinct count

Added countApproxDistinct() to RDD and countApproxDistinctByKey() to PairRDDFunctions to approximately count distinct number of elements and distinct number of values per key, respectively. Both functions use HyperLogLog from stream-lib for counting. Both functions take a parameter that controls the trade-off between accuracy and memory consumption. Also added Scala docs and test suites for both methods.

upgrade Netty from 4.0.0.Beta2 to 4.0.13.Final

the changes are listed at https://github.com/netty/netty/wiki/New-and-noteworthy

Bug fixes for file input stream and checkpointing

- Fixed bugs in the file input stream that led the stream to fail due to transient HDFS errors (listing files when a background thread it deleting fails caused errors, etc.)
- Updated Spark's CheckpointRDD and Streaming's CheckpointWriter to use SparkContext.hadoopConfiguration, to allow checkpoints to be written to any HDFS compatible store requiring special configuration.
- Changed the API of SparkContext.setCheckpointDir() - eliminated the unnecessary 'useExisting' parameter. Now SparkContext will always create a unique subdirectory within the user specified checkpoint directory. This is to ensure that previous checkpoint files are not accidentally overwritten.
- Fixed bug where setting checkpoint directory as a relative local path caused the checkpointing to fail.

Changed naming of StageCompleted event to be consistent

The rest of the SparkListener events are named with "SparkListener"
as the prefix of the name; this commit renames the StageCompleted
event to SparkListenerStageCompleted for consistency.

This reverts commit 79b20e4d, reversing
changes made to 7375047d.

Fix typo in the Accumulators section

Change 'val' to 'var'

val => var

Removed unused failed and causeOfFailure variables (in TaskSetManager)

Removed unused OtherFailure TaskEndReason.

The OtherFailure TaskEndReason was added by @mateiz 3 years ago in this commit: https://github.com/apache/incubator-spark/commit/24a1e7f8380bfd8d4fbdda688482a451bd6ea215

Unless I am missing something, it doesn't seem to have been used then, and is not used now, so seems safe for deletion.

Remove unused hasPendingTasks methods

The rest of the SparkListener events are named with "SparkListener"
as the prefix of the name; this commit renames the StageCompleted
event to SparkListenerStageCompleted for consistency.

Fixed >100char lines in DAGScheduler.scala

There's no changed functionality here -- only line spacing and one grammatical fix in a comment.

    Also clean up a bit.

Minor: Decrease margin of left side of Log page

Before
![before](https://f.cloud.github.com/assets/1400247/1812647/1a4be53e-6e87-11e3-9d5b-f851274be0e9.png)

After
![after](https://f.cloud.github.com/assets/1400247/1812648/1ca1ea2c-6e87-11e3-946c-31be9258f450.png)

It's a start anyway...

SPARK-1007: spark-class2.cmd should change SCALA_VERSION to be 2.10

Reported by Qiuzhuang Lian

Avoid a lump of coal (NPE) in JobProgressListener's stocking.

Changed file stream to not catch any exceptions related to finding new files (FileNotFound exception is still caught and ignored).

Renamed ClusterScheduler to TaskSchedulerImpl for yarn and new-yarn package

Removed slack time in file stream and added better handling of exceptions due to failures due FileNotFound exceptions.