-
- Downloads
Do not launch JavaGateways on workers (SPARK-674).
The problem was that the gateway was being initialized whenever the pyspark.context module was loaded. The fix uses lazy initialization that occurs only when SparkContext instances are actually constructed. I also made the gateway and jvm variables private. This change results in ~3-4x performance improvement when running the PySpark unit tests.
Loading
Please register or sign in to comment