Skip to content
Snippets Groups Projects
Commit d4893540 authored by Davies Liu's avatar Davies Liu Committed by Davies Liu
Browse files

[SPARK-16077] [PYSPARK] catch the exception from pickle.whichmodule()

## What changes were proposed in this pull request?

In the case that we don't know which module a object came from, will call pickle.whichmodule() to go throught all the loaded modules to find the object, which could fail because some modules, for example, six, see https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling

We should ignore the exception here, use `__main__` as the module name (it means we can't find the module).

## How was this patch tested?

Manual tested. Can't have a unit test for this.

Author: Davies Liu <davies@databricks.com>

Closes #13788 from davies/whichmodule.
parent a4851ed0
No related branches found
No related tags found
No related merge requests found
...@@ -169,7 +169,12 @@ class CloudPickler(Pickler): ...@@ -169,7 +169,12 @@ class CloudPickler(Pickler):
if name is None: if name is None:
name = obj.__name__ name = obj.__name__
modname = pickle.whichmodule(obj, name) try:
# whichmodule() could fail, see
# https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling
modname = pickle.whichmodule(obj, name)
except Exception:
modname = None
# print('which gives %s %s %s' % (modname, obj, name)) # print('which gives %s %s %s' % (modname, obj, name))
try: try:
themodule = sys.modules[modname] themodule = sys.modules[modname]
...@@ -326,7 +331,12 @@ class CloudPickler(Pickler): ...@@ -326,7 +331,12 @@ class CloudPickler(Pickler):
modname = getattr(obj, "__module__", None) modname = getattr(obj, "__module__", None)
if modname is None: if modname is None:
modname = pickle.whichmodule(obj, name) try:
# whichmodule() could fail, see
# https://bitbucket.org/gutworth/six/issues/63/importing-six-breaks-pickling
modname = pickle.whichmodule(obj, name)
except Exception:
modname = '__main__'
if modname == '__main__': if modname == '__main__':
themodule = None themodule = None
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment