Skip to content
Snippets Groups Projects
Commit 126baa8d authored by ding's avatar ding Committed by Joseph K. Bradley
Browse files

[SPARK-17559][MLLIB] persist edges if their storage level is non in PeriodicGraphCheckpointer

## What changes were proposed in this pull request?
When use PeriodicGraphCheckpointer to persist graph, sometimes the edges isn't persisted. As currently only when vertices's storage level is none, graph is persisted. However there is a chance vertices's storage level is not none while edges's is none. Eg. graph created by a outerJoinVertices operation, vertices is automatically cached while edges is not. In this way, edges will not be persisted if we use PeriodicGraphCheckpointer do persist. We need separately check edges's storage level and persisted it if it's none.

## How was this patch tested?
 manual tests

Author: ding <ding@localhost.localdomain>

Closes #15124 from dding3/spark-persisitEdge.
parent d2dc8c4a
No related branches found
No related tags found
No related merge requests found
......@@ -87,7 +87,10 @@ private[mllib] class PeriodicGraphCheckpointer[VD, ED](
override protected def persist(data: Graph[VD, ED]): Unit = {
if (data.vertices.getStorageLevel == StorageLevel.NONE) {
data.persist()
data.vertices.persist()
}
if (data.edges.getStorageLevel == StorageLevel.NONE) {
data.edges.persist()
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment