Skip to content
Snippets Groups Projects
Commit 8817fc7f authored by Takeshi Yamamuro's avatar Takeshi Yamamuro Committed by Ankur Dave
Browse files

[SPARK-4620] Add unpersist in Graph and GraphImpl

Add an IF to uncache both vertices and edges of Graph/GraphImpl.
This IF is useful when iterative graph operations build a new graph in each iteration, and the vertices and edges of previous iterations are no longer needed for following iterations.

Author: Takeshi Yamamuro <linguin.m.s@gmail.com>

This patch had conflicts when merged, resolved by
Committer: Ankur Dave <ankurdave@gmail.com>

Closes #3476 from maropu/UnpersistInGraphSpike and squashes the following commits:

77a006a [Takeshi Yamamuro] Add unpersist in Graph and GraphImpl
parent 2e6b736b
No related branches found
No related tags found
No related merge requests found
...@@ -104,6 +104,12 @@ abstract class Graph[VD: ClassTag, ED: ClassTag] protected () extends Serializab ...@@ -104,6 +104,12 @@ abstract class Graph[VD: ClassTag, ED: ClassTag] protected () extends Serializab
*/ */
def checkpoint(): Unit def checkpoint(): Unit
/**
* Uncaches both vertices and edges of this graph. This is useful in iterative algorithms that
* build a new graph in each iteration.
*/
def unpersist(blocking: Boolean = true): Graph[VD, ED]
/** /**
* Uncaches only the vertices of this graph, leaving the edges alone. This is useful in iterative * Uncaches only the vertices of this graph, leaving the edges alone. This is useful in iterative
* algorithms that modify the vertex attributes but reuse the edges. This method can be used to * algorithms that modify the vertex attributes but reuse the edges. This method can be used to
......
...@@ -70,6 +70,12 @@ class GraphImpl[VD: ClassTag, ED: ClassTag] protected ( ...@@ -70,6 +70,12 @@ class GraphImpl[VD: ClassTag, ED: ClassTag] protected (
replicatedVertexView.edges.checkpoint() replicatedVertexView.edges.checkpoint()
} }
override def unpersist(blocking: Boolean = true): Graph[VD, ED] = {
unpersistVertices(blocking)
replicatedVertexView.edges.unpersist(blocking)
this
}
override def unpersistVertices(blocking: Boolean = true): Graph[VD, ED] = { override def unpersistVertices(blocking: Boolean = true): Graph[VD, ED] = {
vertices.unpersist(blocking) vertices.unpersist(blocking)
// TODO: unpersist the replicated vertices in `replicatedVertexView` but leave the edges alone // TODO: unpersist the replicated vertices in `replicatedVertexView` but leave the edges alone
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment