Skip to content
  • Joseph K. Bradley's avatar
    842d0003
    [SPARK-5461] [graphx] Add isCheckpointed, getCheckpointedFiles methods to Graph · 842d0003
    Joseph K. Bradley authored
    Added the 2 methods to Graph and GraphImpl.  Both make calls to the underlying vertex and edge RDDs.
    
    This is needed for another PR (for LDA): [https://github.com/apache/spark/pull/4047]
    
    Notes:
    * getCheckpointedFiles is plural and returns a Seq[String] instead of an Option[String].
    * I attempted to test to make sure the methods returned the correct values after checkpointing.  It did not work; I guess that checkpointing does not occur quickly enough?  I noticed that there are not checkpointing tests for RDDs; is it just hard to test well?
    
    CC: rxin
    
    CC: mengxr  (since related to LDA)
    
    Author: Joseph K. Bradley <joseph@databricks.com>
    
    Closes #4253 from jkbradley/graphx-checkpoint and squashes the following commits:
    
    b680148 [Joseph K. Bradley] added class tag to firstParent call in VertexRDDImpl.isCheckpointed, though not needed to compile
    250810e [Joseph K. Bradley] In EdgeRDDImple, VertexRDDImpl, added transient back to partitionsRDD, and made isCheckpointed check firstParent instead of partitionsRDD
    695b7a3 [Joseph K. Bradley] changed partitionsRDD in EdgeRDDImpl, VertexRDDImpl to be non-transient
    cc00767 [Joseph K. Bradley] added overrides for isCheckpointed, getCheckpointFile in EdgeRDDImpl, VertexRDDImpl. The corresponding Graph methods now work.
    188665f [Joseph K. Bradley] improved documentation
    235738c [Joseph K. Bradley] Added isCheckpointed and getCheckpointFiles to Graph, GraphImpl
    842d0003
    [SPARK-5461] [graphx] Add isCheckpointed, getCheckpointedFiles methods to Graph
    Joseph K. Bradley authored
    Added the 2 methods to Graph and GraphImpl.  Both make calls to the underlying vertex and edge RDDs.
    
    This is needed for another PR (for LDA): [https://github.com/apache/spark/pull/4047]
    
    Notes:
    * getCheckpointedFiles is plural and returns a Seq[String] instead of an Option[String].
    * I attempted to test to make sure the methods returned the correct values after checkpointing.  It did not work; I guess that checkpointing does not occur quickly enough?  I noticed that there are not checkpointing tests for RDDs; is it just hard to test well?
    
    CC: rxin
    
    CC: mengxr  (since related to LDA)
    
    Author: Joseph K. Bradley <joseph@databricks.com>
    
    Closes #4253 from jkbradley/graphx-checkpoint and squashes the following commits:
    
    b680148 [Joseph K. Bradley] added class tag to firstParent call in VertexRDDImpl.isCheckpointed, though not needed to compile
    250810e [Joseph K. Bradley] In EdgeRDDImple, VertexRDDImpl, added transient back to partitionsRDD, and made isCheckpointed check firstParent instead of partitionsRDD
    695b7a3 [Joseph K. Bradley] changed partitionsRDD in EdgeRDDImpl, VertexRDDImpl to be non-transient
    cc00767 [Joseph K. Bradley] added overrides for isCheckpointed, getCheckpointFile in EdgeRDDImpl, VertexRDDImpl. The corresponding Graph methods now work.
    188665f [Joseph K. Bradley] improved documentation
    235738c [Joseph K. Bradley] Added isCheckpointed and getCheckpointFiles to Graph, GraphImpl
Loading