Skip to content
Snippets Groups Projects
Commit 276a7e13 authored by gatorsmile's avatar gatorsmile Committed by Michael Armbrust
Browse files

[SPARK-11633][SQL] LogicalRDD throws TreeNode Exception : Failed to Copy Node

When handling self joins, the implementation did not consider the case insensitivity of HiveContext. It could cause an exception as shown in the JIRA:
```
TreeNodeException: Failed to copy node.
```

The fix is low risk. It avoids unnecessary attribute replacement. It should not affect the existing behavior of self joins. Also added the test case to cover this case.

Author: gatorsmile <gatorsmile@gmail.com>

Closes #9762 from gatorsmile/joinMakeCopy.
parent 72d150c2
No related branches found
No related tags found
No related merge requests found
......@@ -74,6 +74,10 @@ private[sql] case class LogicalRDD(
override def children: Seq[LogicalPlan] = Nil
override protected final def otherCopyArgs: Seq[AnyRef] = {
sqlContext :: Nil
}
override def newInstance(): LogicalRDD.this.type =
LogicalRDD(output.map(_.newInstance()), rdd)(sqlContext).asInstanceOf[this.type]
......
......@@ -1110,6 +1110,20 @@ class DataFrameSuite extends QueryTest with SharedSQLContext {
}
}
// This test case is to verify a bug when making a new instance of LogicalRDD.
test("SPARK-11633: LogicalRDD throws TreeNode Exception: Failed to Copy Node") {
withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
val rdd = sparkContext.makeRDD(Seq(Row(1, 3), Row(2, 1)))
val df = sqlContext.createDataFrame(
rdd,
new StructType().add("f1", IntegerType).add("f2", IntegerType),
needsConversion = false).select($"F1", $"f2".as("f2"))
val df1 = df.as("a")
val df2 = df.as("b")
checkAnswer(df1.join(df2, $"a.f2" === $"b.f2"), Row(1, 3, 1, 3) :: Row(2, 1, 2, 1) :: Nil)
}
}
test("SPARK-10656: completely support special chars") {
val df = Seq(1 -> "a").toDF("i_$.a", "d^'a.")
checkAnswer(df.select(df("*")), Row(1, "a"))
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment