Duplicate rows could be remove or drop from Spark SQL DataFrame using distinct() and dropDuplicates() functions, distinct() can be used to remove rows
Remove duplicated plan node check in DataFrameSetOperationsSuite Why are the changes needed? Code is unnecessarily checking forInMemoryTableScanExecin executed plan twice. Does this PR introduceanyuser-facing change? No How was this patch tested? UT Was this patch authored or co-authored using gene...