This is Schema I got this error.. Traceback (most recent call last): File "/HOME/rayjang/spark-2.2.0-bin-hadoop2.7/python/pyspark/cloudpickle.py", line 148, in dump return Pickler.dump(self, obj) File "/HOME/anaconda3/lib/python3.5/pickle.py", line 408, in dump self.save(obj) ...
In Spark foreachPartition() is used when you have a heavy initialization (like database connection) and wanted to initialize once per partition where as