Spark's RDD (elastic distributed data set) returns a large result set

（ One ） Too large result set results in Driver End OOM error

A long time ago 《 Learn big data platform from scratch 》 Just mentioned this problem .
adopt collect() Return result set , If the amount of data is too large, it will report driver End memory error .
So here I am , Return tens of millions of data , As a result Out of memory, perhaps GC overhead limit exceeded.

Spark Core elastic distributed data set , about rdd.collect() That's how it's described ：

java.util.List collect() //Python The version is similar
Returns an array containing RDD All the elements in
.
remarks
.
This method can only be used when the result set is small , Because all the data will be loaded into driver The memory of the .

（ Two ） Various solutions

How can we do if the results are extensive ？
I looked it up on the Internet , Everyone said ：

1. For large data sets rdd write in HDFS Method of file , such as rdd.saveAsTextFile,rdd.saveAsNewAPIHadoopFile Each node writes the results to HDFS A directory of （ A pile of documents ）.
2. Not to driver But rdd.foreach,rdd.foreachPartition Print it to the screen .
3. Through serializable types , Send to the database and so on .

（ 3、 ... and ） Return result sets in batches

But I'm lazy , I hope the whole process can switch seamlessly with the previous one ,
So I used collectPartitions, partial Return the data .

Java The code is as follows ：

...
        if (ColBatch < out_2.getNumPartitions()) {
    // If the batch value is set , And less than the number of partitions , Then in batches collect
            int[] Par = new int[ColBatch];
            for (int i = 0; i < out_2.getNumPartitions(); i += Par.length) {
    
                int ParLen = 0;
                for (int j = 0; j < Par.length; j++) {
    
                    if (i + j < out_2.getNumPartitions()) {
    
                        Par[j] = i + j;
                        ParLen++;
                    }
                }
                TmpLine = String.format(" At present : %d - %d\n", i, i + ParLen - 1);
                System.out.print(TmpLine);
                List<Tuple2<String, String>>[] output2 = out_2.collectPartitions(Par);
                for (int j = 0; j < ParLen; j++) {
    
                    for (Tuple2<String, String> tuple : output2[j]) {
    
                        oF002.write(String.format("%s|%s\n", tuple._1(), tuple._2()));
                        g002++;
                    }
                }
            }
        } else {
    
            List<Tuple2<String, String>> output1 = out_2.collect();
            for (Tuple2<String, String> tuple : output1) {
    
                oF002.write(String.format("%s|%s\n", tuple._1(), tuple._2()));
                g002++;
            }
        }
...

PS： but Python Not found in collectPartitions Methods: ？？？ What about swelling ！！！ .