blocks|key|763868|text|这完全不是Java的问题。您需要研究一种有效的算法来对未完全读取到内存中的数据进行排序。对Merge-Sort进行一些调整就可以实现这一点。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|763869|看看这个：http://en.wikipedia.org/wiki/Merge_sort|offset|length|763870|和：http://en.wikipedia.org/wiki/External_sorting|763871|基本上，这里的想法是将文件拆分成较小的片段，对它们进行排序(使用merge+sort或其他方法)，然后使用Merge+from+merge-sort创建新的排序文件。|763872|entityMap|0|LINK|mutability|MUTABLE|url|http://en.wikipedia.org/wiki/Merge_sort|1|http://en.wikipedia.org/wiki/External_sorting^0|0|5|13|0|0|2|19|1|0|0^^$0|@$1|2|3|4|5|6|7|T|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|U|8|@]|9|@$D|V|E|W|1|X]]|A|$]]|$1|F|3|G|5|6|7|Y|8|@]|9|@$D|Z|E|10|1|11]]|A|$]]|$1|H|3|I|5|6|7|12|8|@]|9|@]|A|$]]|$1|J|3|-4|5|6|7|13|8|@]|9|@]|A|$]]]|K|$L|$5|M|N|O|A|$P|Q]]|R|$5|M|N|O|A|$P|S]]]]

That isn't exactly a Java problem. You need to look into an efficient algorithm for sorting data that isn't completely read into memory. A few adaptations to Merge-Sort can achieve this.

Take a look at this:
<a href="http://en.wikipedia.org/wiki/Merge_sort" rel="noreferrer">http://en.wikipedia.org/wiki/Merge_sort</a>

and:
<a href="http://en.wikipedia.org/wiki/External_sorting" rel="noreferrer">http://en.wikipedia.org/wiki/External_sorting</a>

Basically the idea here is to break the file into smaller pieces, sort them (either with merge sort or another method), and then use the Merge from merge-sort to create the new, sorted file.

blocks|key|4685075|text|由于您的记录已经是平面文件文本格式，您可以将它们通过管道传输到UNIX+sort(1)，例如sort+-n+-t'+'+-k1,1+<+input+>+output。它将自动将数据分块，并使用可用内存和/tmp执行合并排序。如果需要的空间超过了可用内存，请在命令中添加-T+/tmpdir。|type|unstyled|depth|inlineStyleRanges|offset|length|style|CODE|entityRanges|data|4685076|有趣的是，每个人都告诉您下载大型C#或Java库，或者自己实现merge-sort，因为您可以使用在每个平台上都可用并且已经存在了几十年的工具。|4685077|entityMap^0|10|7|1A|10|2T|4|3Q|A|0|0^^$0|@$1|2|3|4|5|6|7|J|8|@$9|K|A|L|B|C]|$9|M|A|N|B|C]|$9|O|A|P|B|C]|$9|Q|A|R|B|C]]|D|@]|E|$]]|$1|F|3|G|5|6|7|S|8|@]|D|@]|E|$]]|$1|H|3|-4|5|6|7|T|8|@]|D|@]|E|$]]]|I|$]]

Since your records are already in flat file text format, you can pipe them into UNIX <code>sort(1)</code> e.g. <code>sort -n -t' ' -k1,1 &lt; input &gt; output</code>. It will automatically chunk the data and perform merge sort using available memory and <code>/tmp</code>. If you need more space than you have memory available, add <code>-T /tmpdir</code> to the command.

It's quite funny that everyone is telling you to download huge C# or Java libraries or implement merge-sort yourself when you can use a tool that is available on every platform and has been around for decades.

blocks|key|4967996|text|你需要一个外部的合并排序来做到这一点。Here是它的一个Java实现，可以对非常大的文件进行排序。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4967997|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/lemire/externalsortinginjava^0|J|4|0|0^^$0|@$1|2|3|4|5|6|7|L|8|@]|9|@$A|M|B|N|1|O]]|C|$]]|$1|D|3|-4|5|6|7|P|8|@]|9|@]|C|$]]]|E|$F|$5|G|H|I|C|$J|K]]]]

You need an external merge sort to do that. <a href="https://github.com/lemire/externalsortinginjava" rel="noreferrer">Here</a> is a Java implementation of it that sorts very large files.

blocks|key|4684937|text|而不是一次将所有数据加载到内存中，您可以只读取行开始处的键和索引(可能还有长度)，例如|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4684938|class+Line+{
+++int+key,+length;
+++long+start;
}|code-block|syntax|javascript|4684939|这将使用大约每行40个字节。|4684940|对此数组进行排序后，可以使用RandomAccessFile按照行的出现顺序读取它们。|4684941|注意:由于您将随机命中磁盘，而不是使用内存，因此这可能会非常慢。一个典型的磁盘随机访问数据需要8毫秒，如果你有1000万行数据，这将需要大约一天的时间。(这绝对是最坏的情况)在内存中大约需要10秒。|4684942|entityMap^0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|O|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|P|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|Q|8|@]|9|@]|A|$]]|$1|I|3|J|5|6|7|R|8|@]|9|@]|A|$]]|$1|K|3|L|5|6|7|S|8|@]|9|@]|A|$]]|$1|M|3|-4|5|6|7|T|8|@]|9|@]|A|$]]]|N|$]]

Instead of loading all the data into memory at once, you could read just the keys and an index to where the line starts (and possibly the length as well) e.g.

<pre><code>class Line {
 int key, length;
 long start;
}
</code></pre>

This would use about 40 bytes per line.

Once you have sorted this array, you can use RandomAccessFile to read the lines in the order they appear.

Note: since you will be randomly hitting the disk, instead of using memory this could be very slow. A typical disk takes 8 ms to randomly access data and if you have 10 million lines this will take about one day. (This is absolute worst case) In memory it would take about 10 seconds.

blocks|key|4968311|text|您需要执行外部排序。这是Hadoop/MapReduce背后的驱动思想，只是它没有考虑分布式集群，并且工作在单个节点上。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4968312|为了获得更好的性能，您应该使用Hadoop/Spark。|4968313|​|4968314|📷|atomic|offset|length|4968315|根据您的系统更改此行。fpath是您的一个大输入文件(测试为20+is+)。shared路径是存储执行日志的位置。fdir是存储和合并中间文件的位置。根据您的计算机更改这些路径。|style|CODE|4968316|public+static+final+String+fdir+=+"/tmp/";
++++public+static+final+String+shared+=+"/exports/home/schatterjee/cs553-pa2a/";
++++public+static+final+String+fPath+=+"/input/data-20GB.in";
++++public+static+final+String+opLog+=+shared%2B"Mysort20GB.log";|code-block|syntax|javascript|4968317|然后运行以下程序。将在fdir路径中创建名为op401的最终排序文件。最后一行Runtime.getRuntime().exec("valsort++"+%2B+fdir+%2B+"op"+%2B+(treeHeight*100)%2B1+%2B+"+>+"+%2B+opLog);检查输出是否排序。如果您没有安装valsort或者输入文件不是使用gensort(http://www.ordinal.com/gensort.html)生成的，请删除此行。|4968318|另外，不要忘记将int+totalLines+=+200000000;更改为文件中的总行数。线程计数(int+threadCount+=+16)应该始终是2的幂，并且应该足够大，以便(总大小*2/无线程)数量的数据可以驻留在内存中。更改线程计数将更改最终输出文件的名称。例如，对于16，它将是op401，对于32，它将是op501，对于8，它将是op301，依此类推。|4968319|好好享受吧。|4968320|++++import+java.io.*;
++++import+java.nio.file.Files;
++++import+java.nio.file.Paths;
++++import+java.util.ArrayList;
++++import+java.util.Comparator;
++++import+java.util.stream.Stream;


++++class+SplitFile+extends+Thread+{
++++++++String+fileName;
++++++++int+startLine,+endLine;

++++++++SplitFile(String+fileName,+int+startLine,+int+endLine)+{
++++++++++++this.fileName+=+fileName;
++++++++++++this.startLine+=+startLine;
++++++++++++this.endLine+=+endLine;
++++++++}

++++++++public+static+void+writeToFile(BufferedWriter+writer,+String+line)+{
++++++++++++try+{
++++++++++++++++writer.write(line+%2B+"\r\n");
++++++++++++}+catch+(Exception+e)+{
++++++++++++++++throw+new+RuntimeException(e);
++++++++++++}
++++++++}

++++++++public+void+run()+{
++++++++++++try+{
++++++++++++++++BufferedWriter+writer+=+Files.newBufferedWriter(Paths.get(fileName));
++++++++++++++++int+totalLines+=+endLine+%2B+1+-+startLine;
++++++++++++++++Stream<String>+chunks+=
++++++++++++++++++++++++Files.lines(Paths.get(Mysort20GB.fPath))
++++++++++++++++++++++++++++++++.skip(startLine+-+1)
++++++++++++++++++++++++++++++++.limit(totalLines)
++++++++++++++++++++++++++++++++.sorted(Comparator.naturalOrder());

++++++++++++++++chunks.forEach(line+->+{
++++++++++++++++++++writeToFile(writer,+line);
++++++++++++++++});
++++++++++++++++System.out.println("+Done+Writing+"+%2B+Thread.currentThread().getName());
++++++++++++++++writer.close();
++++++++++++}+catch+(Exception+e)+{
++++++++++++++++System.out.println(e);
++++++++++++}
++++++++}
++++}

++++class+MergeFiles+extends+Thread+{
++++++++String+file1,+file2,+file3;
++++++++MergeFiles(String+file1,+String+file2,+String+file3)+{
++++++++++++this.file1+=+file1;
++++++++++++this.file2+=+file2;
++++++++++++this.file3+=+file3;
++++++++}

++++++++public+void+run()+{
++++++++++++try+{
++++++++++++++++System.out.println(file1+%2B+"+Started+Merging+"+%2B+file2+);
++++++++++++++++FileReader+fileReader1+=+new+FileReader(file1);
++++++++++++++++FileReader+fileReader2+=+new+FileReader(file2);
++++++++++++++++FileWriter+writer+=+new+FileWriter(file3);
++++++++++++++++BufferedReader+bufferedReader1+=+new+BufferedReader(fileReader1);
++++++++++++++++BufferedReader+bufferedReader2+=+new+BufferedReader(fileReader2);
++++++++++++++++String+line1+=+bufferedReader1.readLine();
++++++++++++++++String+line2+=+bufferedReader2.readLine();
++++++++++++++++//Merge+2+files+based+on+which+string+is+greater.
++++++++++++++++while+(line1+!=+null+%7C%7C+line2+!=+null)+{
++++++++++++++++++++if+(line1+==+null+%7C%7C+(line2+!=+null+&&+line1.compareTo(line2)+>+0))+{
++++++++++++++++++++++++writer.write(line2+%2B+"\r\n");
++++++++++++++++++++++++line2+=+bufferedReader2.readLine();
++++++++++++++++++++}+else+{
++++++++++++++++++++++++writer.write(line1+%2B+"\r\n");
++++++++++++++++++++++++line1+=+bufferedReader1.readLine();
++++++++++++++++++++}
++++++++++++++++}
++++++++++++++++System.out.println(file1+%2B+"+Done+Merging+"+%2B+file2+);
++++++++++++++++new+File(file1).delete();
++++++++++++++++new+File(file2).delete();
++++++++++++++++writer.close();
++++++++++++}+catch+(Exception+e)+{
++++++++++++++++System.out.println(e);
++++++++++++}
++++++++}
++++}

++++public+class+Mysort20GB+{
++++++++//public+static+final+String+fdir+=+"/Users/diesel/Desktop/";
++++++++public+static+final+String+fdir+=+"/tmp/";
++++++++public+static+final+String+shared+=+"/exports/home/schatterjee/cs553-pa2a/";
++++++++public+static+final+String+fPath+=+"/input/data-20GB.in";
++++++++public+static+final+String+opLog+=+shared%2B"Mysort20GB.log";

++++++++public+static+void+main(String[]+args)+throws+Exception{
++++++++++++long+startTime+=+System.nanoTime();
++++++++++++int+threadCount+=+16;+//+Number+of+threads
++++++++++++int+totalLines+=+200000000;
++++++++++++int+linesPerFile+=+totalLines+/+threadCount;
++++++++++++ArrayList<Thread>+activeThreads+=+new+ArrayList<Thread>();

++++++++++++for+(int+i+=+1;+i+<=+threadCount;+i%2B%2B)+{
++++++++++++++++int+startLine+=+i+==+1+?+i+:+(i+-+1)+*+linesPerFile+%2B+1;
++++++++++++++++int+endLine+=+i+*+linesPerFile;
++++++++++++++++SplitFile+mapThreads+=+new+SplitFile(fdir+%2B+"op"+%2B+i,+startLine,+endLine);
++++++++++++++++activeThreads.add(mapThreads);
++++++++++++++++mapThreads.start();
++++++++++++}
++++++++++++activeThreads.stream().forEach(t+->+{
++++++++++++++++try+{
++++++++++++++++++++t.join();
++++++++++++++++}+catch+(Exception+e)+{
++++++++++++++++}
++++++++++++});

++++++++++++int+treeHeight+=+(int)+(Math.log(threadCount)+/+Math.log(2));

++++++++++++for+(int+i+=+0;+i+<+treeHeight;+i%2B%2B)+{
++++++++++++++++ArrayList<Thread>+actvThreads+=+new+ArrayList<Thread>();

for+(int+j+=+1,+itr+=+1;+j+<=+threadCount+/+(i+%2B+1);+j+%2B=+2,+itr%2B%2B)+{
++++++++++++++++++++int+offset+=+i+*+100;
++++++++++++++++++++String+tempFile1+=+fdir+%2B+"op"+%2B+(j+%2B+offset);
++++++++++++++++++++String+tempFile2+=+fdir+%2B+"op"+%2B+((j+%2B+1)+%2B+offset);
++++++++++++++++++++String+opFile+=+fdir+%2B+"op"+%2B+(itr+%2B+((i+%2B+1)+*+100));

++++++++++++++++++++MergeFiles+reduceThreads+=
++++++++++++++++++++++++++++new+MergeFiles(tempFile1,tempFile2,opFile);
++++++++++++++++++++actvThreads.add(reduceThreads);
++++++++++++++++++++reduceThreads.start();
++++++++++++++++}
++++++++++++++++actvThreads.stream().forEach(t+->+{
++++++++++++++++++++try+{
++++++++++++++++++++++++t.join();
++++++++++++++++++++}+catch+(Exception+e)+{
++++++++++++++++++++}
++++++++++++++++});
++++++++++++}
++++++++++++long+endTime+=+System.nanoTime();
++++++++++++double+timeTaken+=+(endTime+-+startTime)/1e9;
++++++++++++System.out.println(timeTaken);
++++++++++++BufferedWriter+logFile+=+new+BufferedWriter(new+FileWriter(opLog,+true));
++++++++++++logFile.write("Time+Taken+in+seconds:"+%2B+timeTaken);
++++++++++++Runtime.getRuntime().exec("valsort++"+%2B+fdir+%2B+"op"+%2B+(treeHeight*100)%2B1+%2B+"+>+"+%2B+opLog);
++++++++++++logFile.close();
++++++++}
++++}|4968321|entityMap|0|IMAGE|mutability|IMMUTABLE|imageUrl|https://ask.qcloudimg.com/http-save/yehe-900000/e59baaa9fa9310dd776f5985cdbc7b58.jpeg|imageAlt|1|LINK|MUTABLE|url|http://www.ordinal.com/gensort.html^0|0|0|0|0|1|0|0|B|5|12|6|1L|4|0|0|B|4|13|2I|4Q|Z|1|0|8|R|1F|K|0|0|0^^$0|@$1|2|3|4|5|6|7|1F|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|1G|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|1H|8|@]|9|@]|A|$]]|$1|F|3|G|5|H|7|1I|8|@]|9|@$I|1J|J|1K|1|1L]]|A|$]]|$1|K|3|L|5|6|7|1M|8|@$I|1N|J|1O|M|N]|$I|1P|J|1Q|M|N]|$I|1R|J|1S|M|N]]|9|@]|A|$]]|$1|O|3|P|5|Q|7|1T|8|@]|9|@]|A|$R|S]]|$1|T|3|U|5|6|7|1U|8|@$I|1V|J|1W|M|N]|$I|1X|J|1Y|M|N]]|9|@$I|1Z|J|20|1|21]]|A|$]]|$1|V|3|W|5|6|7|22|8|@$I|23|J|24|M|N]|$I|25|J|26|M|N]]|9|@]|A|$]]|$1|X|3|Y|5|6|7|27|8|@]|9|@]|A|$]]|$1|Z|3|10|5|Q|7|28|8|@]|9|@]|A|$R|S]]|$1|11|3|-4|5|6|7|29|8|@]|9|@]|A|$]]]|12|$13|$5|14|15|16|A|$17|18|19|-4]]|1A|$5|1B|15|1C|A|$1D|1E]]]]

You need to perform an external sort. It's kind the driving idea behind Hadoop/MapReduce , just that it doesn't take distributed cluster into account and works on a single node.

For better performance, you should use Hadoop/Spark.

<a href="https://i.stack.imgur.com/TcP6z.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TcP6z.jpg" alt="enter image description here"></a>
Change this lines according to your system . <code>fpath</code> is your one big input file (tested with 20GB). <code>shared</code> path is where the execution log is stored. <code>fdir</code> is where the intermediate files will be stored and merged. Change these paths according to your machine.

<pre><code>public static final String fdir = "/tmp/";
 public static final String shared = "/exports/home/schatterjee/cs553-pa2a/";
 public static final String fPath = "/input/data-20GB.in";
 public static final String opLog = shared+"Mysort20GB.log";
</code></pre>

Then run the following programme. Your final sorted file will be created with the name op401 in <code>fdir</code> path. the last line <code>Runtime.getRuntime().exec("valsort " + fdir + "op" + (treeHeight*100)+1 + " &gt; " + opLog);</code> checks the output is sorted or not . Remove this line if you dont have valsort installed or the input file is not generated using gensort(<a href="http://www.ordinal.com/gensort.html" rel="nofollow noreferrer">http://www.ordinal.com/gensort.html</a>) .

Also dont forget to change <code>int totalLines = 200000000;</code> to the total lines in your file. and thread count (<code>int threadCount = 16</code>) should be always in power of 2 and large enough so that (total size * 2 / no of thread) amount of data can reside in memory. Changing Thread count will change the name of final output file. Like for 16, it will be op401, for 32 it will be op501, for 8 it will be op301 etc.

Enjoy.

<pre><code> import java.io.*;
 import java.nio.file.Files;
 import java.nio.file.Paths;
 import java.util.ArrayList;
 import java.util.Comparator;
 import java.util.stream.Stream;


 class SplitFile extends Thread {
 String fileName;
 int startLine, endLine;

 SplitFile(String fileName, int startLine, int endLine) {
 this.fileName = fileName;
 this.startLine = startLine;
 this.endLine = endLine;
 }

 public static void writeToFile(BufferedWriter writer, String line) {
 try {
 writer.write(line + "\r\n");
 } catch (Exception e) {
 throw new RuntimeException(e);
 }
 }

 public void run() {
 try {
 BufferedWriter writer = Files.newBufferedWriter(Paths.get(fileName));
 int totalLines = endLine + 1 - startLine;
 Stream&lt;String&gt; chunks =
 Files.lines(Paths.get(Mysort20GB.fPath))
 .skip(startLine - 1)
 .limit(totalLines)
 .sorted(Comparator.naturalOrder());

 chunks.forEach(line -&gt; {
 writeToFile(writer, line);
 });
 System.out.println(" Done Writing " + Thread.currentThread().getName());
 writer.close();
 } catch (Exception e) {
 System.out.println(e);
 }
 }
 }

 class MergeFiles extends Thread {
 String file1, file2, file3;
 MergeFiles(String file1, String file2, String file3) {
 this.file1 = file1;
 this.file2 = file2;
 this.file3 = file3;
 }

 public void run() {
 try {
 System.out.println(file1 + " Started Merging " + file2 );
 FileReader fileReader1 = new FileReader(file1);
 FileReader fileReader2 = new FileReader(file2);
 FileWriter writer = new FileWriter(file3);
 BufferedReader bufferedReader1 = new BufferedReader(fileReader1);
 BufferedReader bufferedReader2 = new BufferedReader(fileReader2);
 String line1 = bufferedReader1.readLine();
 String line2 = bufferedReader2.readLine();
 //Merge 2 files based on which string is greater.
 while (line1 != null || line2 != null) {
 if (line1 == null || (line2 != null &amp;&amp; line1.compareTo(line2) &gt; 0)) {
 writer.write(line2 + "\r\n");
 line2 = bufferedReader2.readLine();
 } else {
 writer.write(line1 + "\r\n");
 line1 = bufferedReader1.readLine();
 }
 }
 System.out.println(file1 + " Done Merging " + file2 );
 new File(file1).delete();
 new File(file2).delete();
 writer.close();
 } catch (Exception e) {
 System.out.println(e);
 }
 }
 }

 public class Mysort20GB {
 //public static final String fdir = "/Users/diesel/Desktop/";
 public static final String fdir = "/tmp/";
 public static final String shared = "/exports/home/schatterjee/cs553-pa2a/";
 public static final String fPath = "/input/data-20GB.in";
 public static final String opLog = shared+"Mysort20GB.log";

 public static void main(String[] args) throws Exception{
 long startTime = System.nanoTime();
 int threadCount = 16; // Number of threads
 int totalLines = 200000000;
 int linesPerFile = totalLines / threadCount;
 ArrayList&lt;Thread&gt; activeThreads = new ArrayList&lt;Thread&gt;();

 for (int i = 1; i &lt;= threadCount; i++) {
 int startLine = i == 1 ? i : (i - 1) * linesPerFile + 1;
 int endLine = i * linesPerFile;
 SplitFile mapThreads = new SplitFile(fdir + "op" + i, startLine, endLine);
 activeThreads.add(mapThreads);
 mapThreads.start();
 }
 activeThreads.stream().forEach(t -&gt; {
 try {
 t.join();
 } catch (Exception e) {
 }
 });

 int treeHeight = (int) (Math.log(threadCount) / Math.log(2));

 for (int i = 0; i &lt; treeHeight; i++) {
 ArrayList&lt;Thread&gt; actvThreads = new ArrayList&lt;Thread&gt;();

for (int j = 1, itr = 1; j &lt;= threadCount / (i + 1); j += 2, itr++) {
 int offset = i * 100;
 String tempFile1 = fdir + "op" + (j + offset);
 String tempFile2 = fdir + "op" + ((j + 1) + offset);
 String opFile = fdir + "op" + (itr + ((i + 1) * 100));

 MergeFiles reduceThreads =
 new MergeFiles(tempFile1,tempFile2,opFile);
 actvThreads.add(reduceThreads);
 reduceThreads.start();
 }
 actvThreads.stream().forEach(t -&gt; {
 try {
 t.join();
 } catch (Exception e) {
 }
 });
 }
 long endTime = System.nanoTime();
 double timeTaken = (endTime - startTime)/1e9;
 System.out.println(timeTaken);
 BufferedWriter logFile = new BufferedWriter(new FileWriter(opLog, true));
 logFile.write("Time Taken in seconds:" + timeTaken);
 Runtime.getRuntime().exec("valsort " + fdir + "op" + (treeHeight*100)+1 + " &gt; " + opLog);
 logFile.close();
 }
 }
</code></pre>

blocks|key|4685192|text|使用java库big-sorter，该库可用于对非常大的文本或二进制文件进行排序。|type|unstyled|depth|inlineStyleRanges|entityRanges|offset|length|data|4685193|下面是你的问题是如何实现的：|4685194|//+write+the+input+to+a+file
String+s+=+"0052304+0000004000000000000000000000000000000041+++John+Teddy+++000023\n"
++++++++%2B+"0022024+0000004000000000000000000000000000000041+++George+Clan+00013";
File+input+=+new+File("target/input");
Files.write(input.toPath(),s.getBytes(StandardCharsets.UTF_8),+StandardOpenOption.WRITE);

File+output+=+new+File("target/output");


//sort+the+input
Sorter
++++.serializerLinesUtf8()
++++.comparator((a,b)+->+{
++++++++String+ida+=+a.substring(0,+a.indexOf('+'));
++++++++String+idb+=+b.substring(0,+b.indexOf('+'));
++++++++return+ida.compareTo(idb);
++++})+
++++.input(input)+
++++.output(output)+
++++.sort();

//+display+the+output
Files.readAllLines(output.toPath()).forEach(System.out::println);|code-block|syntax|javascript|4685195|输出：|4685196|0022024+0000004000000000000000000000000000000041+++George+Clan+00013
0052304+0000004000000000000000000000000000000041+++John+Teddy+++000023|4685197|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/davidmoten/big-sorter^0|7|A|0|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|W|8|@]|9|@$A|X|B|Y|1|Z]]|C|$]]|$1|D|3|E|5|6|7|10|8|@]|9|@]|C|$]]|$1|F|3|G|5|H|7|11|8|@]|9|@]|C|$I|J]]|$1|K|3|L|5|6|7|12|8|@]|9|@]|C|$]]|$1|M|3|N|5|H|7|13|8|@]|9|@]|C|$I|J]]|$1|O|3|-4|5|6|7|14|8|@]|9|@]|C|$]]]|P|$Q|$5|R|S|T|C|$U|V]]]]

Use the java library <a href="https://github.com/davidmoten/big-sorter" rel="nofollow noreferrer">big-sorter</a> which can be used for sorting very large text or binary files.

Here's how your exact problem would be implemented:

<pre class="lang-java prettyprint-override"><code>// write the input to a file
String s = "0052304 0000004000000000000000000000000000000041 John Teddy 000023\n"
 + "0022024 0000004000000000000000000000000000000041 George Clan 00013";
File input = new File("target/input");
Files.write(input.toPath(),s.getBytes(StandardCharsets.UTF_8), StandardOpenOption.WRITE);

File output = new File("target/output");


//sort the input
Sorter
 .serializerLinesUtf8()
 .comparator((a,b) -&gt; {
 String ida = a.substring(0, a.indexOf(' '));
 String idb = b.substring(0, b.indexOf(' '));
 return ida.compareTo(idb);
 }) 
 .input(input) 
 .output(output) 
 .sort();

// display the output
Files.readAllLines(output.toPath()).forEach(System.out::println);
</code></pre>

output:

<pre><code>0022024 0000004000000000000000000000000000000041 George Clan 00013
0052304 0000004000000000000000000000000000000041 John Teddy 000023
</code></pre>

blocks|key|4968125|text|您可以使用SQL+Lite文件db，将数据加载到db，然后让它排序并为您返回结果。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4968126|优点:不用担心写出最好的排序算法。|4968127|缺点:需要磁盘空间，处理速度慢。|4968128|https://sites.google.com/site/arjunwebworld/Home/programming/sorting-large-data-files|offset|length|4968129|entityMap|0|LINK|mutability|MUTABLE|url^0|0|0|0|0|2D|0|0^^$0|@$1|2|3|4|5|6|7|Q|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|R|8|@]|9|@]|A|$]]|$1|D|3|E|5|6|7|S|8|@]|9|@]|A|$]]|$1|F|3|G|5|6|7|T|8|@]|9|@$H|U|I|V|1|W]]|A|$]]|$1|J|3|-4|5|6|7|X|8|@]|9|@]|A|$]]]|K|$L|$5|M|N|O|A|$P|G]]]]

You can use SQL Lite file db, load the data to the db and then let it sort and return the results for you.

Advantages: No need to worry about writing the best sorting algorithm. 

Disadvantage: You will need disk space, slower processing.

<a href="https://sites.google.com/site/arjunwebworld/Home/programming/sorting-large-data-files" rel="nofollow">https://sites.google.com/site/arjunwebworld/Home/programming/sorting-large-data-files</a>

blocks|key|4967980|text|您需要做的是通过流将文件分块并单独处理它们。然后，您可以将文件合并在一起，因为它们将已经排序，这类似于合并排序的工作方式。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4967981|这个SO问题的答案将是有价值的：Stream+large+files|offset|length|4967982|entityMap|0|LINK|mutability|MUTABLE|url|https://stackoverflow.com/questions/55709/streaming-large-files-in-a-java-servlet/55788#55788^0|0|G|I|0|0^^$0|@$1|2|3|4|5|6|7|N|8|@]|9|@]|A|$]]|$1|B|3|C|5|6|7|O|8|@]|9|@$D|P|E|Q|1|R]]|A|$]]|$1|F|3|-4|5|6|7|S|8|@]|9|@]|A|$]]]|G|$H|$5|I|J|K|A|$L|M]]]]

What you need to do is to chunk the files in via a stream and process them separately. Then you can merge the files together as they will already be sorted, this is similar to how merge sort works. 

The answer from this SO question will be of value: <a href="https://stackoverflow.com/questions/55709/streaming-large-files-in-a-java-servlet/55788#55788">Stream large files</a>

blocks|key|4968189|text|操作系统自带强大的文件排序实用程序。调用bash脚本的简单函数应该会有所帮助。|type|unstyled|depth|inlineStyleRanges|entityRanges|data|4968190|public+static+void+runScript(final+Logger+log,+final+String+scriptFile)+throws+IOException,+InterruptedException+{
++++final+String+command+=+scriptFile;
++++if+(!new+File+(command).exists()+%7C%7C+!new+File(command).canRead()+%7C%7C+!new+File(command).canExecute())+{
++++++++log.log(Level.SEVERE,+"Cannot+find+or+read+"+%2B+command);
++++++++log.log(Level.WARNING,+"Make+sure+the+file+is+executable+and+you+have+permissions+to+execute+it.+Hint:+use+\"chmod+%2Bx+filename\"+to+make+it+executable");
++++++++throw+new+IOException("Cannot+find+or+read+"+%2B+command);
++++}
++++final+int+returncode+=+Runtime.getRuntime().exec(new+String[]+{"bash",+"-c",+command}).waitFor();
++++if+(returncode!=0)+{
++++++++log.log(Level.SEVERE,+"The+script+returned+an+Error+with+exit+code:+"+%2B+returncode);
++++++++throw+new+IOException();
++++}

}|code-block|syntax|javascript|4968191|entityMap^0|0|0^^$0|@$1|2|3|4|5|6|7|I|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|J|8|@]|9|@]|A|$E|F]]|$1|G|3|-4|5|6|7|K|8|@]|9|@]|A|$]]]|H|$]]

Operating systems come with powerful file sorting utility. A simple function that calls a bash script should help.

<pre><code>public static void runScript(final Logger log, final String scriptFile) throws IOException, InterruptedException {
 final String command = scriptFile;
 if (!new File (command).exists() || !new File(command).canRead() || !new File(command).canExecute()) {
 log.log(Level.SEVERE, "Cannot find or read " + command);
 log.log(Level.WARNING, "Make sure the file is executable and you have permissions to execute it. Hint: use \"chmod +x filename\" to make it executable");
 throw new IOException("Cannot find or read " + command);
 }
 final int returncode = Runtime.getRuntime().exec(new String[] {"bash", "-c", command}).waitFor();
 if (returncode!=0) {
 log.log(Level.SEVERE, "The script returned an Error with exit code: " + returncode);
 throw new IOException();
 }

}
</code></pre>

blocks|key|767758|text|我使用自己的逻辑并按格式对一个大的JSON文件进行了排序|type|unstyled|depth|inlineStyleRanges|entityRanges|data|767759|{"name":"hoge.finance","address":"0xfAd45E47083e4607302aa43c65fB3106F1cd7607"}|code-block|syntax|javascript|767760|完整的源代码可以在https://github.com/sitetester/token-sorter上找到，以及测试用例。代码有很好的文档记录，所以很容易理解。|offset|length|767761|它将输入文件拆分成多个较小的排序文件(可配置)，然后比较数据。|767762|在此粘贴一些评论...|767763|//+at+this+point,+we+have+sorted+data+sets+in+respective+files
//+next,+we+will+take+first+token+from+first+file+and+compare+it+with+tokens+of+all+other+files
//+during+comparison,+if+some+token+from+other+file+is+in+sorted+order,+then+we+make+it+default/initial+sorted+token
//+&+jump+to+next+file,+since+all+remaining+tokens+in+THAT+file+are+already+in+sorted+form
//+at+end+of+comparisons+with+all+files,+we+remove+it+from+specific+file+(so+it's+not+compared+next+time)+and+put/append+in+final+sorted+file
//+this+process+continues,+until+all+entries+are+matched
//+if+some+file+has+no+entries,+then+we+simply+delete+it+(so+it's+not+compared+next+time)|767764|entityMap|0|LINK|mutability|MUTABLE|url|https://github.com/sitetester/token-sorter^0|0|0|9|16|0|0|0|0|0^^$0|@$1|2|3|4|5|6|7|Y|8|@]|9|@]|A|$]]|$1|B|3|C|5|D|7|Z|8|@]|9|@]|A|$E|F]]|$1|G|3|H|5|6|7|10|8|@]|9|@$I|11|J|12|1|13]]|A|$]]|$1|K|3|L|5|6|7|14|8|@]|9|@]|A|$]]|$1|M|3|N|5|6|7|15|8|@]|9|@]|A|$]]|$1|O|3|P|5|D|7|16|8|@]|9|@]|A|$E|F]]|$1|Q|3|-4|5|6|7|17|8|@]|9|@]|A|$]]]|R|$S|$5|T|U|V|A|$W|X]]]]

I used my own logic and sorted a BIG JSON file in format
<pre><code>{&quot;name&quot;:&quot;hoge.finance&quot;,&quot;address&quot;:&quot;0xfAd45E47083e4607302aa43c65fB3106F1cd7607&quot;}
</code></pre>
Full source code is available on <a href="https://github.com/sitetester/token-sorter" rel="nofollow noreferrer">https://github.com/sitetester/token-sorter</a> along with test case. Code is well documented, so easy to understand.
It splits input file into multiple smaller SORTED files (configurable) and then compare data.
Pasting some comments here...
<pre><code>// at this point, we have sorted data sets in respective files
// next, we will take first token from first file and compare it with tokens of all other files
// during comparison, if some token from other file is in sorted order, then we make it default/initial sorted token
// &amp; jump to next file, since all remaining tokens in THAT file are already in sorted form
// at end of comparisons with all files, we remove it from specific file (so it's not compared next time) and put/append in final sorted file
// this process continues, until all entries are matched
// if some file has no entries, then we simply delete it (so it's not compared next time)
</code></pre>

I have some files that should be sorted according to id at the beginning of each line.
The files are about 2-3 gb. I tried to read all data into an <code>ArrayList</code> and sort them. But memory is not enough to keep them all. It does not work. 

Lines look like 
<code>0052304 0000004000000000000000000000000000000041 John Teddy 000023</code> 
<code>0022024 0000004000000000000000000000000000000041 George Clan 00013</code>

How can I sort the files??

How do I sort very large files

翻译质量差，导致语言生硬或混乱。

没有提供实际的解决方法或示例。

解答不清晰，无法理解或解决问题。

页面排版不美观，阅读体验差。

文章

问答

视频

教程

学习中心

腾讯云实验室

直播

竞赛

腾讯云代码分析专区

腾讯iOA零信任安全管理系统专区

腾讯云架构师技术同盟交流圈

腾讯云数据库专区

腾讯云智能顾问专区

腾讯云原生专区

腾讯混元专区

腾讯云TCE专区

腾讯云Lighthouse专区

腾讯云HAI专区

腾讯云Edgeone专区

腾讯云存储专区

腾讯云智能专区

腾讯轻联专区 

腾讯云开发专区

TAPD专区

腾讯轻量云游戏服专区

腾讯云最具价值专家

腾讯云架构师技术同盟

腾讯云创作之星

腾讯云开发者先锋

腾讯云AI代码助手

云原生构建

TAPD 敏捷项目管理

Cloud Studio

SDK中心

API中心

命令行工具

功能1上新10个字符

功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符功能2描述100个字符。

功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符功能2上新100个字符。

功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符功能5描述100个字符

功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符功能5上新100个字符

功能4上新

文章&问答评论现已支持表情

全新交互，全新视觉，新增快捷键、悬浮工具栏、高亮块等功能并同时优化现有功能，全面提升创作效率和体验

社区富文本编辑器全新改版！诚邀体验～ 

精选全网热门MCP server，让你的AI更好用 🚀

💥开发者 MCP广场重磅上线！

涵盖代码开发、场景应用、自动测试全流程，助你从零构建专属AI助手

一站式MCP教程库，解锁AI应用新玩法

聚焦“写作效率、视觉美观与运行性能”三方面进行全面升级，为您提供更高效、稳定的创作环境

社区富文本&Markdown编辑器全新改版上线，欢迎大家体验!

诚挚邀请您参与本次调研，分享您的真实使用感受与建议。您的反馈至关重要，感谢您的支持与参与！

社区新版编辑器体验调研

我有一些文件应该根据每行开头的id进行排序。文件大小约为2-3 gb。我尝试将所有数据读取到一个ArrayList中并对它们进行排序。但是内存并不足以保存所有这些数据。它不起作用。线条看起来像这样0052304 0000004000000000000000000000000000000041   John Teddy   0000230022024 000000400000000000000000

问如何对非常大的文件进行排序
EN

回答 10

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何对非常大的文件进行排序EN

回答 10

Stack Overflow用户

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问如何对非常大的文件进行排序
EN