How to copy files fastly

First let’s generate a dummy file for test. We can do it with the following cmd superly fast.

1
fallocate -l 2G copy_test.img

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import java.io.*;
import java.nio.file.Files;

public class Main {

    public static void main(String[] args) {
        File source = new File("/home/benson/develop/test/copy_test.img");
        File target1 = new File("/home/benson/develop/test/target1.img");
        File target2 = new File("/home/benson/develop/test/target2.img");
        File target3 = new File("/home/benson/develop/test/target3.img");
        System.out.println("Source file is of size: " + source.length());
        long pre = System.currentTimeMillis();
        long now = -1;
        try(FileInputStream fin = new FileInputStream(source); FileOutputStream fout = new FileOutputStream(target1)) {
            fin.getChannel().transferTo(0, source.length(), fout.getChannel());
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
        now = System.currentTimeMillis();
        System.out.println("NIO tranfer takes: " + (now - pre));
        pre = now;
        try {
            Files.copy(source.toPath(), target2.toPath());
        } catch (IOException e) {
            e.printStackTrace();
        }
        now = System.currentTimeMillis();
        System.out.println("Files.copy takes: " + (now - pre));
        pre = now;
        try (BufferedInputStream bin = new BufferedInputStream(new FileInputStream(source));
             BufferedOutputStream bout = new BufferedOutputStream(new FileOutputStream(target3));) {
            byte[] bs = new byte[512];
            int count = -1;
            while ((count = bin.read(bs)) != -1) {
                bout.write(bs, 0, count);
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
        now = System.currentTimeMillis();
        System.out.println("BufferedInputStream takes: " + (now - pre));
    }

}

Output:

1
2
3
4
Source file is of size: 2147483648
NIO tranfer takes: 893
Files.copy takes: 3087
BufferedInputStream takes: 13113

NIO FileChannel::transfer has the best performance. It utilizes direct ByteBuffer, which directly operates the file data in OS’s kernal space. While Files::copy and BufferedInputStream has to transfer data from kernal space and vise versa.

I planned to also test for MappedByteBuffer, but it doesn’t support to map more than 2 GB(precisely 2^31-1 bytes) data into memory directly. So I give it up. But basically, I guess it would have the same performace as FileChannel::transferTo.

But currently I don’t have a clue why BufferedInputStream is way worse than Files.copy. Let’s explore it later.

Please not that direct buffer is allocated outside of Java heap space. And it’s only cleaned at Full GC. So we have to use it carefully to avoid OOM. To adjust max direct memory size, we can use

1
-XX:MaxDirectMemorySize=512M