Extremely slow O_DIRECT

[pavel-odintsov]

Hello!

I'm not sure about problem in ncw/directio but it's possible.

Version with directio.OpenFile:

echo 3 > /proc/sys/vm/drop_caches
 /root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
 539MB 0:00:25 [21,1MB/s] 

Version with os.OpenFile:

echo 3 > /proc/sys/vm/drop_caches
/root/fast_backup_sender/cat_page_cache_safe debian-6.0-x86_64-isplite.tar.gz |pv >/dev/null
 539MB 0:00:03 [ 148MB/s] 

As u can see O_DIRECT version is much times slower then same code without O_DIRECT. But it's very strange because I cleaned page cache before tests.

Full code you can find here: https://gist.github.com/pavel-odintsov/f303309f5284e09bc359

Thank you!

[ncw]

Remember that O_DIRECT does no buffering at all. If you want it to do buffering you need to provide it yourself. Upping the buffer_block in your example like this makes it run a lot faster.

buffer_block := directio.AlignedBlock(4 * 1024 * directio.BlockSize) // 16MB buffer

Another thing the OS is doing for you without O_DIRECT is read ahead, so if you set one go routine up to read and another to write you'll gain a bit more speed as those things can work in parallel again.

So I don't think it is a bug, it is just a pitfall of using O_DIRECT.

[ncw]

I'm closing this as I don't think this is a bug in this library, just a pitfall of O_DIRECT. (See above for explanation).

[pavel-odintsov]

Thank you for answer! I will add read ahead to my code :)