readall(`echo hello`) leaks memory on master and 0.3

[StefanKarpinski]

The following program leaks memory on 0.3 and 0.4 to varying degrees:

while true
    readall(`echo hello`)
    gc() # <= with or without this
end

On 0.3 without the gc() call is the worst – memory usage grows to 1.2 GB on my machine in a couple of minutes. If you put the gc() call in then it's much better – indicating that the memory pressure isn't being felt as it should – but there's still a gradual leak. On 0.4 things are better – and memory pressure is felt as it should be even without gc() – but there is still a slow leak.

[yuyichao]

Could this be related to #9767?

I would be quite surprized if the GC doesn't free the memory it owns.

[JeffBezanson]

We malloc libuv handles, which are very very small but with lots of them it might matter?

[vtjnash]

what OS have you tested this on?

[yuyichao]

On 0.4 Linux running

for i in 1:1000000
       readall(`echo hello`)
       i % 1000 == 0 && println(i)
end

I could see a linear memory usage increase of the julia process (measured with resident size with no swap) ~ 220 bytes per iteration.

[StefanKarpinski]

Also on OS X.

[vtjnash]

after running this for 75000 loops, then canceling with ^C, the most interesting thing I seemed to get was:

julia> ^D
Assertion failed: (uv__stream_fd(stream) >= 0), function uv_shutdown, file src/unix/stream.c, line 1166.

signal (6): Abort trap: 6
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Abort trap: 6
/home/me$ 

for reference, on OS X, the size of the malloc objects is:

julia> Int(Base._sizeof_uv_process)
104

julia> Int(Base._sizeof_uv_named_pipe)
240

after 10-20 minutes of running the test on 0.4 OS X, my machine exhibits behavior on the order of yuyichao's results (200-600 bytes/loop) after accomplishing ~50_000 loops. but that's only a net change of 30 MB and is not approaching nowhere near 1.2 GB.

[yuyichao]

A much faster way to get the memory leak is

function f(n)
    for i in 1:n
        fd = open("a")
        readall(fd)
        close(fd)
        i % 1000 == 0 && println(i)
    end
end

Edit: where "a" is an empty file.

[yuyichao]

After all this might still be GC related.

julia> function f1(n)
       for i in 1:n
       fd = open("a")
       close(fd)
       i % 1000 == 0 && println(i)
       end
       end
f1 (generic function with 1 method)

julia> function f2(n)
       for i in 1:n
       fd = open("a")
       close(fd)
       finalize(fd)
       i % 1000 == 0 && println(i)
       end
       end
f2 (generic function with 1 method)

f1 "leaks" memory while f2 doesn't.

[StefanKarpinski]

@vtjnash, note that the 1.2 GB is on release-0.3. On master, Julia feels the memory pressure as it should so the leak is relatively slow (but still present).