runtime: multi-ms sweep termination pauses

[rhysh]

What version of Go are you using (go version)?

go version devel +3df059e Mon Nov 7 06:46:01 2016 +0000

What operating system and processor architecture are you using (go env)?

linux/amd64

What did you do?

I have a server program that makes http requests to collect a large volume of data which it then unmarshals and transforms. It runs with around 1000 active goroutines, has a live heap of around 30–100MB and executes around 50 GCs per second. The GC does a fine job keeping up with the program's throughput demands.

What did you expect to see?

I expected sub-millisecond stop-the-world times during garbage collections; per #17503 (comment), "The hybrid barrier is now live on master."

What did you see instead?

I see mark termination STW times of over 1ms somewhat regularly (which I thought the issue 17503 hybrid barrier would eliminate).

I see sweep termination STW times of over 1ms in the majority of GC cycles, which I find particularly surprising since I thought there was very little work required in that phase.

gc 20072 @698.871s 14%: 0.14+5.1+0.35 ms clock, 4.7+14/41/1.1+11 ms cpu, 104->104->46 MB, 107 MB goal, 36 P
gc 20073 @698.886s 14%: 3.1+4.5+6.6 ms clock, 90+57/30/0.74+191 ms cpu, 91->92->61 MB, 92 MB goal, 36 P
gc 20074 @698.917s 14%: 1.5+4.3+0.64 ms clock, 46+47/35/0+19 ms cpu, 121->122->69 MB, 123 MB goal, 36 P
gc 20075 @698.935s 14%: 7.9+4.5+1.0 ms clock, 230+35/30/0+31 ms cpu, 142->143->85 MB, 143 MB goal, 36 P
gc 20076 @698.967s 14%: 6.2+10+1.2 ms clock, 180+29/34/0+35 ms cpu, 160->164->86 MB, 170 MB goal, 36 P

It looks like sometimes the runtime stops most of the goroutines in preparation for a GC, but a few remain running. See for instance Proc 15 around 285–289ms (Proc 28 offscreen is also running), or Proc 0 and Proc 19 around 312–314ms

/cc @aclements @RLH

[rhysh]

Below are the distributions of GC STW pauses for the program (in milliseconds), via https://godoc.org/github.com/aclements/go-moremath/cmd/dist

Sweep termination:

N 9991  sum 31376.8  mean 3.1405  gmean 2.22543  std dev 1.93111  variance 3.7292

     min 0.023
   1%ile 0.0772467
   5%ile 0.25
  25%ile 1.5
  median 3.3
  75%ile 4.6
  95%ile 6.2
  99%ile 8.1
     max 11

⠀⠀⠀⠀⠀⠀⠀⡰⠒⢦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⡤⠤⠤⠤⠤⢤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.247
⠀⠀⠀⠀⠀⠀⣰⠁⠀⠀⠓⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠋⠉⠉⠉⠉⠉⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠒⠦⠤⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠴⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠓⠒⠒⠒⠒⠲⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
      0                               5

Mark termination:

N 9991  sum 6526.75  mean 0.653263  gmean 0.502061  std dev 0.848719  variance 0.720324

     min 0.19
   1%ile 0.26
   5%ile 0.3
  25%ile 0.36
  median 0.42
  75%ile 0.54
  95%ile 1.8
  99%ile 5.2
     max 13

⠀⠀⠀⠀⠀⠀⠠⢲⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 4.273
⠀⠀⠀⠀⠀⠀⠨⠀⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠤⠎⠀⠈⠳⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
    0                                            5

[RLH]

Do we know what the running goroutines are doing? Could they be in a tight
long running loop that doesn't involve a preemption point such as a call?

On Mon, Nov 7, 2016 at 11:06 AM, Rhys Hiltner notifications@github.com
wrote:

Below are the distributions of GC STW pauses for the program (in
milliseconds), via https://godoc.org/github.com/
aclements/go-moremath/cmd/dist

Sweep termination:

N 9991 sum 31376.8 mean 3.1405 gmean 2.22543 std dev 1.93111 variance 3.7292

 min 0.023

1%ile 0.0772467
5%ile 0.25
25%ile 1.5
median 3.3
75%ile 4.6
95%ile 6.2
99%ile 8.1
max 11

⠀⠀⠀⠀⠀⠀⠀⡰⠒⢦⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⣀⣀⣀⡤⠤⠤⠤⠤⢤⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 0.247
⠀⠀⠀⠀⠀⠀⣰⠁⠀⠀⠓⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠋⠉⠉⠉⠉⠉⠉⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠒⠦⠤⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠴⠃⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠉⠓⠒⠒⠒⠒⠲⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠙⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
0 5

Mark termination:

N 9991 sum 6526.75 mean 0.653263 gmean 0.502061 std dev 0.848719 variance 0.720324

 min 0.19

1%ile 0.26
5%ile 0.3
25%ile 0.36
median 0.42
75%ile 0.54
95%ile 1.8
99%ile 5.2
max 13

⠀⠀⠀⠀⠀⠀⠠⢲⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡖ 4.273
⠀⠀⠀⠀⠀⠀⠨⠀⢇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇
⠠⠤⠤⠤⠤⠤⠎⠀⠈⠳⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠄⠧ 0.000
⠈⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠋⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠁
0 5

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17831 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn2OCnmSd0PnEkA5zex5Bgd5bqpZWks5q70yWgaJpZM4KrPQ8
.

[rhysh]

Hmm, yes. It looks like every time the beginning of a GC cycle is delayed, one of the running goroutines appears to be in a single expensive call to encoding/base64.StdEncoding.DecodeString; the "End Stack Trace" indicates a line right after that kind of call.

The base64 messages that are decoded are around 1MB, which could keep a goroutine busy for several milliseconds.

[RLH]

Can that code be rewritten so that it makes a preemptable call once every
100usecs or whatever your tolerance level is?

On Mon, Nov 7, 2016 at 11:58 AM, Rhys Hiltner notifications@github.com
wrote:

Hmm, yes. It looks like every time the beginning of a GC cycle is delayed,
one of the running goroutines appears to be in a single expensive call to
encoding/base64.StdEncoding.DecodeString; the "End Stack Trace" indicates
a line right after that kind of call.

The base64 messages that are decoded are around 1MB, which could keep a
goroutine busy for several milliseconds.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17831 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn9GR6awzyYvse9JKrC_MFrOQ6Mlfks5q71i1gaJpZM4KrPQ8
.

[rhysh]

Are you suggesting a change to the encoding/base64 package, or to all applications that use it for large slices?

My application uses encoding/base64 indirectly via a third-party package, so making a localized change isn't a straightforward solution.

Fixing #10958 would be pretty helpful here. The program I've encountered this in is very throughput sensitive and somewhat latency sensitive, which goes against the assumptions used to de-prioritize that issue.

Gradually disabling each P in preparation for a GC pause means that my 36-core server is effectively a 1- or 2-core server for several milliseconds each cycle.

[RLH]

The appropriate place would seem to be the encoding/base64 package.

On Mon, Nov 7, 2016 at 12:32 PM, Rhys Hiltner notifications@github.com
wrote:

Are you suggesting a change to the encoding/base64 package, or to all
applications that use it for large slices?

My application uses encoding/base64 indirectly via a third-party package,
so making a localized change isn't a straightforward solution.

Fixing #10958 #10958 would be pretty
helpful here. The program I've encountered this in is very throughput
sensitive and somewhat latency sensitive, which goes against the
assumptions used to de-prioritize that issue.

Gradually disabling each P in preparation for a GC pause means that my
36-core server is effectively a 1- or 2-core server for several
milliseconds each cycle.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17831 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn1y4uqHuwTmhwVgtMIEfVxYFKBh8ks5q72CqgaJpZM4KrPQ8
.

[bradfitz]

The appropriate place would seem to be the encoding/base64 package.

I strongly disagree.

We are not going to litter the standard library and every package containing a backwards branch with calls to help the GC or scheduler.

Maybe @rhysh can hack up a local fork of the standard library as a short-term workaround, but fixing #10958 has to be the right long term-term answer.

[aclements]

Fixing #10958 would be pretty helpful here. The program I've encountered this in is very throughput sensitive and somewhat latency sensitive, which goes against the assumptions used to de-prioritize that issue.

The main reason that issue has been de-prioritized is not because of a concern over throughput (though that does complicate the implementation), but just because we haven't seen it arise in "real" code. So, this is actually an excellent example to motivate re-prioritizing it.

Unfortunately, fixing this is rather complicated (and it's clearly not a regression), so I think it's too late in Go 1.8 to put a real fix in the compiler.

@rhysh, to confirm that we really do understand the issue (and as a possible temporary fix), could you modify encoding/base64.Encoding.decode to force a preemption check at the top of the outer loop? It's slightly annoying to do this because our compiler is getting more clever, but calling the following function seems to do it:

var alwaysFalse = false

//go:noinline
func preempt(x int) {
    if alwaysFalse {
        println()
    }
}

[RLH]

Whether the compiler puts it in or it is put in by hand the appropriate
place is in encoding/base64. We have avoid putting it into the compiler due
to the (anticipated) performance issues. This bug is interesting since it
argues for adding the check to improve performance.

On Mon, Nov 7, 2016 at 2:12 PM, Brad Fitzpatrick notifications@github.com
wrote:

The appropriate place would seem to be the encoding/base64 package.

I strongly disagree.

We are not going to litter the standard library and every package
containing a backwards branch with calls to help the GC or scheduler.

Maybe @rhysh https://github.com/rhysh can hack up a local fork of the
standard library as a short-term workaround, but fixing #10958
#10958 has to be the right long
term-term answer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17831 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AA7Wn96-HofcxfrWfv3gGkoOPmPwlUHNks5q73gWgaJpZM4KrPQ8
.

[ianlancetaylor]

I think a simpler way to add a preemption point would be to call runtime.Gosched(). That seems like the right approach for a long-running loop anyhow. (And then if necessary we should speed up Gosched.)

[aclements]

I agree that that's simpler, but Gosched() there will force a preemption every 4 characters consumed from the buffer. The cost of entering the scheduler for that, even if we just decide to return to the same goroutine, will surely significantly outweigh the cost of a loop iteration.

[ianlancetaylor]

Sorry, my unstated assumption was that you would not do anything on every single loop iteration. Just every 1024 iterations or something.

[aclements]

Oh, I see. That would be fine as a temporary solution, of course. Though in terms of an experiment with adding implicit preemption points in this loop (and in general), I'd rather try something closer to what the compiler would presumably do.