proposal: unsafe: clarify unsafe.Pointer rules for package syscall

[mdempsky]

unsafe.Pointer's rule 4 says:

Conversion of a Pointer to a uintptr when calling syscall.Syscall.

The Syscall functions in package syscall pass their uintptr arguments directly to the operating system, which then may, depending on the details of the call, reinterpret some of them as pointers. That is, the system call implementation is implicitly converting certain arguments back from uintptr to pointer.

It talks about "the Syscall functions", but on Windows, there's also Proc.Call and LazyProc.Call.

Q1: Are these two functions "Syscall functions"? Strictly speaking, I would interpret "Syscall functions" to mean Syscall{,6,9,12,15,18}. Perhaps the docs should be clarified.

Q2: Do functions have to be called directly, or are indirect calls allowed? The rule explicitly warns that conversions have to be performed directly in the argument list, but it doesn't say anything about indirect calls.

Q3: For Proc.Call and LazyProc.Call, are direct calls via method expressions allowed? E.g., if x.Call(uintptr(p), 0, 0) is valid, then is (*Proc).Call(x, uintptr(p), 0, 0) also valid?

Clarifying this is relevant to determining how far we need to go in fixing #34474.

Incidentally, Q2 and Q3 also apply to rule 5. The status quo there is that cmd/compile safely handles unsafe.Pointer(f(...)) for all f(...), so we do allow indirect and method expression calls to reflect.Value.Pointer and reflect.Value.UnsafeAddr.

/cc @rsc @ianlancetaylor

[ianlancetaylor]

The part you quoted is the rationale. The real rule is in the part you didn't quote:

The compiler handles a Pointer converted to a uintptr in the argument list of a call to a function implemented in assembly by arranging that the referenced allocated object, if any, is retained and not moved until the call completes, even though from the types alone it would appear that the object is no longer needed during the call.

For the compiler to recognize this pattern, the conversion must appear in the argument list:

My answers, such as they are, to your questions.

Q1: Are these two functions "Syscall functions"? Strictly speaking, I would interpret "Syscall functions" to mean Syscall{,6,9,12,15,18}. Perhaps the docs should be clarified.

Per above, the rule applies to any function implemented in assembly. syscall.Proc.Call and syscall.LazyProc.Call are not implemented in assembly, and, as such, this rule does not apply to them.

Q2: Do functions have to be called directly, or are indirect calls allowed? The rule explicitly warns that conversions have to be performed directly in the argument list, but it doesn't say anything about indirect calls.

I think the functions have to be called directly, and we should update the unsafe package docs to say that.

Q3: For Proc.Call and LazyProc.Call, are direct calls via method expressions allowed? E.g., if x.Call(uintptr(p), 0, 0) is valid, then is (*Proc).Call(x, uintptr(p), 0, 0) also valid?

I suppose that first we have to come up with a rule for Proc.Call and LazyProc.Call.

When I introduced go:uintptrescapes for #16035, my thinking was basically that there was no general rule that could make that code work, but since it already existed and people were using it we should at least try to keep those programs working. It wasn't intended to introduce a new general unsafe.Pointer exception, just one for those two methods.

[mdempsky]

The part you quoted is the rationale. The real rule is in the part you didn't quote:

Hm, I've always interpreted the first line as the rule, and everything below as the rationale, restrictions, and details. For example, that rule 4 is you can convert unsafe.Pointer-to-uintptr when calling syscall.Syscall*, and the part you describe as the real rule is just an explanation of the implementation details about how the compilers implement this today (i.e., that syscall.Syscall* are implemented as assembly functions, and the compilers generically handle calls to assembly functions this way).

If rule 4 is in fact that all assembly functions should be handled that way (with syscall.Syscall as just a special case thereof), I think we should reword the first sentence to better emphasize that.

[ianlancetaylor]

Yes, my interpretation is that this applies to all functions written in assembly, since the rule clearly applies to more than the single function syscall.Syscall. But perhaps I am mistaken.

[beoran]

I think making //go:uintptrescapes work correctly for these functions is probably the best way to go, since there really isn't a general rule, it depends on semantics of the OS call.

[mdempsky]

@beoran The issue here is we've never defined exactly what it means for //go:uintptrescapes to "work correctly". As far as I can tell, it's not documented anywhere (e.g., neither cmd/compile's godocs, nor src/runtime/HACKING.md).

I've generally been under the assumption that (1) rule 4 is meant to apply specifically to package syscall's functions like Syscall, RawSyscall, Proc.Call, and LazyProc.Call, and (2) //go:uintptrescapes and cmd/compile's semantics for calling assembly functions are implementation details about how we guarantee rule 4 today.

For example, I'm under the impression that while today users can write their own assembly functions that pass pointers as uintptr or //go:uintptrescapes-annotated Go functions that do the same, we do not guarantee this to work in the future and reserve the right at any point to break this (as long as package syscall's Syscall functions continue to work correctly).

However, it seems at least @ianlancetaylor had a different interpretation.

[rsc]

My thoughts, not definitive:

Q1: It seems like the answer must be yes or else those functions are impossible to use safely at all. That's the same reason we answer yes to syscall.Syscall.

Q2: It would be nice if indirect calls worked, but it seems like they probably can't be made to work without significant overhead, and it also seems like they don't work today. If both those are true, then probably we should say that indirect calls don't apply.

Q3: In general there's almost zero difference between x.M() and T.M(x), so I don't see why we'd introduce a difference here.

So I guess I'm suggesting yes/no/yes.

[mdempsky]

Thanks for sharing your thoughts, @rsc.

Q2: It would be nice if indirect calls worked, but it seems like they probably can't be made to work without significant overhead, and it also seems like they don't work today. If both those are true, then probably we should say that indirect calls don't apply.

It should only introduce overhead for indirect calls involving unsafe.Pointer->uintptr conversions as arguments, and I don't think those are very common. So I don't expect the overhead would be significant, but I'll measure this.

Q3: In general there's almost zero difference between x.M() and T.M(x), so I don't see why we'd introduce a difference here.

That's my tentative position as well. The two counter arguments here though are (1) historically it hasn't worked; and (2) package unsafe imposes its own set of rules, and maybe we think T.M(x) is uncommon enough to not allow it.

[mdempsky]

Across the standard library and Kubernetes, the only indirect call that involves a unsafe.Pointer->uintptr conversion is here:

go/src/syscall/syscall_unix.go

Lines 95 to 98 in 32b6eb8

	// Unmap the memory and update m.
	if errno := m.munmap(uintptr(unsafe.Pointer(&b[0])), uintptr(len(b))); errno != nil {
	return errno
	}

(And the same code appears in x/sys/unix.)

For this particular call, the overhead would amount to an extra stack slot and an instruction to populate it.

[gopherbot]

Change https://golang.org/cl/200137 mentions this issue: cmd/compile: warn about indirect calls with unsafe.Pointer->uintptr conversions