Postmortem of fs.realpath() changes leading to userland breakage

[rvagg]

@nodejs/collaborators

I'm opening this issue so we can look critically at our processes and figure out how we might improve things to avoid things like this in the future.

I'm not totally around the issue but here's the high-level view that I'm seeing

1. fs.realpath 70x slower than native node-v0.x-archive#7902 -> fs.realpath 70x slower than native  node#2680 fs.realpath() is identified as being significantly slower than realpath(3)
2. fs: optimize realpath using uv_fs_realpath() node#3594 introduces a change that cuts the JS implementation and replaces it with a new libuv implementation that directly uses realpath(3) this is deemed to be a breaking change but only because it replaces the cache argument with an options argument (both are Object)
3. fs: optimize realpath using uv_fs_realpath() node#3594 (comment) change is finally landed on the 15th of April
4. fs: optimize realpath using uv_fs_realpath() node#3594 (comment) citgm picks up glob failure on the 22nd of April
5. Discussion and problem dissection takes place, an attempt to fix glob is made @ fix: catch ELOOP isaacs/node-glob#259 but is ultimately deemed by @isaacs to lead to too big a breaking change and is not accepted, although that happened well after v6 went out
6. v6.0.0 is released containing the change on the 27th if April
7. Granular flake support citgm#126 citgm is updated and glob is added as flaky, although the PR doesn't mention glob explicitly
8. Minor concerns are raised post-v6 in the original PR but no further movement is made and discussion ends
9. We get a string of Windows errors that @addaleax is identifying as being rooted in this issue, see Node 6 fs.realpath behavior changes node#7175 (comment)
10. Node 6 fs.realpath behavior changes node#7175 @isaacs opens an issue with a detailed rundown of the problems its causing on the 6th of June, discussion ensues, decisive action is yet to be taken (may change once we discuss at CTC meeting today)

We didn't intend to break anything more than the cache option (and even then, the breakage was not your code won't run breakage, it just won't run quite how you expect unless you're doing something funky with the cache). We discovered that it broke glob's tests. The breakage of glob was marked as ignorable (flaky) and we proceeded with v6. Even though we've had a full postmortem of the problems by now and we've also been able to identify other breakages coming from this, we've still not acted.

Here's some questions to get us going, and let's try to leave discussion about the specific actions on this one to nodejs/node#7175 and focus on process here and see if we can figure out:

1. Clarify the steps that occurred in the chain of actions—is there anything to change about what I've listed above?
2. Do we collectively think anything is broken in our processes
3. If we have process breakage, what can we do to improve?

My personal judgement on (2) is that we have handled this poorly and that something is broken and needs to be identified and fixed. This has delivered a poor experience for Node.js users and we should consider this a black-eye and something to avoid in the future, i.e. a mistake to learn from. I see the breakdown purely as process rather than about any action by individuals. I'm also concerned by the lack of decisiveness on taking action here, we've had v6 out for a couple of months now and we still haven't done anything on this.

The stdio issues are bear some similarity I think. In certain areas we're having difficulty acting decisively to address real user experience issues. One theme that I'm seeing, although not overt and and certainly not by everyone, is a preference for correctness, performance and purity over other concerns. I don't know if this is a real problem because it comes out of the diversity of our collaborator group, but it's possible that having this in the discussion mix is partly responsible for our difficulty in making decisive headway in dealing with problems like this. Maybe we need to more clearly build priorities into the culture we have around core.

When critiquing our actions we should put things in perspective, because overall I think we've done an amazing job since v4 to lift standards and earned an impressive amount of trust and respect from our users. Let's just strive to always do better.

[bnoordhuis]

The stdio issues are bear some similarity I think.

I don't think it does. The realpath thing is a clear v5->v6 regression with a fairly straightforward solution (revert.)

The stdio woes are different: there is no single good solution, plus it's a problem that goes back a long way (v0.6 and possibly even before that.)

[rvagg]

wrt stdio, I'm also including the non-flushed output on exit which was only introduced soon after v1 but we've let that persist even after having reports about it and it being an obvious problem for first-line postmortem debugging.

[kzc]

nodejs/node#6773 fixes flushing stdio upon process.exit() on UNIX, be it tty or piped. Some may deem it to be inelegant or not in spirit with the philosophy of libuv but it's simple and does work as shown by this known-fail test being converted to a normal passing test. Windows does not appear to have this flushing stdio upon process.exit() issue - for piped stdio output at least.

Other approaches to resolve this longstanding bug are welcome, I just put forward that solution to show that it was indeed possible.

nodejs/node#6773 doesn't address truncating stdio upon uncaught exceptions, but it's essentially the same issue and would require a similar fix.

[isaacs]

@bnoordhuis Not to speak for @rvagg, but I think the similarity is not that they're identical issues exactly, but they were similar in the sense of "We thought we understood the scope of a breaking change, but it turns out we did not, and this caused some community unrest which has been challenging to address to everyone's satisfaction."

Clearly, you're right, the technical root cause and the simplicity of the solution are very different in these two cases, so they may call for different fixes. But there's a similar process issue that should probably get addressed, apart from the specific technical details.

Rod, I appreciate you raising this issue and treating it as an opportunity to improve the CTC's processes moving forward. Thank you for this :)

[ChALkeR]

@bnoordhuis

The realpath thing is a clear v5->v6 regression with a fairly straightforward solution (revert.)

Simply reverting nodejs/node#3594 would be a semver-major change, as it will break the encoding option, so the fix (if any) would have to take that into an account.

[kzc]

"We thought we understood the scope of a breaking change, but it turns out we did not, and this caused some community unrest which has been challenging to address to everyone's satisfaction."

process.exit() truncating stdio piped output was a known bug before the node 6.0.0 release with an expected fail test for it:

https://github.com/nodejs/node/blame/master/test/known_issues/test-stdout-buffer-flush-on-exit.js

[bnoordhuis]

nodejs/node#6773 fixes flushing stdio upon process.exit() on UNIX

@kzc With the downside that node blocks indefinitely if the read end of the pipe isn't reading. We may deem that to be an acceptable trade-off but it will certainly trip up some users.

That's what I mean with no single good solution, all the proposed fixes have _some _ drawbacks.

Clearly, you're right, the technical root cause and the simplicity of the solution are very different in these two cases, so they may call for different fixes. But there's a similar process issue that should probably get addressed, apart from the specific technical details.

@isaacs Why I don't think you can compare the two:

1. The realpath change introduced a clear but unintentional regression. There is a straightforward fix but we haven't applied it for $reasons.
2. The issues with stdio are well understood but exist in this uncomfortable twilight zone where one man's bug fix is another's regression. That's been going on for years, it's not new.

It's apples and oranges, IMO. If we're going to do some soul searching, it should be around issues where there is a clear institutional failure, not ones where we're stuck between a rock and a hard place.

[kzc]

With the downside that node blocks indefinitely if the read end of the pipe isn't reading. We may deem that to be an acceptable trade-off but it will certainly trip up some users.

@bnoordhuis How is that different than any other UNIX process with blocking stdout piped to a non-responsive process? When weighed against the present behavior of always truncating stdio output after 64KB upon process.exit() it's certainly acceptable behavior. I don't see any downside.

[bnoordhuis]

You're making a judgment call. You may think it's acceptable behavior but there will most certainly be people who disagree.

[kzc]

My judgement call is based on how most UNIX utilities function. It's expected behavior.

[isaacs]

@bnoordhuis Those are good points.

Would it be possible to redirect the discussion about the similarity to the reaction to fs.realpath changes and stdio changes to a separate thread? Regardless of the difference or similarity in the two cases, I think that @rvagg is bringing up a valid point that is in danger of being derailed.

I think if there's a conversation to be had about what can be learned from the stdio impact, that should maybe be a second discussion. Perhaps it will lead to some overlapping conclusions to this one, perhaps not, but they're probably both useful.

[eljefedelrodeodeljefe]

@kzc imho, that even is very opinionated. UNIX doesn't lend itself well for that kind of determinism. Also your points, imo, are not in scope of this thread.

[kzc]

@eljefedelrodeodeljefe I was just pointing out common UNIX practise. We're discussing why there is resistance to fix a fundamental stdio bug. No other computer language that I'm aware of truncates stdio output upon exit or uncaught exception. It goes against the principle of least astonishment.

[eljefedelrodeodeljefe]

@kzc again, stdio issues are not the main topic of this thread. It could have been left out in the description, but it hasn't, to point out a more managerial issue. The realpath issue is big enough, let's give it some room here.

I trust @bnoordhuis, @isaacs et al to generally be very practical (note this as opposed to philosophy) about this one issue and then move to the next one.

Thx.