C# source files should have a byte order mark

[sharwell]

Currently source files in this repository are inheriting an unexpected charset value of utf-8 from the [*] group of .editorconfig:

aspnetcore/.editorconfig

Lines 16 to 24 in 2121105

	[*]
	indent_style = space
	charset = utf-8
	trim_trailing_whitespace = true
	insert_final_newline = true
	[*.cs]
	indent_size = 4
	dotnet_sort_system_directives_first = true

C# source files should be encoded with byte order mark (charset = utf-8-bom) to avoid cases where an editor incorrectly interprets the files as an incompatible system default encoding (e.g. ANSI).

I'm happy to submit a pull request to normalize the files in this repository, at which point it would match other repositories like dotnet/runtime and dotnet/roslyn.

[Pilchie]

I thought the current preferred was UTF-8 without BOM because BOMs are fairly windows specific?

[mqudsi]

It would be great if Visual Studio didn't stomp over all UTF-8 files and add a BOM against all current recommendations. I actually use a Visual Studio extension to strip the BOM from all files, as well as a git pre-commit hook to complain if a BOM makes it through - they cause all sorts of problems on non-Windows applications/hosts.

[sharwell]

I thought the current preferred was UTF-8 without BOM because BOMs are fairly windows specific?

BOM would be considered OS-specific in any case where an OS uses a Unicode encoding by default for the OS worldwide. Problems arise when the BOM is missing in cases where the OS/application is capable of inferring an encoding which is compatible with UTF-8 for the code points currently in the file, but is not compatible for all code points. Over time, I've seen various manifestations of incorrect file contents resulting from this.

they cause all sorts of problems on non-Windows applications/hosts

This applies in some cases, but not for C# source files. All C# compilers on all platforms support byte order marks, as they have been the default for this language since its first release. This issue only relates to the encoding of these files within the repository.

and add a BOM against all current recommendations

I'm assuming you are referring to the Unicode Consortium's FAQ suggesting that the byte order mark is unnecessary for UTF-8 encodings. This recommendation is derived in a world where all possible encodings are Unicode compatible, in which case the guidance is certainly true. However, the reality of computing today includes many encodings that are not Unicode compatible, and the strongest mechanism currently available to avoid misinterpretation of text data by an application is to include a byte order mark in source files that allow it.

[mqudsi]

Thanks for taking the time to flesh out your stance further. I appreciate that first-party utilities ingesting C# source files may have been written with BOM in mind, but that assumes a .cs file is a black box both generated and consumed solely by first-party utilities. The truth is that .cs files are text files and may be used with the entire arsenal of text-processing tools and utilities - where even those that use unicode by default will have problems handling a BOM. It wouldn't be wrong to stamp all source files with a BOM, but it would be highly inconvenient and needlessly burdensome to the developers in the community with little gain.

I'd like to point out that there's a partial inconsistency in your argument for BOM: the argument against "it can cause problems" is that "all C# compilers support it" but the argument against removing it is that "many encodings [used by programs that are not c# compilers] that are not Unicode compatible" - but if we're only looking at the picture so far as C# compilers on all platforms are concerned, then any encoding not already natively supported by a C# compiler doesn't matter (i.e. only need to disambiguate between codepages supported by C# compilers, which unless I'm mistaken, is exclusively Unicode).

However, the reality of computing today includes many encodings that are not Unicode compatible, and the strongest mechanism currently available to avoid misinterpretation of text data by an application is to include a byte order mark in source files that allow it.

Of course the byte-order marker under discussion (\uFEFF) is part of the Unicode spec and is not defined to have any particular meaning outside it, i.e. an application that isn't Unicode aware isn't going to ascribe any particular meaning to it or benefit by its presence. Keeping that in mind:

OS/application is capable of inferring an encoding which is compatible with UTF-8 for the code points currently in the file, but is not compatible for all code points. Over time, I've seen various manifestations of incorrect file contents resulting from this.

I'm guessing this has to do with UCS-2? In all cases, this isn't a case that an optional BOM for UTF-8 documents would prevent.

Currently, it seems the dotnet frontend to Roslyn only supports UTF-8 with/without a BOM and UTF-16 with a BOM (i.e. no BOM == assume UTF-8). Again, continuing from the C# compilers perspective: there's no gain to using the BOM with UTF8 because it doesn't support any other BOM-free encoding (if I were to use ISO-8859-1 (aka latin1), it's not going to work anyway). (Notably - but neither here nor there - UTF-32 is not supported with or without a BOM.)

With UTF-16, a BOM serves a very crucial role in distinguishing between UTF-16LE and UTF-16BE. In fact, that's why the BOM exists (along with distinguishing between UTF-32LE and UTF-32BE). But UTF-8 is byte-oriented and thankfully Big Endian/Little Endian does not come into play.

Basically, there's no good argument for using an optional BOM with UTF-8 if you're already going to support UTF-8 without a BOM (which everyone really must for ASCII compatibility) and do not support UTF-16/UTF-32 without a BOM (which isn't even legal unless you explicitly request/specify an Endianness). The most important thing is that all UTF-16 documents are decorated with a BOM and that unadorned UTF-16LE (and UTF-16BE) encodings are absolutely not supported (which is where all the UCS-2 problems come in).

Thank you for contacting us. Due to a lack of activity on this discussion issue we're closing it in an effort to keep our backlog clean. If you believe there is a concern related to the ASP.NET Core framework, which hasn't been addressed yet, please file a new issue.

This issue will be locked after 30 more days of inactivity. If you still wish to discuss this subject after then, please create a new issue!