File encoding is not maintained when first line of file is a conditional statement

[ajaybhargavb]

The file generated after doing dotnet new has a different encoding than what was in the original file for the template.

Repro:

• Install dotnet 3.1 sdk
• Unzip C:\Program Files\dotnet\templates\3.1.0\microsoft.dotnet.web.projecttemplates.3.1.3.1.0.nupkg and go to Web.ProjectTemplates/content/BlazorServerWeb-Csharp/
• Verify the encoding of App.razor (It should be UTF-8 with BOM)
• Now create a new app with dotnet new blazor
• Open the generated App.razor and check its encoding

Expected:
UTF-8 with BOM

Actual:
Windows Codepage 1252

cc @phenning

[phenning]

This looks like a bug when the first line of the processed file contains a conditional construct such as #if.

I just verified that either adding a blank line to the top of the file, or moving the #if to the next line as follows results in the encoding being preserved.

@*
#if (NoAuth)
<Router AppAssembly="@typeof(Program).Assembly">

@ajaybhargavb This could be used as a a workaround in the template content until the root cause of underlying bug is found and fixed.

[ajaybhargavb]

@phenning,

@*
#if (NoAuth)
<Router AppAssembly="@typeof(Program).Assembly">

doesn't work.

Adding a newline before the @* works but it ends up adding a blank line at the top of the generated file.

[phenning]

You are right, I wasn't looking at the generated file correctly with that version of the workaround.

[ajaybhargavb]

Is there a workaround that wouldn't add a newline at the top of the generated file?

[mkArtakMSFT]

@phenning I'm concerned regarding taking a workaround for this in a template as that would be a per-file fix, and we can potentially run into this issue again in the future.
What about treating /forcing all the files to be UTF8? This should be very small change in the templating engine.
//cc @pranavkm

[phenning]

That seems rather heavy handed, and is not the right fix either.

Not all files are UTF-8. The right fix should be to fix the fact that the BOM gets stripped when the first line contains a conditional statement. The conditional itself gets stripped, so there is likely some sort of issue with that code when that line happens to be the first line in the file after the BOM.

AFAIK, this is the only file in any of the .NET Core templates which have a conditional on the very first line of the file.

[mkArtakMSFT]

AFAIK, this is the only file in any of the .NET Core templates which have a conditional on the very first line of the file.

That is probably true but the fact that future changes in templates can potentially re-introduce this issue make me believe that fixing the file is not a good way moving forward. I agree, that for now it'll work, but the right fix you mention is required too.

[phenning]

Also, for what it's worth, the encoding is preserved, only the BOM is removed. This template file though does not contain any non-ANSI characters. To verify this, I inserted a Chinese character into the template file locally, and while the BOM was removed when the project was generated, the generated file was still UTF-8 encoded.

[ajaybhargavb]

Also, for what it's worth, the encoding is preserved, only the BOM is removed. This template file though does not contain any non-ANSI characters. To verify this, I inserted a Chinese character into the template file locally, and while the BOM was removed when the project was generated, the generated file was still UTF-8 encoded.

@phenning this is not what we observed. In fact, this bug was discovered because the generated file was not UTF-8 (with or without BOM). Right now if you run dotnet new blazor with a 3.1 SDK, and check the encoding of App.razor you'll see that it is Windows Code Page 1252 and not UTF-8. The encoding of the original template file is UTF-8 BOM.

[phenning]

@ajaybhargavb The template as it exists now does not have any UTF-8 characters in it, so without a BOM, it would be detected as an CP-1252 file when it is generated.

I manually updated the App.razor file within the template's nupkg in my %userprofile%.templateengine\dotnetcli\v3.1.100\packages folder to include a Chinese character; I just added one to the first <p> tag in the first block.

The generated file showed as having UTF-8 encoding (but without a signature)

The bug here is that the BOM does not get preserved when a conditional is on the first line of the template file.

[ajaybhargavb]

In that case, isn't the real bug still is that the actual encoding (not just BOM) isn't preserved right?
E.g, if my template file is UTF8 without BOM and it doesn't contain any unicode characters yet. Then the templating engine would generate it as CP-1252 instead of UTF8. But we would want it to preserve UTF-8 regardless of whether any unicode characters are present in the original template or not.

Am I missing something?

[phenning]

In the case of the absence of the signature, it is up to the consumer of the file to determine whether a file should be treated as UTF-8 (or not), as by looking at the contents of the file - a file can still still be treated as UTF-8 even if it only contains ANSI characters - but thats up to the editor/consumer.

For example: VS Code detects and treats this file as UTF-8:

While notepad2 detects it as ANSI (1252):

To further illustrate, if I click on UTF-8 in the toolbar in VS Code and resave the file as 1252, then close and reopen it, it still shows up as UTF-8 in VS Code. Opening it in notepad2 will show it as being "ANSI". I can do the same in notepad2, that is explictly change the encoding to UTF-8 and resave; if I reopen in either editor, it will show up as UTF-8 in VS Code and ANSI in notepad2.

Also a quick search on Stack Overflow show much confusion on whether to, and when, to use a UTF-8 signature and on which file types. Regardless of that though, I agree that the we should preserve the UTF-8 signature, if present, in the template file when generating the file.