Implement emitting utf8 charset when seen in output

[mgreter]

#629: "when we run into utf-8 characters, we throw in the charset"

There are some unknowns and open things here:

• Does ruby sass support other encodings than utf-8
• We don't, so anything else is out of perspective here!
• IMO this should be configurable via a context option -> optional
• Code needs some re-factoring (use less copy paste) -> todo Unify common code for different output styles #734
• Needs to update spec/basic/48_case_conversion -> Regenerate test since libsass adds @charset sass-spec#178

[KittyGiraudel]

I was myself just encountering an issue where @charset "UTF-8" does not get printed, even if explicitly asked for it.

Shall I open another issue?

[mgreter]

What do you mean by "explicitly asked for it."?

[KittyGiraudel]

Test:

@charset "UTF-8";

test {
  content: "øáéíóúüñ¿éàŤDžǂɊɱʭʬѪ҈ݓ";
}

Expected:

@charset "UTF-8";
test {
  content: "øáéíóúüñ¿éàŤDžǂɊɱʭʬѪ҈ݓ"; }

Result:

test {
  content: "øáéíóúüñ¿éàŤDžǂɊɱʭʬѪ҈ݓ"; }

[xzyfer]

FYI as of Sass 3.4.

Sass now follows the CSS Syntax Level 3 specification for determining a stylesheet’s encoding. In addition, it now only emits UTF-8 CSS rather than trying to match the source encoding.

[mgreter]

👍 Thanks for the info! So I guess this implementation is on par with ruby sass! If we see a char above 127 we will output the utf-8 charset declaration, simple as that!

[xzyfer]

There's more detailed information here - https://github.com/sass/sass/blob/28ce632750571de900a4471c4c97d268d1f847a2/doc-src/SASS_REFERENCE.md#encodings

When running on Ruby 1.9 and later, Sass is aware of the character encoding of 
documents. Sass follows the CSS spec to determine the encoding of a stylesheet, 
and falls back to the Ruby string encoding. This means that it first checks the Unicode 
byte order mark, then the @charset declaration, then the Ruby string encoding. If none 
of these are set, it will assume the document is in UTF-8.

To explicitly specify the encoding of your stylesheet, use a @charset declaration just 
like in CSS. Add @charset "encoding-name"; at the beginning of the stylesheet (before 
any whitespace or comments) and Sass will interpret it as the given encoding. Note 
that whatever encoding you use, it must be convertible to Unicode.

Sass will always encode its output as UTF-8. It will include a @charset declaration if 
and only if the output file contains non-ASCII characters. In compressed mode, a UTF-8 
byte order mark is used in place of a @charset declaration.

[HamptonMakes]

Tres simple!

On Thu, Dec 11, 2014 at 3:32 PM, Marcel Greter [email protected]
wrote:

👍 Thanks for the info! So I guess this implementation is on par with ruby sass! If we see a char above 127 we will output the utf-8 charset declaration, simple as that!

Reply to this email directly or view it on GitHub:
#717 (comment)

[mgreter]

Unfortunately the link @xzyfer posted makes this related to #381 🙊
Since we probably will not have other encodings than Unicode (utf8 and maybe utf16) antime soon, I guess we can go on with the current implementation!? IMO this could be a task for implementors to resolve. I know this would be fairly easy with perl-libsass and custom importers, since I have all the necessary encoding/decoding rules available in perl.

[mgreter]

OK, how do we proceed here? IMO we could ship this as-is with 3.0.3. To use less copy & paste we will need to create a common class for each output style. I already did that, but I think this needs a lot more work and fits perfectly into the source_map task, which I also have started some time ago.

So I would propose to ship this as-is and open two new issues:

• Add option to make utf8 charset output optional
• Create a parent class for compressed and nested output styles to unify common code

The second issue will be a pretty big task, since I think we should unfiy all the source mapping code into this class, so the actual implementation only passes "stuff" the be in the output to it. It basically should handle the buffer and create the source map automatically.

The only downside with the current code is that it repeats the same code block 5 times and that the feature is not optional (does ruby sass have a switch for that?). But it will not break working css!

I guess I let you decide on this one!? But I don't think I can get this in in time otherwise!

// CC @xzyfer, @hcatlin

[xzyfer]

AFAIK Ruby sass offer no options regarding @charset. It simply emits the utf-8 charset if it detects utf-8 characters. IMO this doesn't need to be configurable via a context option.

As for the state of the code I'm fine with it. I agree we should create an internal issue to come back around to this. Presumable when we tackle the expanded output mode.

[mgreter]

Just fixed a bug I haven't seen. Charset should be the very first thing in the css, so it should be after regular css imports. Therefore I had to move the place where the charset is added to the get_buffer function, since otherwise generated_imported would be the first in output.

[xzyfer]

LGTM

[coveralls]

Coverage decreased (-0.1%) when pulling dd2c068 on mgreter:feature/utf8-charset into 92e19f1 on sass:master.