Remove text/html; fragment from accept header

[koddsson]

This is a breaking change and would need a major version bump.

Varnish will choke on ; in a accept header so we need to remove it and only rely on the text/fragment+html header.

[dbussink]

Thanks for quickly jumping on this.

[keithamus]

To clarify; Varnish does not choke on this - nginx does. See #36 (review):

👍

I'd like to put it here for posterity:

The Accept header, as defined in RFC2616 Sec 14.1 does allow for text/html; fragment. An Accept header consists of a media-range (which is a media-type where the mime can be or type/* or */*) followed by any number of accept-params (which are ; token or ; token = "string" ). The Content-Type header (defined by RFC2616 Sec 14.17) allows for a media-type (defined in Sec 3.7) which looks much the same but cannot use the media-range wildcards.

text/html; fragment is a perfectly valid media-type, therefore a perfectly valid Content-Type header and also a valid Accept header.

The reason this needs to be done is that for certain parts of Nginx - namely the gzip module, and how it determines it can gzip content - only accepts mime-types not media-types. Mime Types are different to Media Types and only allow for a grammar of type/subtype or type/extension+subtype (as defined in RFC2045).

In other words, when nginx sees a Content-Type header of text/html; fragment - it discards it as an invalid Mime Type and will not gzip it. Nginx is both wrong and right here - text/html; fragment is not a valid mime type! But Content-Type is not a header that contains only mime types! So nginx is not following the spec.

Getting nginx to follow the spec and have gzip allow lists use Media Types over Mime Types is an uphill battle compared to simply changing our Media Type to be one that is acceptable as a Mime Type, hence this change.

[koddsson]

To clarify; Varnish does not choke on this - nginx does. See #36 (review):

Ah yes, my mistake. I keep conflating the two.

[keithamus]

After speaking with @dbussink, normalisation is present in our VCL so this is actually Varnish causing this issue. That being said the nginx issue may still be prevalent here.