Change default EOL to "any" for input files?

[rmkaplan]

I have added a new EOL convention constant, ANY.EOLC, and updated the code for EOL interpretation so that it will return (CHARCODE EOL) if it sees any of LF, CR, or CR followed by LF if the EOL convention of the stream is ANY.EOLC.

I propose to make that the default EOL convention for input streams if there is no explicit EOL specification in the open-stream parameters. This would change the current default, which is to use the EOL convention of the file device. This would be a minor change to \DO.PARAMS.AT OPEN.

The EOL convention for output streams would not change--the current default (LF) would persist.

This change is intended to reflect the fact that the provenance of existing files is typically unknown and not necessarily correlated with the convention of the current environment.

Is there any reason not to change the input default?

[nbriggs]

What would (GETFILEINFO X 'EOL) return on such a file?

[rmkaplan]

If called on an open stream, it would return ANY if that’s what is set there. An extra little cost-free twist could be to have the EOL interpretation code change ANY to a more specific value after it sees and decodes the first EOL instance. This assumes that the file would be consistent and that the first instance predicts everything that follows. If called on a (non-open) file name, it would return whatever the file-device’s GETFILEINFO operation would return, presumably the default for the device, as now. Because it actually can't know what’s going on in the file itself, without opening and looking.

…

On Jun 23, 2021, at 10:35 AM, Nick Briggs ***@***.***> wrote: What would (GETFILEINFO X 'EOL) return on such a file? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJMRDOPNLYDSBOEQGCTTUILM7ANCNFSM47GHOJKQ>.

[nbriggs]

While I like the idea, I suspect there will be a number of SHOULDNTs because LF/CR/CRLF are currently the known values, and it breaks the assumption that the result of (GETFILEINFO X 'EOL) is a suitable argument to CHARCODE

[rmkaplan]

Apart from ANY, I see something else in the code that doesn't look right to me. In the CRLF case, it reads the first character (CR) using the external format function for reading a single character, possibly seeing the shift bytes to CSET 0 in a runcoded XCCS file. But then (at least in \RSTRING2) it looks for the line feed by a simple BIN or PEEKBIN, not using the format function.

That is OK for a runcoded XCCS stream and for UTF8 and any other single-byte-for-Ascii format, but I don't think it would be correct for a 2-byte (non-runcoded) XCCS stream (where the next byte is 0), and I presume it also wouldn't be correct for UTF16.

So I think it should look for the LF in the generic way, not optimize for a byte in a file that it thinks is CRLF.

[rmkaplan]

On CHARCODE… I’m not so sure about preserving this behavior. It works meaningfully for CR and LF, but (CHARCODE CRLF) returns 12, the same as FORM and PAGE. (And looking at CHARCODE.DECODE, I don’t see how it even comes up with that). CHARCODE could return an equally arbitrary value for ANY, if that’s an important consideration.

…

On Jun 23, 2021, at 11:03 AM, Nick Briggs ***@***.***> wrote: While I like the idea, I suspect there will be a number of SHOULDNTs because LF/CR/CRLF are currently the known values, and it breaks the assumption that the result of (GETFILEINFO X 'EOL) is a suitable argument to CHARCODE — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJNMO4RM5Z24L4Q6DV3TUIOWTANCNFSM47GHOJKQ>.

[rmkaplan]

The constant ANY.EOLC is defined, and \CHECKEOLC will interpret it. So I believe the capability is there, but I don't think that it is specified in all the places where it might make sense to be agnostic about the EOL convention of input files. Probably the cases have to be considered one at a time.

[masinter]

isn't it possible to reduce the number of cases to the ones actually encountered?
e.g. link the EOLC to the EXTERNALFORMAT + readtable rather than allowing it to be a third independent variable?

[nbriggs]

What about the external format and readtable tells you that you're on a PC and want a different EOLC than your Unix box? Or that you've got an NTFS volume mounted on your Unix box and need the PC style EOLC?

[masinter]

EXTERNALFORMAT: UTF-8 implies EOLC LF
EXTERNALFORMAT: XCCS implies EOLC CR
EXTERNALFORMAT: US-ASCII implies EOLC CRLF

On input the readtable’s escape character hides the next character read from EOL conversion
On output (in print and terpri etc) any internal EOL is output according to the EXTERNALFORMAT's EOLC.
Other combinations can be set up by defining new EXTERNALFORMATs.

External format is chosen from the preferred name in the IANA character encoding set registry (where I will register XCCS).

Is this consistent with current behavior?

[nbriggs]

Except that you're adding implications that aren't actually part of the encoding standard. UTF-8 is compatible with ASCII in the control characters, both of them have CR and LF, and it's the platform that you're running on that determines which EOL convention it will use not whether you're using the ASCII subset of UTF-8. The fact that Xerox systems used XCCS for their character encoding wasn't what drove the use of CR rather than LF as the EOLC.

[nbriggs]

The EOL field of an EXTERNALFORMAT, if not NIL, is an override of what the stream would have had (according to Ron's documentation) for output.

If the EOL of any of the predefined formats is not NIL then they become mostly useless and you have to have your own logic for which external format you're writing with -- depending on the system and/or filesystem that you're writing to, so that you can create a new (or choose an existing) external format with the "correct" EOL convention.

[nbriggs]

@rmkaplan --

On CHARCODE… I’m not so sure about preserving this behavior. It works meaningfully for CR and LF, but (CHARCODE CRLF) returns 12, the same as FORM and PAGE. (And looking at CHARCODE.DECODE, I don’t see how it even comes up with that). CHARCODE could return an equally arbitrary value for ANY, if that’s an important consideration.

CHARCODE.DECODE looks at CHARACTERNAMES (which should probably, for completeness, have all the standard control character names, see RFC20)

[rmkaplan]

I think there is a cross product of different factors: input vs. output file external format that does or does not care about the EOL convention on its files default eol convention in the local environment (or more specifically, on the file device that the file is being written on/read from) The problem not recognized in the original EOLC design is that files move around from environment to environment, and their provenance gets lost over time. ftp was set up to do eol conversion if you declared a file of type text, but that of course wouldn’t work for interlisp source files if going from LF or CR to CRLF—filemaps would be off. For output source files, the define-file-info now contains the reader-environment, which includes an EOLC field to say what EOL convention to use. Some formats may really care how the EOL is represented, in which case the EOL convention in the format should override whatever is the local default convention. For example, if some Japanese format really wants CRLF and you are running in an LF environment, still you want the file to have CRLF. In what we inherited, the externalformat also has a field EOLVALID which I think must have been intended to address this situation—if EOLVALID is set, then a file with that format should use the EOLC of the format no matter what the local default is. (I’m not sure it is now, or ever was, implemented this way, but I can now see the point of it.) Separately, there should be a site-greeting variable DEFAULT-EOLCONVENTION (LF on a mac) that is picked up when the format is created and its own EOL isn’t specified. Maybe there is, I should look for it—but see the note below. That’s for output. It is a little tricker for input files, which is really what this issue is about. For files whose provenance is unknown and that either don’t have a reader environment or don’t have a reader environment whose EOLVALID is true, then what bytes on the file should we map to the internal EOL? (Let’s presume that a file with an EOLVALID reader-environment is modern and hasn’t been mangled by ftp-like transfers, otherwise even that scenario is hopeless.) It will usually be safe to blindly assume that such a file was created with the default EOL convention of the local environment, because CR and LF are typically seprs and most source files wil be insensitive to whatever convention is in play (hopefully no CR/LF distinctions are coded in strings and atoms). But not so other kinds of files (plain text), and not so for all operations on source files (e.g. PF) that don’t just read and write. So what is the more-safe approach that works for a wider range of files? The ANY.EOLC proposal as the default for input files basically says that if it looks like it could have been created as an EOL in some environment then we will treat it as an EOL in this environment, no matter what our local convention is. This would be for files that are not overridden by an EOLVALID format or by an EOL specification in the OPENSTREAM parameters. This would cover more of the EOL-unknown cases in a way that almost always safely captures the intent of the file. It could still go wrong in uncomon cases, for example the case of a file that was made with CR but that actually contains a CRLF byte sequence that is intended to be recognized as EOL followed by LF. If you knew that you had such a file, then you would have to provide a more specific EOL in the opening. —— One additional issue that this discussion reveals is that the notion of the “local environment” should also be staged. We should ad a field to the filedevice that gives the default to be used on that device, and that would override anything that may be the background default for the site (e.g. the default convention variable). On open for output, the stream should consider the priority order as valid external-format default for device default for site

…

On Aug 7, 2021, at 11:07 AM, Nick Briggs ***@***.***> wrote: What about the external format and readtable tells you that you're on a PC and want a different EOLC than your Unix box? Or that you've got an NTFS volume mounted on your Unix box and need the PC style EOLC? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJKTCAZJ6JCA4HU3PNTT3VY5FANCNFSM47GHOJKQ>.

[nbriggs]

I think I'd also add a possibility: the same EOLC for output as was present on input (in the previous version of the file)