First column not accessible via property name in case of BOM mismatch

[jacobq]

Describe the bug

When parsing a CSV file that includes BOM using the columns: true option, serializing the record shows the expected values, but the first column is not accessible by its key name. Setting option bom: true fixes the problem (and I'd love to tell that to my 2-hours-ago self), but why can't the default behavior deal with this case correctly?

• csv-parse@4.8.8
• stream-transform@2.0.1

To Reproduce

test.csv (input)

X
1
3

test.js (program)

const { createReadStream } = require('fs');

const parse = require('csv-parse');
const parser = parse({
  columns: true,
});
const transform = require('stream-transform');
const transformer = transform(function(record, callback) {
  callback(null, `X=${record.X}, record=${JSON.stringify(record)}\n`);
}, {
  parallel: 5
});

createReadStream('test.csv')
  .pipe(parser)
  .pipe(transformer)
  .pipe(process.stdout)
  ;

Execution / Demo

(Git Bash / MinGW64)

$ file test.csv
test.csv: UTF-8 Unicode (with BOM) text, with CRLF line terminators

$ node test.js
X=undefined, record={"X":"1"}
X=undefined, record={"X":"3"}

$ dos2unix.exe test.csv
dos2unix: converting file test.csv to Unix format...

$ file test.csv
test.csv: ASCII text

$ node test.js
X=1, record={"X":"1"}
X=3, record={"X":"3"}

Additional context

• CSV file was created with Microsoft Excel 2016 MSO (16.0.12527.20260) 64-bit using the "CSV UTF-8 (Comma delimited) (*.csv)" format
• OS: Windows 10 64-bit (Version 10.0.18363.720)
• node: v12.16.0
• npm: 6.13.4

[bok-]

I just hit this as well, so you just saved me the two hours you lost 🙏

• CSV was exported from Notion.so
• OS: macOS 10.15.3
• csv-parse: v4.8.8
• node: v13.6.0
• npm: 6.13.4

[wdavidw]

Thank you guys, that is a nice one. Still looking for an explanation...

[wdavidw]

OK, The bom option default to false and with the column option, it ends up inside the property name. Setting the bom option to true solve the issue. In your example, the property length is not 1 but 4 bytes, but other people with alternative options will get those bytes inside the value as well.

What shall we do? For now, this behavior is by design, we intend to not be intrusive and not interpret the bom if we are not told so. The parser is encoding agnostic. We could leave it this way and improve the doc. Or we could discover and remove the bom but we have no way to know that the input is 'utf-8' encoded and I am not too enthusiastic about such intrusive behavior. I am open to discuss it.

[wdavidw]

I have added a section Hidden BOM in output in the doc.

[jacobq]

Thanks for clarifying the docs. What I really wanted to understand was why it was looking OK in the serialized output. I figured it was some hidden character, but when I tried prepending \xFE\xFF (and the reverse) and it didn't seem to work then I gave up. Now I see I should've tried \uFEFF instead.

[Tomabana]

@wdavidw Thanks for clarifying this. We were also struggling with this problem (adding BOM to the first column name).

One question, because I'm not sure if I understand it correctly.

Or we could discover and remove the bom but we have no way to know that the input is 'utf-8' encoded

If you delete BOM you can just save the info that it was present and treat the file as before (as if BOM was there). Or is this something connected with your design and you just can't store this info?

[wdavidw]

The parser doesn't know the encoding, it just deal with bytes. The part of the code reading the file shall know the encoding and, if utf8, strip the bom itself or indicate the parse to do it for him. We provide the bom option for conveniency but we shouldn't affect the parser behavior unless requested to do so with this option.

[RaoufYounesBuzzy]

@jacobq you're a lifesaver, I still think the option should be made more clear on the website.

[wdavidw]

Please provide a pull request on the documentation website with your suggestions. That would be great. There is a recipe to deal with the filesystem.

[hadyrashwan]

Well, I spend over that 2 hours looking for an explanation. Glad I found it.

[wdavidw]

@hadyrashwan Does the doc on hidden bom helps (you probably didn't find it before landing on this issue) ?

[hadyrashwan]

Yeah it helps. But I wasn't sure at first was it from my side or the library. So I went straight to the github issues afterwards.