[SR-7195] JSONSerialization encodes Double differently on Darwin and Linux

[florianreinhart]

Previous ID	SR-7195
Radar	None
Original Reporter	@florianreinhart
Type	Bug
Status	Resolved
Resolution	Done

Environment

Swift 4.0.3, macOS 10.13.3 and Ubuntu 16.04

Additional Detail from JIRA

Votes	0
Component/s	Foundation
Labels	Bug
Assignee	None
Priority	Medium

md5: e38af2c3f6bfc9b1fe10fcfba2bf9763

relates to:

• SR-5961 JSONSerialization does not properly convert 1.1

Issue Description:

`JSONSerialization` and thus `JSONEncoder` encode floating point values in a different way on Linux than on Darwin. This can cause a loss of precision and decoding a different value than the one that was encoded on Linux.

Running the following code demonstrates the issue.

import Foundation

let value1 = 7.7087009966199993
let value2 = 7.7087009966200002

// Encode using JSONSerialization
do {
    print("Encoding using JSONSerialization")
    let dict1 = ["value": value1]
    let dict2 = ["value": value2]
    let jsonData1 = try! JSONSerialization.data(withJSONObject: dict1)
    let jsonData2 = try! JSONSerialization.data(withJSONObject: dict2)
    let jsonString1 = String(decoding: jsonData1, as: UTF8.self)
    let jsonString2 = String(decoding: jsonData2, as: UTF8.self)
    print("json1 is: \(jsonString1). Dump of input value follows.")
    dump(dict1)
    print("json2 is: \(jsonString2). Dump of input value follows.")
    dump(dict2)
    
    // Decode using JSONSerialization
    print("\nDecoding using JSONSerialization")
    let decodedDict1 = try! JSONSerialization.jsonObject(with: jsonData1) as! [String : Double]
    let decodedDict2 = try! JSONSerialization.jsonObject(with: jsonData2) as! [String : Double]
    print("decoded dictionary 1 is\(dict1 == decodedDict1 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedDict1)
    print("decoded dictionary 2 is\(dict2 == decodedDict2 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedDict2)
}

struct DoubleWrapper: Equatable, Codable {
    let value: Double
    
    static func ==(lhs: DoubleWrapper, rhs: DoubleWrapper) -> Bool {
        return lhs.value == rhs.value
    }
}

do {
    // Encode using JSONEncoder
    print("\n\n\nEncoding using JSONEncoder")
    let wrappedValue1 = DoubleWrapper(value: value1)
    let wrappedValue2 = DoubleWrapper(value: value2)
    let jsonEncoder = JSONEncoder()
    let jsonData1 = try! jsonEncoder.encode(wrappedValue1)
    let jsonData2 = try! jsonEncoder.encode(wrappedValue2)
    let jsonString1 = String(decoding: jsonData1, as: UTF8.self)
    let jsonString2 = String(decoding: jsonData2, as: UTF8.self)
    print("json1 is: \(jsonString1). Dump of input follows.")
    dump(wrappedValue1)
    print("json2 is: \(jsonString2). Dump of input follows.")
    dump(wrappedValue2)
    
    // Decode using JSONDecoder
    print("\nDecoding using JSONDecoder")
    let jsonDecoder = JSONDecoder()
    let decodedWrappedValue1 = try! jsonDecoder.decode(DoubleWrapper.self, from: jsonData1)
    let decodedWrappedValue2 = try! jsonDecoder.decode(DoubleWrapper.self, from: jsonData2)
    print("decoded value 1 is\(wrappedValue1 == decodedWrappedValue1 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedWrappedValue1)
    print("decoded value 1 is\(wrappedValue2 == decodedWrappedValue2 ? "" : " not") equal to the input. Dump of decoded value follows.")
    dump(decodedWrappedValue2)
}

Swift 4.0.3 on macOS 10.13.3 produces the following output and encodes and decodes all values as expected.

Encoding using JSONSerialization
json1 is: {"value":7.7087009966199993}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
json2 is: {"value":7.7087009966200002}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002

Decoding using JSONSerialization
decoded dictionary 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002



Encoding using JSONEncoder
json1 is: {"value":7.7087009966199993}. Dump of input follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966199993
json2 is: {"value":7.7087009966200002}. Dump of input follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966200002

Decoding using JSONDecoder
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966199993
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ DoubleTable.DoubleWrapper
- value: 7.7087009966200002

And Swift 4.0.3 on Ubuntu 16.04. Both input double values are encoded as 7.70870099662 in the resulting JSON.

Encoding using JSONSerialization
json1 is: {"value":7.70870099662}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966199993
json2 is: {"value":7.70870099662}. Dump of input value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002

Decoding using JSONSerialization
decoded dictionary 1 is not equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ 1 key/value pair
▿ (2 elements)
- key: "value"
- value: 7.7087009966200002



Encoding using JSONEncoder
json1 is: {"value":7.70870099662}. Dump of input follows.
▿ test.DoubleWrapper
- value: 7.7087009966199993
json2 is: {"value":7.70870099662}. Dump of input follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002

Decoding using JSONDecoder
decoded value 1 is not equal to the input. Dump of decoded value follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002
decoded value 1 is equal to the input. Dump of decoded value follows.
▿ test.DoubleWrapper
- value: 7.7087009966200002

[belkadan]

cc @itaiferber. There may have been a dup of this already.

[florianreinhart]

@itaiferber, I am still seeing this issue in Swift 4.2 on Linux. Could you take a look at this?

[spevans]

On Linux, JSONSerialization uses CFNumberFormatterCreateStringWithNumber to encode Float and Double and this ultimately uses ICU underneath. Interestingly in ICU-61 the Double -> String algorithm has changed (Ive been looking at upgrading the ICU on Linux) and was thinking that a better fix for scl-foundation would be to just use String to encode Double and Float. I have a patch that implements this if we want to make this change.

Note that Ubuntu18 currently uses ICU-60 so upgrading ICU to a later version would change the output format of JSONSerialization anyway but not in a way guaranteed to match Darwin.

[itaiferber]

Looks like floating-point numbers are printed out via

let string = CFNumberFormatterCreateStringWithNumber(nil, _numberformatter, num._cfObject)._swiftObject
writer(string) 

with _numberformatter defined as

private lazy var _numberformatter: CFNumberFormatter = {
    let formatter: CFNumberFormatter
    formatter = CFNumberFormatterCreate(nil, CFLocaleCopyCurrent(), kCFNumberFormatterNoStyle)
    CFNumberFormatterSetProperty(formatter, kCFNumberFormatterMaxFractionDigits, NSNumber(value: 15))
    CFNumberFormatterSetFormat(formatter, "0.###############"._cfObject)
    return formatter
}() 

This indeed doesn't match what we do on Darwin, which is snprintf(bufPtr, NUMBERBUF_SIZE, "%0.*g", DBL_DECIMAL_DIG, [obj doubleValue]) (i.e. on Darwin we print with more digits of precision). However, this has been true on swift-corelibs-foundation since before Swift 4, so I'm not sure what changed here in Swift 4.0.3 specifically on Linux.

cc @spevans — you've worked more on JSONSerialization on swift-corelibs-foundation than I have. Any recollection why this may have happened? I don't have a Linux machine to test on at the moment but I can try to get something running.

[florianreinhart]

@itaiferber, I just tested this with Swift 3.1.1 on Linux and issues exists there as well. So this not a regression in Swift 4.

@spevans, Using String to encode Double/Float should be a good solution for now. This ensures round-trip correctness, which is more important than having the exact same JSON on both platforms, I guess.

[spevans]

Fixed in #1722