Skip to content

Avoidable overflow errors in division #154

@kimikage

Description

@kimikage

If the output uses FixedPoint, as shown below, there is some rationale for the overflow error.

julia> Gray{N0f8}(1) / Gray{N0f8}(1)
Gray{N0f8}(1.0)

julia> Gray{N0f8}(1) / Gray{N0f8}(0.6)
ERROR: ArgumentError: N0f8 is an 8-bit type representing 256 values from 0.0 to 1.0; cannot represent 1.6666666

On the other hand, if the output uses AbstractFloat, the error should be avoided.

julia> Gray(1.0) / Gray{N0f8}(0.6)
Gray{Float64}(1.6666666666666667)

julia> Gray(1.0) / 0.6N0f8
ERROR: ArgumentError: N0f8 is an 8-bit type representing 256 values from 0.0 to 1.0; cannot represent 1.6666666

Originally posted by @kimikage in #148 (comment)

This is a bug in the optimization technique using reciprocals. E.g.:

(/)(c::AbstractGray, f::Real) = (one(f)/f)*c

By properly promoting the type before calculating the reciprocal, this problem should be avoided. E.g.:

(/)(c::AbstractGray, f::Real) = (oneunit(divtype(eltype(c), typeof(f)))/f)*c

That said, division is fast enough on modern CPUs, so we might verify how effective the technique of using reciprocals is in the first place.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions