Skip to content

PrimitiveDataFrameColumnComputations produce wrong result for Min/Max functions #5759

@0xcafebabee

Description

@0xcafebabee

System information

  • OS version/distro: Windows 10
  • .NET Version (eg., dotnet --info): netstandard2.0

Issue

Microsoft.Data.Analysis\PrimitiveDataFrameColumnComputations.cs provide wrong result for Min/Max functions

For these functions (and probably some other) we have wrong computation results:

We start computing from default value (code produced from tt templates):

Default value for integer is 0 and on each iteration we set 0 to ret not regarding that I have (for example) only positive values in column container:

public void Min(PrimitiveColumnContainer column, IEnumerable rows, out int ret)
{
ret = default;
var readOnlySpan = column.Buffers[0].ReadOnlySpan;
long minRange = 0;
long maxRange = ReadOnlyDataFrameBuffer.MaxCapacity;
long maxCapacity = maxRange;
IEnumerator enumerator = rows.GetEnumerator();
while (enumerator.MoveNext())
{
long row = enumerator.Current;
if (row < minRange || row >= maxRange)
{
int bufferIndex = (int)(row / maxCapacity);
readOnlySpan = column.Buffers[bufferIndex].ReadOnlySpan;
minRange = checked(bufferIndex * maxCapacity);
maxRange = checked((bufferIndex + 1) * maxCapacity);
}
row -= minRange;
ret = (int)(Math.Min(readOnlySpan[(int)row], ret));
}
}

As a result I have ret = 0. The same case is in Max function if I have only negative values in readOnlySpan collection.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions