-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
System information
- OS version/distro: Windows 10
- .NET Version (eg., dotnet --info): netstandard2.0
Issue
Microsoft.Data.Analysis\PrimitiveDataFrameColumnComputations.cs provide wrong result for Min/Max functions
For these functions (and probably some other) we have wrong computation results:
We start computing from default value (code produced from tt templates):
Default value for integer is 0 and on each iteration we set 0 to ret not regarding that I have (for example) only positive values in column container:
public void Min(PrimitiveColumnContainer column, IEnumerable rows, out int ret)
{
ret = default;
var readOnlySpan = column.Buffers[0].ReadOnlySpan;
long minRange = 0;
long maxRange = ReadOnlyDataFrameBuffer.MaxCapacity;
long maxCapacity = maxRange;
IEnumerator enumerator = rows.GetEnumerator();
while (enumerator.MoveNext())
{
long row = enumerator.Current;
if (row < minRange || row >= maxRange)
{
int bufferIndex = (int)(row / maxCapacity);
readOnlySpan = column.Buffers[bufferIndex].ReadOnlySpan;
minRange = checked(bufferIndex * maxCapacity);
maxRange = checked((bufferIndex + 1) * maxCapacity);
}
row -= minRange;
ret = (int)(Math.Min(readOnlySpan[(int)row], ret));
}
}
As a result I have ret = 0. The same case is in Max function if I have only negative values in readOnlySpan collection.