PERF: Using Numpy C-API arange

[ShaharNaveh]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

---

This PR was opened as @jbrockmendel suggested (ref #32177 (comment))

---

Benchmarks:

Master:

In [1]: import pandas._libs.internals as internals

In [2]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array
1.55 ms ± 143 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

---

PR:

In [1]: import pandas._libs.internals as internals

In [2]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array
1.46 ms ± 3.55 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

[jbrockmendel]

Since the perf numbers are pretty similar, can you report a few rounds of each

[ShaharNaveh]

Since the perf numbers are pretty similar, can you report a few rounds of each

Sure!

---

In [1]: import pandas._libs.internals as internals                                                            

In [2]: %timeit internals.BlockPlacement(slice(1)).as_array                                                   
1.46 µs ± 45.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
586 ns ± 35.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [3]: %timeit internals.BlockPlacement(slice(10)).as_array                                                  
1.65 µs ± 21.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
832 ns ± 19.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [4]: %timeit internals.BlockPlacement(slice(100)).as_array                                                 
1.74 µs ± 14.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # master
879 ns ± 12 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [5]: %timeit internals.BlockPlacement(slice(1_000)).as_array                                               
2.94 µs ± 105 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # master
2.1 µs ± 77.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) # PR

In [6]: %timeit internals.BlockPlacement(slice(10_000)).as_array                                              
10.1 µs ± 198 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # master
9.39 µs ± 170 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each) # PR

In [7]: %timeit internals.BlockPlacement(slice(100_000)).as_array                                             
81 µs ± 1.75 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) # master
83 µs ± 1.36 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) # PR

In [8]: %timeit internals.BlockPlacement(slice(1_000_000)).as_array                                           
1.72 ms ± 165 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # master
1.56 ms ± 13.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # PR

[jbrockmendel]

thats a nice improvement for the small cases, thanks

[jbrockmendel]

nitpick: i like to add an extra space at the beginning of multi-line comments specifically to avoid confusing this for commented-out code. quotation marks or backticks or ... are also good

[jbrockmendel]

lgtm, thanks for accomodating my nitpicking

[ShaharNaveh]

@jbrockmendel I have a list of your nitpicks and requests, I try to address them when I have the time :)

[WillAyd]

Why does this generate anything different from Cython? Shouldn't it generate the same result?

[jbrockmendel]

Why does this generate anything different from Cython? Shouldn't it generate the same result?

The generated C code has to call np.arange in python-space (and lookup np.int64, and ...)

[WillAyd]

So doesn’t work like the built in range call then huh? Was expecting it would automatically convert to C without the Python lookup

…

Sent from my iPhone

On Mar 13, 2020, at 3:34 PM, jbrockmendel ***@***.***> wrote:  Why does this generate anything different from Cython? Shouldn't it generate the same result? The generated C code has to call np.arange in python-space (and lookup np.int64, and ...) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jreback]

thanks I agree we should not just wholesale convert calls to use the c-api, but in specific cases where we are actually returning a numpy array (as opposed to iteration), ok