add script to generate a example folder for collecting many files

[RonnyPfannschmidt]

No description provided.

[nicoddemus]

Not sure if this is a good idea, that many files does incur an overhead for checkout doesn't it?

[blueyed]

I also do not like it. While it might be small in Git it still uses inodes etc, and adds unnecessary noise.

Instead, there should be a script to generate this, and maybe a Makefile around it, which then could grow into a performance/benchmark test suite.

[RonnyPfannschmidt]

we totally need some tooling around manual testing support (this would be a manual test where we want to drop into a optional manual interaction point with a ipython shell or a system shell in the testdir

then we could jsut start some tests in "interactive mode, and mess around

[blueyed]

@RonnyPfannschmidt
Sure. But it is fast to generate those files locally, and would be trivial through a Makefile (using the script that was provided already).

[RonnyPfannschmidt]

@blueyed i dislike incomplete error prone tooling because ^^ - and i absolutely refuse makefiles - those things should die already

[nicoddemus]

I concur with @RonnyPfannschmidt that Makefiles are not the way to go. 😁

I propose we commit a Python script which can generate the files; this way we don't incur the overhead of 1000s files on each checkout, plus the script can be configurable (how many files to generate, and depth of folders, etc) which will make it even more useful in the future (say we want to test with 10k or 100k files, 6 directories deep).

I think the script can reside inside the examples folder, say in testing/example_scripts/perf_examples/collect_stats/generate_files.py, and contain a .gitignore file which ignores all the files generated by it to avoid accidental commits.

[RonnyPfannschmidt]

@nicoddemus good thinking

[codecov]

Codecov Report

Merging #4240 into master will increase coverage by 2.48%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4240      +/-   ##
==========================================
+ Coverage   93.38%   95.87%   +2.48%     
==========================================
  Files         109      109              
  Lines       24585    24590       +5     
  Branches     2389     2392       +3     
==========================================
+ Hits        22959    23575     +616     
+ Misses       1308      720     -588     
+ Partials      318      295      -23

Flag	Coverage Δ
#docs	29.06% <ø> (ø)	⬆️
#doctesting	29.06% <ø> (ø)	⬆️
#linting	29.06% <ø> (ø)	⬆️
#linux	95.65% <ø> (?)
#nobyte	92.01% <ø> (?)
#numpy	93.01% <ø> (?)
#pexpect	41.68% <ø> (?)
#py27	94.05% <ø> (+2.89%)	⬆️
#py34	92.35% <ø> (?)
#py35	92.36% <ø> (+0.65%)	⬆️
#py36	94% <ø> (+2.29%)	⬆️
#py37	92.37% <ø> (?)
#trial	93.01% <ø> (?)
#windows	94.13% <ø> (+0.74%)	⬆️
#xdist	93.89% <ø> (?)

Impacted Files	Coverage Δ
testing/code/test_code.py	94.2% <0%> (-1.32%)	⬇️
testing/code/test_source.py	97.04% <0%> (ø)	⬆️
testing/python/fixture.py	99.24% <0%> (+0.1%)	⬆️
src/_pytest/fixtures.py	97.37% <0%> (+0.13%)	⬆️
src/_pytest/config/__init__.py	94.98% <0%> (+0.33%)	⬆️
testing/test_config.py	99.42% <0%> (+0.38%)	⬆️
testing/test_assertrewrite.py	83.38% <0%> (+0.46%)	⬆️
src/_pytest/assertion/rewrite.py	95.7% <0%> (+0.66%)	⬆️
testing/test_runner.py	96.88% <0%> (+0.66%)	⬆️
testing/deprecated_test.py	100% <0%> (+0.76%)	⬆️
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 041044e...bfa1993. Read the comment docs.

[blueyed]

@nicoddemus
That's what I've meant basically.

And a Makefile would just call the script then, preparing the files needed for the perf tests, but it would not be required to be used of course. I am not talking about autoconf or something similar, but just something basic as (in collect_stats):

100x10:
  python generate_files.py 100 10 $@

test: 100x10
  cd 100x10 && pytest -k doesnotmatch

Then CI / perf generation could use make -C …/collect_stats test.

Anyway, this was just an idea for the future, but the main point is that files should be generated dynamically, not being committed, and I am glad you agree.

Closing this PR then.

i just made the script

[nicoddemus]

Nice work! 👍