Potential performance issue: .fillna memory issue in pandas below 1.4.2 version

[TendouArisu]

Issue Description:

Hello.
I have discovered a performance degradation in the .fillna function of pandas version 1.4.1 and below 1.4.2. And I notice the repository depends on pandas 1.4.1 in scripts/eval/science-world/requirements.txt. I am not sure whether this performance problem in pandas will affect this repository. I found some discussions on pandas GitHub related to this issue, including #46149 and #46204.
I also found that scripts/data/interaction/collection/convert_outputs.ipynb and scripts/eval/mint-bench/convert_outputs.py used the influenced api. There may be more files using the influenced api and pandas version below 1.4.2.

Suggestion

I would recommend considering an upgrade to a different version of pandas >= 1.4.2 or exploring other solutions to optimize the performance.
Any other workarounds or solutions would be greatly appreciated.
Thank you!

[xingyaoww]

Hi @TendouArisu, thanks for the issue! I just removed the file scripts/eval/science-world/requirements.txt since we did not use it to setup environment (see here). The code should work fine if you use a newer version of pandas. Please let me know if you encountered more issues!