Skip to content

Conversation

jvpoulos
Copy link

@jvpoulos jvpoulos commented Sep 4, 2025

The prepare script explicitly filters structures to only include train molecules. This is incorrect for this competition --- in the Kaggle competition, structures.csv contains all molecules (both train and test). The test molecules need their structures to make predictions, but they're being filtered out.

Changes:

  • Modified prepare.py to include both train and test molecules in structures.csv
  • Updated checksums.yaml to reflect the new structures.csv checksum
  • Updated assertions to validate both train and test molecules

After fixing the data preparation issue, I used a LightGBM model trained on 25% of the data to make predictions:

  {
    "competition_id": "champs-scalar-coupling",
    "score": 0.5823,
    "gold_threshold": -2.87509,
    "silver_threshold": -2.03119,
    "bronze_threshold": -1.90122,
    "median_threshold": -0.9529,
    "any_medal": false,
    "gold_medal": false,
    "silver_medal": false,
    "bronze_medal": false,
    "above_median": false,
    "submission_exists": true,
    "valid_submission": true,
    "is_lower_better": true,
    "created_at": "2025-09-03T22:56:19.592098",
    "submission_path": "mlebench/competitions/champs-scalar-coupling/submission.csv"
  }

@thesofakillers
Copy link
Collaborator

Thank you for catching this. You are correct and there is a mistake in the prepare.py, and your fix seems right.

As explained in the readme in #66 we won't be merging this fix in yet, and will release it as a batch of fixes in a upcoming v2 to be released on openai/preparedness. I've added as tracked in #71. I will try to put you as co-author for when we release the fix.

For submissions to the v1 leaderboard, please proceed as if this issue was not present.

thesofakillers added a commit that referenced this pull request Sep 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants