Add preprocessing documentation for DeepSeek-r1 and Llama3.1-8b

[anivar]

Summary

Addresses issue #2245 by adding comprehensive preprocessing documentation for models that currently lack reproduction methodology.

Changes

• PREPROCESSING-TEMPLATE.md - Standardized template for future models
• language/llama3.1-8b/PREPROCESSING.md - Complete preprocessing documentation
• language/deepseek-r1/PREPROCESSING.md - Comprehensive preprocessing documentation

Problem Solved

Issue #2245 identified that while llama2-70b/processorca.py provides complete preprocessing transparency, newer models hide preprocessing behind preprocessed datasets, breaking reproducibility.

Solution Approach

• Maintenance-focused: Documents existing patterns rather than introducing new architecture
• Template-based: Establishes standard for consistent future documentation
• Gap analysis: Clearly identifies what preprocessing steps are missing
• Adaptation guidance: Helps users adapt preprocessing for different tokenizers/models

Documentation Added

Llama3.1-8b

• Documents current CNN/DailyMail preprocessing via download_cnndm.py and prepare-calibration.py
• Identifies missing filtering, sampling, and quality control steps
• Provides adaptation guide for different tokenizers
• References llama2-70b/processorca.py patterns for completeness

DeepSeek-r1

• Documents current cloud-based preprocessed dataset approach
• Identifies critical gaps in raw data access and preprocessing scripts
• Explains challenges for tokenizer/model adaptation
• Recommends improvements for full reproducibility

Benefits

1. Immediate transparency - Users understand current preprocessing limitations
2. Future standardization - Template guides consistent documentation
3. Adaptation support - Clear guidance for different use cases
4. Community contribution - Maintenance approach that builds on existing work

Next Steps

This documentation provides foundation for:

• Creating complete preprocessing scripts for affected models
• Establishing preprocessing standards across all models
• Improving benchmark reproducibility and transparency

Closes #2245

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[hanyunfan]

LGTM, more info added for readme files

[arjunsuresh]

@hanyunfan This is a template not actual information. We should pass this to the respective task forces and get the details.

[mrmhodak]

WG Meeting: Will look at this later.