Add majority voting plugin for candidate selection #208
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces a plugin that generates multiple candidate solutions using the OpenAI API and selects the most frequent answer via majority voting. Includes answer extraction, normalization, and a summary of the voting process. Useful for tasks with discrete answers such as math, coding, and multiple choice problems.
Enhanced the extract_answer function to better handle LaTeX boxed answers and multiple choice patterns. Moved majority voting summary from being appended to the response to logging via the logger, ensuring cleaner output.
Introduces a new MLX-compatible ThinkDeeper implementation (`thinkdeeper_mlx.py`) and integrates it into the inference pipeline for MLX models. Updates the inference logic to select the appropriate ThinkDeeper version based on the backend, and refines fallback and error handling for MLX generation. The evaluation script is updated with new test-time compute scaling approaches, including revised ThinkDeeper and majority voting configurations, and improved reporting for test-time compute experiments.
Updated best_of_n_sampling, mixture_of_agents, and majority_voting_plugin to handle providers that do not support the 'n' parameter by generating completions/candidates one by one in a loop. This improves compatibility with a wider range of API providers and ensures robust completion generation even when batch generation is not available.
Update version number in __init__.py and setup.py to 0.1.20 for new release.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces a plugin that generates multiple candidate solutions using the OpenAI API and selects the most frequent answer via majority voting. Includes answer extraction, normalization, and a summary of the voting process. Useful for tasks with discrete answers such as math, coding, and multiple choice problems.
fixes #201
superseeds #83