gh-38394: get rid of one sage_eval in modular
    
and also using the pari library directly

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->

- [x] The title is concise and informative.
- [x] The description explains in detail what this PR is about.
- [ ] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation and checked the documentation
preview.
    
URL: #38394
Reported by: Frédéric Chapoton
Reviewer(s): Matthias Köppe

Author: Release Manager