fix: make invmod more robust for non-prime moduli (#1986)

fix: make invmod more robust for non-prime moduli (#1986) #2465