Except for the last 3 years, the money you erroneously paid into Social Security will remain to your credit in the Social Security fund. The Social Security Administration will include all but those last 3 years in determining your eligibility for, and the amount of, future benefits.
The amount you paid into Social Security for the last 3 years will be used to help pay your CSRS retirement deductions.