Section 32. The collector shall be liable for the tax and the charges of imprisonment of a person discharged, unless he arrested and committed such person within two years after the tax was committed to him for collection, or unless he shall be exonerated therefrom by the town to which the tax is due.