Section 96. In any town in which such appointments are not subject to chapter thirty-one, they shall be made annually or for a term of years not exceeding three years, as the selectmen shall determine, and the selectmen may remove such officers for cause at any time during such appointment after a hearing.