Felon Suffrage

Every once in a while you hear about reconsideration of restoring felons' right to vote after they've been released from prison. But I want to take the question back a step. Should felons lose their voting rights at all? Incarceration can be justified in terms of punishment, but more pressingly in terms of the need to separate certain people from the society for safety reasons. But do I really need to be protected from their votes? Have they shown themselves so irresponsible in the use of their freedoms in trampling the rights of others that they should be stripped of one of the most important elements of citizenship?