You get a -1 for downvotes. If it's spam, flag it, don't downvote it.

You can un-downvote it for your rep points back.