For your costume contest make sure to state on your invitation what the criteria is (most original, best interpretation of the theme, etc.) and what the prizes are (make them good to increase participation).
Set up an area for voting including ballots, pencils, and a container for placing the votes. Threaten no more booze for cheaters. Before counting the ballots, circulate and allow the guests one final chance to vote.
It's easiest to have one category and three prizes: first place, runner-up, and judge's favorite (which gives you a chance to recognize a creative but perhaps less popular costume).