In any bet you need a judge who decides which way the conditions of the bet resolved. The judge is someone trusted by both parties to be impartial and fair. If a lot of people stop trusting polymarket to act fairly and impartially, that will simply mean fewer people participating in the bets.