Same reason people build those prepper bunkers. The towers were presumably not there for the general public. Consider as well other castles and sieges. There is a limit of how many people would be allowed in, and everyone in a siege needed to pull their weight. Really, once a town is under siege, the castle is closed, when the town walls are breached, the pillaging happens and the castle does not open its gates. My point is that medieval town defense was not really a communal endeavor per se.