I honestly don't know how to do it. But I will say that on HN at least, there seems to be a moderation team interested in finding that balance.
The only comments I've seen that get killed are so far off the rails or so hateful as to be completely unproductive.
From what I've seen online, the malevolent ones just seem to want to resort to personal attacks, or make terrible arguments full of basic logic flaws.
I get what you're saying, but I would say that the onus is on the individual arguing to prove they are not malicious. If I type something that can be construed as 'off-topic' or malicious, I always either preface it with an explanation to the word choice (i.e. don't read tone in text, because this is said with a flat emote). Otherwise, I expect my comment to get flagged and killed.
Like this one. This has nothing to do with the original story, so it may get killed. And that's something that I understand and am okay with.
I don't know myself exactly all the circumstances that may lead to a dead post, but I don't really need to know - all I know is that the posts are rather off-the-rails and outliers, or just plain bizarre.
As for correcting the user's behavior - I think we make the calculation that interactive correction is impossible or infeasible at this time and in this forum. This is a self-interested calculation, but I think I must agree that it's a healthy self-interest.
Eventually, if they will be more careful not to alienate other users and make more useful and helpful comments, others who have showdead=Yes (like myself) could vouch for them.
I may have done so once or twice (I don't recall actually doing so), so that circumstance would seem to be rare...
My argument is that it is not clear in what way the dead posts became dead and neither self-correction nor peer-correction are possible any more.