Duplicates will always happen, and as you have pointed out, sometimes there is positive value to that. The idea isn't to avoid duplicates, per se, but to try to ensure that the same discussion doesn't get hashed out again and again in different different, parallel and unconnected threads. There have even been cases[0] where the mods explicitly dipped in and moved comments from one submission to another to avoid the "split discussion" problem.
So the idea is to ensure that anyone interested in an item can, should they choose, easily find and read the comments made by other HN contributors. Feel free to disagree, although the 17 upvotes (at the time of writing - votes may go down as well as up) indicate that there are people who do find it valuable/useful.
Edit: Just checking your profile I see that you've been here less than a year, so you may not be aware that because it's me to whom you're replying, I can't down-vote you. In fact I've up-voted you, because while I disagree with you (obviously) I don't think your comment is especially negative.
Thank you for your reply. While you said my comment wasn't especially negative, I understand that the first sentence could have been taken that way, and that's probably the reason for the downvotes. It's just that I've seen people yelling 'DUPE' since the first year of Slashdot, and it seems like every site has some who make it their personal mission to rain on OP's parade.
I love the idea of stopping the 'split discussion' issue, but I feel like it's at odds with a site like HN where stories fall off of the front page so quickly. I'm not able to read it all day long, and I rarely look at it in the evenings. Especially when we have articles about potential gaming of the article positions, it seems unrealistic to expect that you would have a chance to actually get the real 'news' at any given point without duplicate posts.
What I mean is, there are certainly more than 30 in-progress topics appealing to HN readers. If I look at the front page in the morning and again in the afternoon, I'm only going to get a small subset of the topics that are out there. That's why I appreciate duplicates so much... it's really the only thing that makes a site like HN relevant to the occasional reader.
Anyway, thanks for supporting me even though I wasn't the most ambassadorial. :)
(Thank you for turning your comments in a more civil direction. That's rare enough that it cheers us up every time we see it.)
You make a fair point about duplicates. There used to be a second-hand clothes store in my home town called "New to You". Similarly, if you haven't seen a story, it isn't a dupe.
The problem is that the HN front page only has 30 slots. Those slots are the scarcest resource on the site. Since we have no way of knowing what you've seen, we have to target overall coverage.
We don't treat stories as dupes just because they've been posted already; they have to have had significant discussion. But I get that you're talking about precisely those stories. I do think we have more work to do here.
Just happened to catch your comment in passing, so I'll reply quickly.
It's true that things fall off the front page quickly, but that's why there's a "More" link. It doesn't take long to click through, scan the headlines, and check on the ones that look interesting.
It's also true that things wouldn't fall off the front page if there weren't so many duplications! It's less of a problem now, but I remember occasions when several of the front page submissions were all about the same thing, each with their own conversation, and preventing other topics from making it.
And as you say, at any given moment there are likely to be more than 30 "in progress" topics in the tech world, so the only solution is clicking through to see those items ranked 31 to 60, and 61 to 90. There is nothing else you can do. In part that's why I think it's important and relevant to link related conversations/submissions. Duplications will happen, items will fall off (or never make) the front page, so the best we can hope for is coherent discussions.
Hope that makes things clearer. Now I'm off to do some work. Cheers.
So the idea is to ensure that anyone interested in an item can, should they choose, easily find and read the comments made by other HN contributors. Feel free to disagree, although the 17 upvotes (at the time of writing - votes may go down as well as up) indicate that there are people who do find it valuable/useful.
[0] https://news.ycombinator.com/item?id=8499286
Edit: Just checking your profile I see that you've been here less than a year, so you may not be aware that because it's me to whom you're replying, I can't down-vote you. In fact I've up-voted you, because while I disagree with you (obviously) I don't think your comment is especially negative.