Physicists reject that thought experiments have value? I'd say the context in which I most frequently hear the phrase "thought experiment" (or "Gedankenexperiment") is physics! Indeed, that is historically where the term originated. (And we must also consider examples like the very one of Einstein's you pointed out).
I also don't see any reason why thought experiments and mathematical arguments should be thought of as exclusive... I would think an important part of a thought experiment is often its mathematical analysis.
The way I see it, you hear about thought experiments in physics because of Einstein. Because of him they can't avoid giving them some credence. But the odds of a new thought experiment purporting to show a bug in general consensus physics being given consideration today is nil, and I'm talking about before the text is read. Using thought experiments to discuss current physics is fine as a starting point only. For new physics they are junk that immediately labels one as a crackpot.
Until the time of ~Einstein thought experiments were feasible enough because only the intuitive-upon-closer-inspection parts of physics were discovered.
The big problem is that the finer points of the new physics are just straight-forward unintuitive. To understand even quantum physics (which is very basic) on an intuitive level is sufficiently impossible and has baffled the community for some time (see also the resolution[1]). It is however quite possible to develop a good intuition for the maths describing it.
Sadly, the history of thought experiments and these discussions still linger in the vocabulary used to describe the phenomena. Many words are common words that inspire some people to invent their 'intuitive' own version of the theory, based on what the words mean in every-day discourse[2].
I also don't see any reason why thought experiments and mathematical arguments should be thought of as exclusive... I would think an important part of a thought experiment is often its mathematical analysis.