I dropped out of Physics back in the day, because I loved computers a bit more, and now that I'm kind of fed up with computers I'd like to remove the thorn and do something like this.
But I find that so much time has passed that I would need to brush up parts of my high school maths first, and this kind of discourages me before even starting.
I decided to start self learning theoretical physics late last year. I have been now studying physics every day for almost a year (before and after work). I did have to brush up on calculus and matrices but it came back very quickly (within a few days) after a 25 year gap so I'd say don't let that discourage you.
I have been working on quantum physics since March of this year and am hoping to complete the whole text book (Townsend) by end of the year. Then on to special relativity -> classical field theory in 2024, general relativity and QFT in 2025 and 2026 - at least that's the plan.
I'm preparing to do something similar but attack a lesser beast that is the general relativity. I had a Master's degree in Statistics but unfortunately 1) Statistics does not really match Pure Mathematics and 2) I forgot most of it.
A beast it still is, I think it is contained in its own walls. I can skip any topic in Quantum Physics and others that is irrelevant.
I'm wondering if it's helpful to you too to focus on something smaller.
Using math to model a system instead of learning math qua math does wonders for ease of understanding. Derivates and integrals become easy if you're using them to model the relationship between position/velocity/acceleration. I don't think I really got linear algebra until using it to learn quantum computing.
But I find that so much time has passed that I would need to brush up parts of my high school maths first, and this kind of discourages me before even starting.