"eapecially with end-to-end encryption on the horizon"
https://matrix.org/docs/spec/client_server/r0.1.0.html#end-t... ... unfortunately there is nothing there yet ;-/ ... this is a sign, encryption will be placed on top, which will weaken the idea, as it is an optional addon not a first class citizen.
The Matrix spec is still in development and E2E has been in progress since the beginning. It is not an "optional addon", it will be part of the core when merged. You're looking at the wrong branch: https://matrix.org/speculator/spec/drafts%2Fe2e/client_serve... is the in-progress spec, and https://matrix.org/jira/browse/SPEC-162 tracks all the progress. Meanwhile, it's even landed on the develop branch of Vector already (at least for the 1:1 ratchet).
Personally, I enjoy having a repository of my location history that I can browse easily. Where was I on such and such a date? Easily looked up.
As for the privacy issues, they are real, but my carrier could easily replicate this data if required to anyway. So if it's going to be generated with or without my consent, I may as well have access to my own copy.