Buildings tend not to be built exclusively in brick or stone and even when they are tend to be held together with mortar. Even brick and stone are susceptible to water intrusion and a bunch of other things. For example freeze-thaw is a very natural cause of erosion of stone. Vegetation is another common cause of undermining those sorts of structures.

Of course some materials last longer than others but that doesn’t make the resulting structures habitable.

Most of the houses where I live are two or three hundred years old, and not particularly leaky or draughty. The one I'm in is only about 90 years old.

What are they building houses out of where you live?

With maintenance sure. Which was the very next part of the original post you quoted so this reply to me makes no sense.

For example I’ve got a rubble wall that’s really showing the effects of freeze-thaw just now and if I don’t actively spend the money and time to fix that damage it will fall down eventually.