It's more about linguistic structure than origin of words.
Kannada, Telugu & Malayalam, even pre-dravidan-movement Tamil have lot of borrowed sanskrit vocabulary, but they retain structure of Dravidian languages.
Also most common words used are also Dravidan origin.
Another small thing unique about Dravidian languages is clusivity, there's exclusive 'we' and inclusive 'we'. While modern Kannada, unlike Tamil / Telugu, doesn't retain this feature, some dialects such as Havyaka retain this feature.
Sinhala does not have the structure of Dravidian languages. However, the alphabet used to write Sinhala has a shared origin with the alphabets used to write Dravidian languages. Also due to long coexistence with Tamil in the island, Sinhala burrows lot of loanwords from Tamil.
I am Sinhalese who grew up in Sri Lanka. As such I am exposed to Tamil, but when I watch Tamil TV, at best I can pick few words here and there. But when I watch Hindi show, I understand a lot more due to the structure is very similar to that of Sinhala.
I don't understand sinhala but if it is classified as Indo Aryan language and not Dravidian, then that is the reason.
There will be always be some influence and loanwords from neighbouring languages but comparative linguistics can tell origin / family of the language pretty accurately.
Kannada, Telugu & Malayalam, even pre-dravidan-movement Tamil have lot of borrowed sanskrit vocabulary, but they retain structure of Dravidian languages.
Also most common words used are also Dravidan origin.
For example:
Kannada - Naanu, Tamil - Naan, Telugu - Nenu, Malayalam - Njan
Etc..
Another small thing unique about Dravidian languages is clusivity, there's exclusive 'we' and inclusive 'we'. While modern Kannada, unlike Tamil / Telugu, doesn't retain this feature, some dialects such as Havyaka retain this feature.