An old metaphor: conventional brain-to-brain communication is literally serialization. The thought is serialized as a sequence of letters or sounds, as writing or talking, and deserialized by the reader/listener. As with data structures and JSON/XML, the structures and serial representation might be similar or utterly different. But this mapping is taken care of at each end, so that people can communicate with a common tongue or lingua franca, even when their own internal representations differ greatly from it, and from each other.

My takehome is that without an intermediate serial representation (and mapping to/from), direct brain-to-brain communication will have to tackle mapping between these differences in internal representation in some other way (assuming such differences exist, which I believe they do).