Most of the frustration I remember had to do with the lack of depth perception, especially in Warped (which I remember best). It was certainly a fun game - I'd spend hours on end playing it - but it was also frustrating to be unable to properly judge exactly how far Crash needs to jump lest he plummet into a bottomless pit (or a stack of nitro boxes).
Looks like my memory is halfway correct, since I forgot about the side-scrolling levels, which I thought were from one of the follow-up sequels, and not the original game.