Great post. I also consider myself very lucky, much the same way as you. I've had some setbacks, but nothing that couldn't be handled. I also feel this pull from these two lifestyles, the "boring" stable desk job, or traveling the planet, code whatever I want and live life like there's no tomorrow. My fiancée is two years older than me (27), so that puts some pressure on getting kids and settling down fast. I've managed to save up what I consider a lot of money, but I feel the need to use that towards buying an apartment or house.
Other than being single, I'm in the exact same position as you. I'm pulled between building a career and travelling the world.
I graduated last spring and spent the summer backpacking in Asia before starting a boring desk job. I live with my parents still so I'm saving almost all of my income.
Now I'm seriously considering quitting in six months and heading back to Asia until I figure out what I want to do. My job is terrible, especially since I was effectively lied to about what I'd be doing, but I'm more or less getting paid to do nothing. In six months I'll have enough money to spend a year backpacking across SE Asia, Eastern Europe, South America, wherever.
Or I could just quit now and go find a job I like. I'm honestly not sure what to do.
Life is too short to do something you hate. Do whatever you need to to find out what you're passionate about. Building a career around that will be 100X more fulfilling.
As Steve Jobs said, "If today were the last day of my life, would I want to do what I am about to do today?" And whenever the answer has been "No" for too many days in a row, I know I need to change something."