I remember starting to feel the crunch of deciding which path to take for residency right about this time of the year too. I'm a 5th year applying this year to Ortho, but I didn't choose to go 100% Ortho until after two Sub-I's in EM when I was a 4th year.
I liked EM for the same reasons you listed--fast paced, lots of volume, short training, days off, etc. It took two sub-I's in EM for me to realize the only things I really enjoyed were procedures, which in reality are few and far between. I found out that EM, in southern Cali at least, is 99% primary care medicine for the millions of uninsured patients floating around out there. I couldn't see myself doing this for the rest of my life.
Ortho was the only thing I really got excited about as 3rd year, but I also knew it was one of the, if not the, most competitive residencies to obtain. I was in no way prepared to apply to Ortho as a 4th year, so I decided to take a year off and do some solid research, then do my Ortho sub-I's as a 5th year. So far my plan has worked out very well, though I can't say for sure until the end of the day 3/13.
I guess the questions you have to ask yourself are, are you or are you not a surgeon, and what is it that attracts you to certain specialties. The differences between a career in Ortho and a career in EM are staggering. The differences between Ortho and Urology are not as profound, but you will find that each has its own quirks and pros/cons.
In the end, the only way to know if a field is right for you before you apply is to gain as much exposure as you can to this field so you can base the decision of what you will do with the rest of your life on as much fact as possible. After all the worrying and careful deliberation though, I think most people will agree that in the end it is your gut feeling that matters the most. And the earlier you get this gut feeling, the better off you will be when it comes time to apply.