There was an article recently about the most difficult things (single actions) to do in the world of sports. #1 was hit a baseball. It might not be the most physically challenging activites, but I can't image swing a thin piece of wood at a ball that is magically moving 95 MPH - often in a od motion (think curve-ball, slider, etc.) and connect with this moving object to put it in a narrow field of play.
At the same time, and thank God for this - you are not on a sheet of ice (although, the Saint Paul Winter Carnival here in Minnesota has an annual Jan. Snow softball tournament - it's not easy to find the baselines).
Physically challenging:basketball and hockey are probably close. With hoxkey - you have more players changing, and on the "fly" - allowing for a little more rest time. Nevermind the roster size comparisons.
Soccer/Footbal(world, not US) is even more physically challenging; very few substitutions; constant running; and ball movement and very few stoppages.
In American football, you have to be ready for quick bursts. This takes a different kind of training.
I am sure there are other TEAM sports that are equally as challenging (I play lacrosse, and it rivals soccer, hockey and basketball) - and maybe more challenging, but I just wanted to throw out the more popular activities.