I, too, find this a bit confusing.
When I cycle to work I usually class it as hard or extremely hard because I tend to go hell for leather, even though it is only 11 miles.
When I do a long ride I grade it by the amount of effort involved (hills, wind, types of road, stops/starts, etc), trying to ignore the time factor. Ironically, this could mean a 10-hour, 100-mile ride or longer being classed as medium (for you swimmers that's a bit like classing a Cross-Channel swim as medium!), so I add in a pain/exhaustion effect, though this may have nothing to do with calories at all.
I'm not too worried at the moment as I am still trying to lose weight, but it will matter more as I get down to my target weight and want to maintain it. To do that I'll have to balance my calorie intake with my expenditure, so I'll need a more accurate measure of the latter. I may get one of those whizzy computers that can measure such things.