Residencies are as varied as, well, doctors - the one thing in common, though, invariably, is the first, or intern, year is hard (and why people look for "cush" positions). In some fields, the years that follow become markedly easier to tolerate, whereas in others (like a lot of surgery), it's a lot of time, all the way through.
16 hours a day, 6 days a week - 96 hours regularly? Maybe in the old days. Now, in New York State, that is verboten, and, soon, it will be nationwide.
"Brutal" is subjective...but most would admit that q2 call (24 hours on, 24 hours off) is truly demonic (I mention that because a radiology resident I know said that he did preliminary surgery at the University of Virginia, they did q2 call - don't know for how long, though).