Yeah, for whatever reason that's where the engine struggles mightily. Makes no sense. I've seen some graphics out of this game I never thought I'd see and @50fps+ out in wide open environments on a single GTX when I've tested it, but group 10 online characters together in a s****y looking room and you have slow-mo all of a sudden.
Hopefully they'll continue to optimize. I do think it's getting better overall but not necessarily in those parts you describe, though I think it's gotten slightly better in those areas.
Oh, and one think I've found that's really helped me out in casting is this: nrdstrm suggested to go ahead and charge the bolts as the first cast, so that's what I've been doing. Instead of freezing first, I'll hit them w/ bolts first since it takes long (they don't bullrush while it charges.) Then after the bolts hit them I can immediately hit them w/ a freeze spell, then again with bolts/char, then again with freeze, and by that time their life is already halfway diminished or more by the time they've gotten to me.
Still helps me none when gang-rushed, but it's definitely an improvement for 1 on 1. I've been taking down my own level creatures pretty easily w/ this new combo.