I agree - everyone doing their character's dialogue in everyone else's parts would make for some difficulty due to different recording equipment and such. The abrupt switch can be pretty jarring for some people, and it's thrown me right out of podfics in the past.

(Edited because my cat stepped on my keyboard before I could finish typing.)