If the only downside you're seeing is "it pushes your group to choose one damage class", then that's not true. You can do that and it works fine but the system really doesn't push you to do that. Some have reported that they had more difficulty when playing a party that focused on one single damage type.
I played a 3/1 party once and it was fine. But yes, you do need to know what you're doing. You can take advantage of summoned minions to compensate for the damage type your party lacks. Basically you can turn your party in to a 3/2, rather than 3/1, with summons.
