Given that the average player will likely experience blockages due to Ice Walls more than object walls, it may make sense to prioritize destroying the Ice Walls over moving objects.

Having enemies use appropriate spells to remove the walls would be cool, but I'm not too sure if can be done safely (fireball is probably the only spell that would sense to cast on them).