You only need one ore for Brandon IIRC. But you can use the rest to make Tenebrium bars to improve your weapons.
You need the password for the door, you can't blow it up. The freed sceleton miners (after you've destroyed the totem) should give you the password.
