Yes, I've also noticed, you can do elemental damage without physical damage, but I think it's fine and works as it should. Because it gives you some extra damage anyway (like if you were unable to do physical damage), then it's like a bonus type of thing