From what I know, the questline itself does not affect critical things such as endings or main quest progress, but it is closely connected to the main quest and not doing it will make you miss out some important context and story, not to mention some NPC will get mad at you for rejecting to do the quest because you canNOT do it. So yeah it can be really frustrating for some players but not so much to others. I unfortunately belong to the former.
