Do you think completely immune enemies are unnecessarily restrictive?

I just started playing during Phoebe's banner, loving her and the game. I wanted to pull only for the characters I like and not worry too much about the end game, at least not yet.

However, I wanted to pull for Zanni and Shorekeeper because I love them both and realized they're (likely) all spectro. As far as I know, this is a huge inconvenience since anytime I run into a spectro immune prism, well, that's it. Got no other choice other than back away from the challenge and restart with a different team.

This really bums me out, it feels like mono teams are hard-gated behind inconvenience just because of the existence of this single, minor enemy.

People usually dismiss this by saying that overworld is easy anyway, but do you think elemental immunity prisms are too restrictive?