I've always found the concept of qualia rather far fetched. It seems like it means people are born with knowledge instead of acquiring it. I think it's unlikely the brain works that way. The development of vision in children suggests to me there is no such things as qualia, instead, the labels for colour are learned and therefore not exactly equal for everybody. But, since it is constrained in both the input side, the architecture in which it has to be implemented and the form of the output that can be used by other brain areas it will still be highly similar. (I'de even say that the meaning we assign to colours will also be highly similar since we live in the same universe, on the same planet in human cultures.) A lot of recent colour experience research actually points in that direction.

So my guess is that colourblind people do not have a label for something they've never encountered. Just like all of us aren't born with platonic ideas for objects that haven't been invented yet.