They do their best to use the number in ways no one but your contacts who use Signal can actually see what that number is, to be fair. And you’re still private either way. What a phone number breaks is anonymity, which is something they don’t explicitely claim to give you. (I think)