I'm dealing with this issue now.
I don't pass, and I can't pass. And I know that virtually no cis person sees someone like me as a woman. That's reality, and I accept it.
I was recently invited to participate in a women's group, by one of the leaders. I was very flattered, and I would certainly love to do it. But I won't, and this is why.
Because I know that none of the women there will see me as a woman, they will necessarily see me as an interloper. My presence will change the energy of the space. They will not feel as uninhibited to be themselves. They will not feel "safe."
If I didn't know this, things would be different; but I do. I don't see this as a matter of rights; I see it as a matter of courtesy and kindness. The last thing I want to do is turn a place where women feel safe to be women into a place where they have to put their guard up. That's thoughtless, selfish and unkind -- at least, I would find myself to be so, if I ignored their feelings by invading their space.
The fact that I believe I am a woman doesn't matter. The problem is, they don't believe it. And I cannot disregard the effect my presence will have on their experience.