The biggest challenge for MtF trans people isn't with men but with women. Passing comfortably in close quarters in their company (assuming that is a goal) is the ultimate test and I remember my friend Sherry telling me this. Of course she transitioned at age 25 and no one would know but still she concerned herself with it for a time.I used to worry about the same thing but as I started to successfully blend in, the worry disappeared and I could just relax and be myself. The passing became a fringe benefit that interestingly I was no longer aiming for.
It was simply a nice to have.