The male dysphoric child is already aware of a gender incongruity before entering puberty (in my day you just shut your mouth) when suddenly there is a conflict present. Feelings towards females are now in direct competition with a desire to be one themselves which has an impact on the development of the burgeoning sexuality. However, this is not so much a misdirected sex drive (or target location error as it has also been called) as the overlapping of two distinct and opposing realities. We are perfectly aware of what is going on which is why many trans people choose to suppress their dysphoric feelings in favor of partnering with a woman.
Before puberty with no sexual drive present, there is no conflict just as with increasing age the arousal tends to go away with the depletion of sexual energy. Regardless of the phase of life, the transgender person's feelings are not diminished in the slightest and remain wholly intact.
Consider that if the entire process were driven exclusively by sexual motives, transition would not make much sense since the process lowers libido starting with the administering of cross sex hormones. You would be best off enjoying your fetish just as many non transsexuals do. However, the Blanchardians tie themselves up in knots using convoluted arguments to fit their hypothesis and end up looking comical in the process.
So, the arousal is a distraction in the sense that it is the result of an overlay of two distinct and opposing goals; in essence the conflict between gender identity and orientation.
Why don't androphilics experience the same thing? simple: their identity and orientation are in alignment instead of conflict. The same applies for the great majority of female to male transsexuals who tend to be attracted to their own birth sex.
I believe that all transsexuals regardless of pre-transition orientation or degree of asexuality experience a dysphoria which is genetic and pre-existing and, unless I see some compelling evidence to the contrary, my mind will not be changed. Harry Benjamin you were entirely correct back in 1966.
But here is the fun part in all this: none of it matters compared to the happiness of the individual and hence the mechanisms that caused their dysphoria become secondary. If the transition solves the problem then this is what matters most. Consider how a non transsexual would realize pretty quickly that they had made a mistake and ask for a refund (hello Walt Heyer). The science only matters in the arena of fighting back against the agents of prejudice who are hell bent on delegitimizing trans people and, in that case, information is power.
But I am an open minded person and even if I have spent years pondering and reading on this subject I might be wrong. So, if you have a compelling and, most importantly, conclusive argument to the contrary please send it along.