It depends on what I am watching and what I am used to hearing from that thing. For example, I have currently only watched GT in English and am very familiar with it, and I mostly watch Super in Japanese. I plan on watching GT in Japanese soon, and I know that it will be....an adjustment, at first. I can get used to it quickly. Similarly, I was thrown off watching Goku's SSBKK in English on YouTube because I'm not as used to associating Super's imagery and Goku's screaming with the English actor as of recently.
So, its an adjustment, but I wouldn't call it a difficult one. Its just natural to "get used" to one thing or the other.
I can switch pretty easily with the Z movies because I have always found enjoyment in watching both for the differences between the versions of each film.

