Well, I think he is always "mystic", as in he has his power unlocked and he doesn't need to turn SSJ nor can he turn SSJ.
All he does is increase and decrease his power level normally to his maximum and minimum (and all in between).
Since there would be no reason for him to hold back fighting Buu in space, I believe he was going all out.
Well, the first time he used the "mystic" power-up, the old Kaioshin told him to try and go Super Saiyan. Gohan did the usual "getting ready for Super Saiyan" powering-up, but instead of going all glowy-golden-hair like usual, he just powered up to his maximum without much of a physical change (besides the "eyes drawn differently" thing.)
So I think the power-up still needs to be activated like a Super Saiyan transformation even though he doesn't use actual Super Saiyan anymore; it's something different than just suppressing and un-suppressing your power depending on whether you're fighting at the moment or not.
But yeah, Gohan would have no reason to hold back super-transformations against somebody like Buu. Maybe if this was happening in the arena and he was afraid of getting disqualified or something, but not way out in the middle of nowhere in space.