Your big definitely did you a favor. This way you can layer under the jacket and wear it year round. Not everything is supposed to fit like a Tapout shirt. (Some people would argue nothing is supposed to fit like that, but I digress.)
As far as the jersey, well, just don't gain any weight. Get your fat ass to the gym and walk instead of driving.