Well the #1 golden rule I was taught when scuba diving is "NEVER HOLD YOUR BREATH!"
Although I don't think you're alone. If I'm hovering very close to some coral or something and I want to hang exactly stationary. Sometimes I'll hold a half-lung of air to keep myself precisely neutrally buoyant. Having only half a lung of air means even if for some reason someone dragged me up, air expansion wouldn't instantly get the better of me.
I use my breathing to control my buoyancy when I need to go up or down a little - but I don't hold it - so for instance if I am hovering and want to go up a little, I'll inhale deeper than usual - but then I
won't be holding my breath - I'll just be breathing out slower to take advantage of the extra buoyancy. I do the oppsite to drop - breath out heavier than usual and breath in slowly as a drop - both cases though, as I say I'm not holding my breath.
I think generally, both of those things are ok and I'm not doing anything overly stupid.
On holding your breath just while diving, you probably are increasing your risk of something going wrong. There shouldn't be any reason to hold your breath while you're just swimming around normally. Perhaps your reg is not giving your air easily enough? If you can get air easier, it should be easily just to breath gently.
PADI says you're meant to breath "slowly and deeply" - which isn't particularly natural - if you are breathing normally and calmly, you tend not to breathe deeply. I've heard the reason for breathing deeply is so you clear out "stale" air at the bottom of your lungs - but I don't know how true that is.
I guess as long as you're not pushing yourself until you really have to breath you're probably going to be ok. Just be careful
I'd be interested to hear other peoples' views on this.