The game was virtually over as a contest by half-time after goals from ex-Hearts man Joe Hamill and teenager Craig Stearman.
But it was curtains for Bury after 51 minutes when experienced marksman Allan Smart foolishly elbowed Nils Eric Johansson.
Referee Mike Pike had no hesitation showing Smart the red card and nine minutes later awarded a Leicester a penalty for John Hardiker's trip on Stearman.
However, until Hamill struck with an 18-yard drive that went in off an upright Bury were very much in the game.
But they paid the price for poor defence from a corner and made the same mistake moments later as Stearman scrambled in a loose ball.
Joey Gudjonsson wrapped up victory on the hour.
"This was a potential banana skin and a no-win game," said manager Craig Levein.
"But it was a professional performance with a number of pleasing aspects. We kept a clean sheet and we were able to give some new players a run out.
"The only disappointment was Patrick Gerrbrand's injury on his ankle again."
Bury have made a dreadful start to the campaign and chants of Barrow Out could he heard before the final whistle.
The Shakers chief was understandably downbeat but was furious with Smart.
"He's supposed to have apologised to his team-mates," he blasted.
"But we've had words and we'll be having more later this week. I regard myself as a players' manager, but I can't defend what happened."