I have been ramming the balls into a pistol for a number of years and never had a problem.
.454" ball is generally used on .44" calibre, and .375" ball on .36".
Perhaps your loading lever on the model of pistol you have is a smaller diameter or thickness than a lot of guns?
It is not practical on some guns to remove the cylinder each time it is loaded. It is convenient if you can though.