I wouldn't bother packing the charge out with semolina. There is no harm using a wad over the powder if you want to but it is probably not necessary.
I use 60 grains of powder to propel a .575" Minnie bullet (560gr) 100 yards so a lower charge at the distance you are shooting would probably be adequate.
I know that in a flintlock pistol firing a .650"patched ball over 38gr of powder at 25 yards is more than enough powder.
Don't fill the flash pan with powder it has to burn through before its sets off the main charge and it causes a delay in ignition.
Use as little powder as possible in the flash pan so ignition of the main charge is quicker.
If you don't get good groups with the intended charges try going up 1/2grain at a time.
The patches and lube can also affect the accuracy. I know someone who took two years to get an antique rifle to shoot tight groups at 100 yards.
I had a Charles Moore percussion pistol by Pedersoli. They recommended .006" patches. The target looked like it had been hit with a shotgun.
I swapped to a thicker softer patch of .010" and it shot well. I tried a .015" patch thinking that the ball being tighter in the bore would make it more accurate, nope, it would not group with the thicker patch, I had to go back to the .010" patch.
Flintlocks are addictive and are good to shoot but they do try your patience sometimes.Some guns like to be swabbed out after each shot. Its all trial and error. All part of the interest I suppose.
Good luck with it.