the referee can kick off as soon as its time to kick off..
one player has to be designated as a goalkeeper and this is identified by a different coloured top to the other players and officials,,( yes in law he shouldnt wear black).
this sounds like an awareness problem before kick off..i always,,count players, check assistants(although in six a side maybe wont have them),quick look to make sure both goalkeepers are ready, if one is still putting gloves on then wait till he has them on..if you think he is messing about maybe a quick shout "all ok " but the amount of time he takes to put glove on would be seconds and not have a great effect on the kick off of the game