The Oxford English Dictionary defines this hyphenated word as "a childish or doll-like person decorated with aglets," but the definition appears to be pure speculation, as Shakespeare's use of the word in this play is the first of only four uses of the word cited by the OED; the other three cited uses are all from the 19th century, more than 200 years after Shakespeare's death.
However, the word "aglet" is still in use; it's the name of the plastic thingey on the end of a shoe-lace. In Shakespeare's time a lot of articles of clothing, such as men's shirts and lady's gowns, were held together with lacings, so there were a lot more aglets, none of them made of plastic, and some of them quite fancy, so perhaps a fancy one in the shape of a human could be called an "aglet-baby."
In the painting below, done about the time Shakespeare was born, there are fancy aglets at the ends of the red ribbons on the sleeve and skirt.
Elizabeth of Valois, Queen of Spain,
wears rose ribbons tipped with large aiguillettes set with pearls,
By Juan Pantoja de la Cruz
MUSEO DEL PRADO