Dolls are perceived as harmless, and they can be gorgeous and/or adorable, but there's still something scary about dolls. It's probably because many of them fit squarely in Uncanny Valley territory. The blank gaze and unmoving stare reminds us too viscerally of corpses, perhaps. This goes even more when the doll is damaged in some way, such as missing limbs or eyes, or having holes in its head.
Another way to do it is make it a clockwork toy (usually an organ-grinder's monkey with cymbals); something that moves on its own when someone winds the key, then not have it wound up for years, and have it click its cymbals in a haunted, mechanical rendition of Terrible Ticking.
In horror, dolls are often used as part of the scenery to help establish the mood, even providing a theme for an episode. Like clowns, what should be a silly, innocent bit of childhood fun can make a scene ironically unnerving, depending on where they're placed, what they're doing, or even simply how they're lit. They may even be the antagonist or be used by the antagonist. Despite how ridiculous a doll trying to kill people should be, it's still seen as quite frightening. A similar idea lies behind the Demonic Dummy and Scary Scarecrows.
The Vengeful Abandoned Toy is a specific type of this trope, who is angry at having been abandoned or forgotten by their owner.
