This idea has been riffed upon in a number of works of fiction, most famously The Body Snatchers, the Jack Finney novel which was the basis of the movie(s) The Invasion of the Body Snatchers. I still remember what for me was the most shudder-inducing moment in the book. The town doctor is seeing a patient for something minor, and she (the patient) remarks that her cousin Wilma is acting oddly, that Wilma thinks that "Uncle Ira isn't Uncle Ira any more." The doctor decides to talk to Wilma about it:
Wilma sat staring at me, eyes intense. "I've been waiting for today," she whispered. "Waiting till he'd get a haircut, and he finally did." Again she leaned toward me, eyes big, her voice a hissing whisper. "There's a little scar on the back of Ira's neck, he had a boil there once, and your father lanced it. You can't see the scar," she whispered, "when he needs a haircut. But when his neck is shaved, you can. Well, today -- I've been waiting for this! -- today, he got a haircut..."
I sat forward, suddenly excited. "And the scar's gone? You mean..."
"No!" she said, almost indignantly, eyes flashing. "It's there -- the scar -- exactly like Uncle Ira's!"
And you sense, of course, that however foolish it sounds, Wilma's right; her uncle actually isn't himself any more. That someone could be replaced, down to the detail of a tiny scar -- well, it gave me what the Scots call "the cauld grue." And later, when Wilma sees the doctor again, and laughingly tells him that she'd been acting so silly, of course Uncle Ira is the real Uncle Ira, I shivered even harder. Because that meant that they had gotten her, too.
The concept is also reflected in the legend of the doppelgänger, or "double walker," an individual out there lurking in the shadows who looks exactly like you. It's interesting how many cultures have a myth based upon this concept. The vardøger of Norse myth, the etiäinen in Finland, the ka of the Ancient Egyptians -- all were physical copies of your body, down to the last freckle, and if you happened to run into it, it could result in anything from bad luck to replacement to death. Even in the changeling myths of Ireland we see this concept; that the Elves could replace a normal human infant with an Elvish copy. The changeling, they said, would grow up wild and uncontrollable, and would have violent reactions in certain situations (especially in church during the sacraments). I've often wondered if this last story, however, was invented to explain chronically unruly children. "He certainly can't be my real son, any son of mine wouldn't act this way. I know... it was those damned Elves!"
It's also interesting to note that a number of famous people, including Goethe, John Donne, Percy Shelley, and Abraham Lincoln, all reported that they'd seen doppelgängers at some time during their lives (Donne's vision was of his wife, who was bedridden at the time).
So, what could cause such a pervasive myth? Once again, what we're probably looking at is a brain-wiring issue. A paper in Nature, published in 2006, described how a sensation of being in the presence of your double could be induced by electrical stimulation of the left temporal-parietal junction. This was discovered quite by accident -- the patient in question was receiving the procedure as a treatment for epilepsy (she was otherwise mentally completely normal). While the stimulation was being applied, she had the sudden, and unpleasant, sensation of having a duplicate of herself immediately behind her. As soon as the stimulation was ceased, the sensation vanished.
Our sense of identity is so wrapped up not only in our knowledge of our own minds and bodies, but in our feeling of uniqueness, that it is profoundly unsettling to consider even in a fictional setting that there might be someone who was our exact duplicate. That such a duplicate could replace us, and fool even our friends and family, is one of the creepiest ideas I know of. Our recognition of people we know is based on a mental network of knowledge, impressions, and emotional responses, both conscious and subconscious -- and when any bit of that network isn't working, it can result in one of the most disturbing and frightening delusions known to medical science.