In the real world, the issue with inbreeding is that it increases the risk of recessive genetic disorders. In fiction, it seems that the single most common recessive genetic disorder is evil.
This allows for the usage of tropes such as Mental Handicap, Moral Deficiency and Evil Cripple while obfuscating the ableism of said tropes. Apparently it's fine to vilify people's genetic disorders as long as their disorder was caused by incest rather than random chance.
The reasons for this trope are varied. Taboos against incest are near-universal, strong, and have only gotten stronger with time. Just as illegitimate children and children of rape have historically been shamed for their irregular parentage, so too are children of incest. Some examples overlap with those tropes, although not all—if all examples of inbreeding did fall under those taboos, there'd be no need to have a separate taboo against inbreeding.
Alternately, inbreeding may provide a convenient justification for the employment of certain tropes. Need to explain how the isolated Cannibal Clan keeps recruiting, and why its members are deformed, disabled, or mentally handicapped? Want a reason for why The Caligula is so Royally Screwed Up? Blame a recessive genetic disorder or hereditary mental illness and claim that the issues were exaggerated by generations of inbreeding!
This trope can be found on both ends of the socioeconomic spectrum, overlapping with Aristocrats Are Evil and Royal Inbreeding as often as it does with Hillbilly Horrors and rural Bandit Clans and Cannibal Clans. Very rare among the middle-class.
Sub-Trope of Villainous Lineage and Hollywood Genetics. See Red Right Hand, Evil Cripple, Mental Handicap, Moral Deficiency, Insane Equals Violent, and The Caligula for what the trope might be used to explain away. May overlap with Villainous Incest on the part of the parents.
Needless to say, this is not how inbreeding works.