Continuity -- or lack thereof -- has always bugged me as well. Not so much with shows like Family Guy, where they don't even try to maintain continuity, but with other shows where they screw it up and think no one is going to notice.

One example I can think of is "Cheers", which later spun off "Frasier" when the original show ended. In the spin-off, Frasier's father is a retired cop. But years previously, on Cheers, Frasier had mentioned that his father was a professor, and had passed away.

Another is the X-Men movies. I loved them, until they screwed up the continuity in the later ones, starting with First Class. At the end of the movie, they explain why Professor X in wheelchair-bound by showing him get a bullet in the spine. That one really annoyed me, because I remember flashback scenes where the professor and Magneto were still working together, still older men, and the professor could still walk. And then in the new movie, they show him getting crippled while he's still a young man, and he and Magneto parting ways at the end of the movie. It mostly got good reviews, and I'm practically screaming inside, "But they screwed it up!" I did, however, find one or two reviews where someone else pointed out they ruined continuity.

I think for me it comes down to how vitally central something is. I've noticed if something greatly affects the story, "I can only go through that wall if I'm wearing a blue shirt and I'm wearing red shirts so I can't pass through that wall" and it's vitally important that they pass through that wall then it will piss me off later if suddenly the color of shirt doesn't matter because that rule only existed to create a Problem that didn't need to exist in the first place.