You may not be aware that the punch line in your skit has been used as a defense in nearly every hate crime perpetrated against transgender people that has come to trial. For example, the "trans panic" defense was infamously used by Allen Ray Andrade, who was convicted in 2009 of beating 19-year-old Angie Zapata to death with a fire extinguisher after learning of her gender history. According to media reports, it has also been the main defense employed by Juan A. Martinez for the killing of Jorge Steven López Mercado, 19, in Puerto Rico last November.
In the letter sent to Letterman and CBS Entertainment president Nina Tassler, the group calls the skit "inappropriate and incendiary" and asks that Letterman apologize to Simpson and the transgender community.
In all likelihood Letterman's writers weren't aware of the "trans panic" defense and were just patting themselves on the back for not making an "A-MAN-da" joke.
The blog Queerty actually defended the skit saying:
If they're willing to give Letterman a smidge of a pass, there's another way of taking this joke: By understanding the funny part is about Simpson being hot. Which - have you seen her? - she is. And that she's accomplished passing as a woman - the ultimate goal of so many transgender women. If Kalter was fantasizing about her (thus, his being so upset upon learning she was born a man), that means her transition was that good.
It's a stretch to read the joke as compliment, but it isn't rabidly transphobic either. While comedy writers understand what is and is not appropriate when it comes to mocking other groups, transgender issues are only beginning to gain visibility (and Simpson's appointment is an important step). Presumably, Letterman's writers weren't aware of the level of violence and persecution that trans people face and are only guilty of having a sophomoric sense of humor. However, the way to change that is to draw attention to the story Angie Zapata and others as the HRC has done, not to give the comedians a pass for being ignorant.
Human Rights Campaign Sends Letter Condemning CBS Late Show With David Letterman Skit, Asks For Apology [HRC.org]
Was David Letterman's Joke About Amanda Simpson Transphobic? Or Actually A Compliment? [Queerty]