I loved his persona and characters on Letterman in the '80s. That core of disdainful hostility he would bring... it just cracked me up and wasn't like a joke you could repeat, if that makes any sense. I need to comb YouTube again for Chris Elliott clips from those days... I think I liked The Guy Under The Seats best but, he was always fun. I also have fond memories of his stint in Mummenschanz and from the Late Night episode (actually it might've been a week?) when people's parents were there and Chris's parents didn't show up, and the false front he put up about how great it was no 'old people' were in his office.
I also remember enjoying the short films he did for whichever Letterman show in the early '90s. Poolside Ecstasy, The Traveling Poet... I know there were more.
And of course I loved Get a Life. I think the 'Cousin Donald' episode was my favorite.
He is just so funny.