Found an interview where Tom talks about this.
How were you involved with Late Night with Conan O’Brien?
Tom: (Late Night writer) Andy Blitz created this boy band on the show, Dudez A-Plenti, and was like “We’re bringing them back, I want you to be the craft service guy, working the bagel station,” and I was like “Ok, I’ll do that.” [Laughing] I go, they have the boy band in there, and Conan just starts doing his thing; I’ve never seen anyone think as fast as him. It was mind-boggling watching how he worked in that capacity.
Then there was a moment where the awesome wardrobe lady, who was this terrible monster who’s name I don’t even know, [Laughing] sees me and goes “Oh, you’re not that fat,” like I’m supposed to be some Walter Hudson, rolling in. And then they’re like “all right, for this next thing we’re going to need you to take your shirt off,” and I was like “No, I’m not doing it.” “Well, we need you to do it.” “Well, then you made a mistake, because no one said anything about me taking my shirt off, and I’m not taking my shirt off at all.” Absolutely not, and I did not. But that horrible woman was like [Grouchy voice] “You’re not as fat as I thought you were going to be.” It’s unbelievable that humans can say that to other people, but she had no problem.