Jimmy Fallon admits this is his “biggest fear” related to his career

Comedian Jimmy Fallon got serious about his greatest fears.

In the Thursday, October 31 episode of Hotthe comedian, 50, sat down with the host Sean Evans ahead of the November 1 release of his comedy album, Holiday spices.

While Fallon may have a successful career as a late night talk show host and actor, he says there is still one thing that scares him more than anything when it comes to his work.

“As a comedian: silence,” Fallon admitted. “The deafening silence is my biggest fear, crying in front of the camera, which is happening now, my biggest fear. Probably my biggest fear is to give up in front of the American public, and I won’t do that.”

The Tonight’s show starring Jimmy Fallon star also shared his best moment from his run from 1998 to 2004 Saturday evening live.

“’Cowbell’ – it’s just crushed, I mean next level,” he said of the classic skit that aired on April 8, 2000.

The sketch, titled “More cowbell” featured Fallon, Christopher Walken, Will Ferrell, Chris Kattan, Chris Parnell And Horatio Sanz portraying rock band Blue Öyster Cult during a recording session.

“Everyone was so funny in that sketch,” Fallon told the 38-year-old Evans. “Christopher Walken didn’t even talk like a human being, he pretended to be an impression of himself.”

Fallon also recalled how one of the scene’s iconic moments came about.

“I remember I had one line and just as I was about to say my line, Kattan and Will improvised and pushed each other,” he said. “Kattan knocked off Will’s sunglasses and I could see the eyes of a madman and I started laughing, and that gave me my reputation as the guy who laughs during all the sketches. That was great, Debby Downer, I was part of that draft. I was very solid at that until the end of the road, the road, the road.

Over the years, Fallon has worked on many comedy sketches with big names.

During his visit to Hotthe Fever height actor also shared the time he realized how serious it was Justin Timberlake takes his trade.

Never miss a story – sign up PEOPLE’s free daily newsletter to stay up to date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

“He’s one of the most talented people on the planet, he’s a perfectionist,” Fallon said of the “Selfish” singer.

“I remember we were doing SNL, and we’re doing Barry Gibb’s talk show, so we’re about to go on, like 20 seconds to air, and he says, ‘Hey Jimmy, remember, press that harmony,’ and I go, ‘Yeah, I will,’ and he squeezed my hand and said, ‘I mean it, dude, don’t forget to touch that harmony.’ And I’m a calm guy, we’re dressed like the Bee Gees, for that second I felt what it was like to be Lance BassFallon joked, referencing Timberlake’s *NSYNC band member.