One Direction singer Niall Horan is recalling his struggle with fame at an early age.

When asked in a recent interview with British media personality Dermot O'Leary if he ever felt "like a prisoner" at the beginning of the English boy band's rise to prominence, Horan admitted he did "a couple of times" — particularly as the group evolved from being a fan-favorite act on The X-Factor to a worldwide force in pop music in 2011, which led to the "proper hysteria" of hordes of fans swarming their hotel suits, press tours, and even "banging on car windows" as they drove by.