It may have taken One Direction two years since its X Factor stint to make a splash in the U.S., but Simon Cowell says he knew almost instantly that the band would be a hit.

In an interview with Rolling Stone, Cowell described his first meeting with the One Direction boys and discussed their quick rise to fame in the United States.

How quickly did you realize that they could be huge?
(Chuckles) When they came to my house in Spain and performed, after about a millionth of a second. I tried to keep a straight face for a bit of drama for the show. I remember sitting next to this girl who I was working with. The second they left we jumped out of my chair and said, “These guys are incredible!” They just had it. They had this confidence. They were fun. They worked out the arrangements themselves. They were like a gang of friends, and kind of fearless as well.

For more of Cowell’s thoughts on the band, check out the full interview.