Vardy made the remark in an interview in 2001, saying the 'Mysterious Girl' singer had the "smallest trouser equipment" she’d seen.
Asked about it on Loose Women on Friday, he quipped: “Are you asking me to address the elephant in the room?”
“Did I say elephant?” he asked, as the panellists giggled.
Becoming serious, Andre said it was difficult that the story kept coming up again and again.
“On reflection on that, I think it is not the fact that something was said, because I think we all say stupid things right?” he said.
“It is more about the fact that something is brought up again and again and again and again. At some point you kind of go, OK, like...
“And that is the only thing, it becomes more than just that. We all say stupid things. Everyone does. I am sure I have said many stupid things in my life.”
In his Instagram video, the singer said he’d been teased for years and that there would be “outrage” if someone had made similar comments about a woman’s anatomy.
“The only one that sits there and takes hit after hit about it is me,” he said.
“Think about how it would feel if it was the other way around, that's all I'm saying.”
“It's not fair that something like this can happen again and again and again and we talk about all sorts of things, about mental health, about being kind and all that sort of stuff, and nothing seems to change,” he added.
Watch: Peter Andre speaks out about Rebekah Vardy's 'chipolata' jibe