The Prince of Wales – whose father Prince Philip is famed for his offensive gaffes – made the comment to writer Anita Sethi.
She said she was left “stunned” by the exchange.
“I feel angry that there could be such casual ignorance in the corridors of power,” she wrote in The Guardian.
Ms Sethi was meeting the royal at the Commonwealth People’s Forum in London, where she was giving a talk on injustice.
After being introduced to the prince, she says he asked where she was from. When she replied "Manchester", he reportedly quipped "Well, you don't look like it", before laughing and walking away.
The claim came as the heir to the throne was chosen as the Queen's successor as head of the Commonwealth – a multicultural institution with member countries from Africa, the Caribbean and Asia.
Ms Sethi said: “Whatever the prince meant or didn’t mean in our fleeting encounter, since it happened I have been through all the feelings – from shock to humiliation to rage.”
She added: “That the next leader of an organisation that represents one-third of the people on the planet commented that I, a brown woman, did not look as if I was from a city in the UK is shocking."
She said the remarks were reminiscent of Charles’ father Philip who has made a number of casually offensive remarks down the years.
Most famously, during a state visit to China in 1986 he told British students: "If you stay here much longer, you'll all be slitty-eyed."
Clarence House declined to comment.