Brown, 35, won the TV baking competition in 2016 and married boyfriend Liam Macaulay in France in 2018. But this week it was revealed the couple have split after less than two years of marriage.
Macauley told MailOnline: “There is nobody else involved. We have just sadly decided it wasn't working.”
But the tree surgeon admitted the photograph of Brown and Bake Off judge Hollywood sharing an “intimate kiss”, published in The Sun newspaper in 2017, had caused tensions in their relationship.
He said: “I'd not long asked Candice to marry me and I had to read reports that my new fiancee was seen kissing Hollywood and getting close to him.
“That was just a kiss on her cheek and there was nothing else to it. But we had only been engaged a few days. It was a lot to have to deal with.
“Sometimes your mind plays tricks on you. It can tell you you're no good, that it's all hopeless.
“But remember this; you are loved, and important and you bring to this world things no one else can. So hold on.”
Macauley said he had moved out of the flat he and Brown shared above the pub they own in Milton Keynes.
As news of the split was made public, Brown tweeted: “Think I’m going to eat a lot of biscuits today.”
Last October she opened up to White Wine Question Time about her fertility struggles.
She said of being asked about plans to start a family: “People have got no idea that sometimes your body doesn't work the way you want it to and that's really hard, but people don't understand that.”
Shortly after the picture of Hollywood kissing Brown on the cheek was published in 2017, the Bake Off judge and his wife Alexandra announced they had separated after almost 20 years together.
Alex previously took Hollywood back after they split in 2013 when the baker confessed to having an affair with Marcela Valladolid, his co-star on the US version of Bake Off.
The TV judge called the incident “the biggest mistake of my life”, with the star winning his wife back two years later.