The Affair star West, 51, plays 32-year-old James' father in the BBC adaptation of Nancy Mitford novel The Pursuit of Love, and the pair hit headlines in October over some photos of them looking very close in Rome.
In the pictures, James and West rode a scooter around the city together and enjoyed a cosy lunch, where West appeared to be nuzzling her neck and kissing her.
Speaking to The Guardian about the media storm around the photos, James said: "Ach, I’m not really willing to talk about that.
"There is a lot to say, but not now, I’m afraid."
After the photos emerged, West and his wife Catherine FitzGerald, who he has four children with, put on a united front outside their home where they hugged, kissed and handed a handwritten note to press which read: "Our marriage is strong and we're very much still together. Thank you."
West and FitzGerald have been married since 2010, but in 2016 he told the Evening Standard: "I mean, I think women should be more indulgent of affairs.
"I really do. It’s daft to kick someone out over a fling. Isn’t it? Everyone should turn a blind eye to men’s behaviour between the ages of 40 and 50. Let it all blow over."
Downton Abbey actor James also talked about another of her Pursuit of Love co-stars, Andrew Scott.
She said: "Oh, he’s extraordinary. He brings so much. That’s another beautiful relationship, Lord Merlin and Linda (their characters)…they’re true soulmates. Andrew and I talked about how you don’t have to define some friendships. All sorts of people can become great loves of your life.
Asked whether she had bought into "Hot Priest mania" over Scott's starring role in Fleabag, she said: "Of course I did. What a show. He was catapulted into everyone’s heart with that part.
Watch: Lily James and Dominic West were 'flirtatious' on Pursuit of Love set