As romantic rumors fly, Michelle Dockery and Allen Leech shared their thoughts on the love stories looming in Downton Abbey season 4.

Fans of Downton Abbey have been batting about the idea of a possible love connection between the newly widowed Mary and the semi-newly widowed Branson since Sybil’s death in season 3. Some fans have been repulsed, others tepid, and some even enthusiastic. TV Guide recently broached the question to actors Michelle Dockery and Allen Leech – and the answer seems to be a very firm (and depending on where you stand, reassuring) “No.”

“Oooh, I can’t do that,” Michelle Dockery laughed in a recent interview. “[Branson is] my brother-in-law!” While Allen Leech isn’t quite as opposed to the idea (“It would be great for drama,” he admitted – and spiking our collective blood pressure, Mr. Leech!) he agrees that the pairing would likely be a disappointment. “I don’t think it would be right for those two,” he admitted. “As much as they respect each other, they come from very different worlds.”

But fans of Downton Abbey can look forward to perhaps a more stable bond forming between the former chauffer and Lady Mary. The respect Leech speaks of will continue to develop over the course of season 4 as Mary and Branson discover further solidarity as single widowed parents. “We will be spending so much more time together, from what we’ve already seen in the early episodes,” Dockery shared, “[and] we’re delighted by that.”

“It’s going to take them both some time before they find someone new,” the actress concluded – though it may not be quite so long as fans would like. According the the interview, Allen Leech has hinted at sparks potentially flying between himself and the new nanny hired to care for his daughter, baby Sybil. As for Michelle Dockery, she will soon be spending quite a bit of time with the newly cast Tom Cullen, who will play Lord Anthony Gillingham – a childhood friend, and Mary’s most likely love interest.

What romantic developments do you expect in season 4?