Rumors have been swirling for weeks about Dan Stevens’ intentions for the future of ITV’s Downton Abbey, and now several news sources announce that he has made his decision.

Grab the handkerchiefs, Downton Abbey fans; it looks like Cousin Matthew may soon be no more at the old estate. The Sunday Express has reported that actor Dan Stevens has decided not to return for future series of the British period drama. Allegedly, a “well-placed production source” provided the news that Stevens was planning to depart the show. “He will probably do the first episode of the fourth series,” the source claims, “But that will be it.”

Rumors of Stevens’ potential departure had been circulating ever since it became known that the actor had not yet renewed his contract for Downton Abbey‘s fourth series. The suggestions were compounded by Stevens’ apparent interest in opportunities elsewhere, including a stint in Broadway’s The Heiress, opposite Jessica Chastain. Still, Stevens made an obscure and somewhat reassuring reference to the rumors on his Twitter, and ITV had not yet renewed Downton Abbey for a fourth series.

But given the exciting news that Downton Abbey will return for another go-round, it seems that the time for assurances may be past, though no official statement has come as yet from either ITV or Stevens himself.

It is hard to imagine how showrunner Julian Fellowes will manage if Stevens does in fact leave Downton Abbey, as Stevens’ character Matthew Crawley has been central the show’s swirling stories. Matthew’s tempestuous relationship with Lady Mary is at the heart of Downton Abbey‘s romantic drama, and his presence has highlighted issues of class, privilege, and the price of war throughout the show’s three series. Fellowes did draw up a very powerful episode to provide an exit for another budding star, but it would be unfortunate if he had to work similarly to remove another popular character from the rich tapestry of Downton Abbey.

In the meanwhile, Downton Abbey fans can look forward to the upcoming Christmas special to cap off series 3. The special promises “both laughter and tears” as the Crawleys venture up to Duneagle Castle in the Scottish Highlands. (Thanks to Hypable reader Anna S. for the tip!) The episode will air Dec. 25 at 9 p.m. on ITV.