Livia made out to Terry, Tony, Marcia and anyone else who would listen that she was desperate for more children. She let it be known, casually, telling people discreetly, one at a time, as befits a secret, that Terry was responsible for the lack of fertility. Given Terry's character, this amplified the problem.
Terry still adored Livia, but sex between them became a relatively rare occurrence. Accusations and the continuing elephant in the room of infertility simply put him off sex.
But Terry had already proved his masculinity, via Luka. Maybe time to prove it again, some other way. Maybe.