It’s the era of enemies-to-lovers, and some Outer Banks fans have been pining after a rather surprising couple whose storyline on the show would be exactly that trope — but the actors aren’t too sure about the idea.

In season 3 of the Netflix series, the interminably angry Rafe Cameron (Drew Starkey) and Kiara Carrera (Madison Bailey) became unlikely allies as they were both kidnapped and held hostage together by a fellow gold hunter. It was the first time the characters had spent any real time together onscreen, and some fans saw their tension as something that could have a romantic tinge to it — but Starkey and Bailey were not on the same wavelength.

“Madison and I were like, ‘Wait, what? No, that doesn’t make much sense,'” Starkey, 30, tells PEOPLE of his and his costar’s response to fans rooting for a romance between Rafe and Kiara.

“Yeah, I mean, we talked about this before. I think it’d be especially unfair to Kiara, you know?” he continues. “I think she deserves a little better than that.”

Chase Stokes, who plays Kiara’s lifelong friend John B. and the love interest to Rafe’s sister Sarah Cameron (Madelyn Cline), also weighed in, telling PEOPLE of the pairing, “It actually makes no sense in my brain.”

Madison Bailey as Kiara, Drew Starkey as Rafe in ‘Outer Banks’ season 3.Jackson Lee Davis/Netflix

In season 4, Starkey says that his character has “found someone of his own, someone who makes him feel comfortable this season and who loves him for where he’s at, and I think that’s huge,” referring to Sofia (Fiona Palomo), who fans first met in season 3.

“I mean, I think he was in a place of drowning in pessimism about the world. He’s lost the only person that he’s really, really cared about and looked up to,” Starkey continues, referring to the death of Rafe and Sarah’s father Ward Cameron (Charles Esten) in season 3.

“And now, all of a sudden this person comes out and he can find comfort in another human being and be himself. So yeah, I’m happy for him.”

Aside from finally having somewhat of a stable partner for the first time in the upcoming season, Rafe remains plagued by his grief over his father’s death. Starkey says of playing into those emotions, “I think there’s something internal that’s always underneath the surface. It’s like a part of you is also lost when you lose someone that close to you.”

“I know in Rafe’s experience at least, I just think he’s kind of turned more towards apathy about the world,” he continues.

“I mean, he has a future in mind with reconnecting with his family and building his family, continuing his legacy, his family’s legacy, but that’s about it. But yeah, I mean, I think we all lose a part of ourselves in a grieving process.”

Charles Esten as Ward Cameron, Drew Starkey as Rafe in ‘Outer Banks’ season 3.Jackson Lee Davis/Netflix

Season 4 picks up with the Pogues — Kiara, John B, Sarah, JJ (Rudy Pankow), Pope (Jonathan Daviss) and Cleo (Carlacia Grant) — building Poguelandia 2.0 back in the OBX after finally finding the gold at El Dorado. But as always, they’re never really out of the G game and when Wes Genrette asks for their help finding Blackbeard’s lost treasure, the gang is back on the hunt.

Rafe, meanwhile, has his sights set on some major money moves of his own in their hometown as he finds himself tempted to work with real estate shark Hollis Robinson (Brianna Brown) on a risky but potentially lucrative project in the cut.