From the very outset, the union of Prince Harry and Meghan, the Duke and Duchess of Sussex, had the makings of a 21st-century fairytale. It was a match that not only brought great personal pleasure to Her Majesty the Queen, then in her 92nd year, but would most certainly future-proof the Royal family.

When the dream went sour, and they left Britain for the US, it was she who insisted the door be left open for their return, post-Megxit. As a grandmother, she felt deep personal hurt at their departure. As the monarch, she understood what a huge asset the proverbial Firm was losing – something that is proving impossible to deny.

As the couple’s current quasi-royal Australian “tour” is amply demonstrating, they possess a star power that is, inarguably, the Royal family’s loss. The Queen may have been adamant there could be no halfway house, but the bitter irony remains that as far as the rest of the world is concerned, the Duke and Duchess of Sussex continue to fly the royal standard, whether officially endorsed by the House of Windsor or not.

“The Queen always adored Harry,” says Robert Hardman, royal commentator and author of Elizabeth II: In Private. In Public. The Inside Story, published to coincide with the centenary of the late monarch’s birth.

“She had an instinctive sympathy for the spare. Her shy father was a spare, unexpectedly thrust into kingship by the abdication of his brother. Her sister, Margaret, was a spare, and Elizabeth was always her greatest defender.

“And as a mother and grandmother, she was conscious that being second sons, both Andrew and Harry had far less clearly defined roles than their firstborn siblings, and that concerned her,” says Hardman.

By any measure, Hardman, 60, is an expert on the House of Windsor – an author, broadcaster and journalist, he has reported on royalty for more than three decades and has covered no fewer than 70 tours, from Anguilla and Abu Dhabi to Turkmenistan and Ukraine.

He first covered the royal beat at The Telegraph before his current berth at the Daily Mail, and has interviewed all the senior members of the Royal family. Tall, laid-back and public-school-posh enough to move seamlessly through their world, Hardman has an encyclopaedic memory, a dry wit and droll anecdotes a-plenty.

“The only time I ever got invited to a state banquet was years ago when I was making a documentary about the monarchy,” he says. “I shook hands with the president of Indonesia and then with the Queen, who turned to him and said, ‘This man has been following me everywhere for a whole year.’ She said it in a jokey way, but looking very regal. When it was to be translated for the president, a look of absolute horror crossed his face, and I could see him thinking, ‘Well, why haven’t you had him arrested?’”

Hardman’s front-row seat at major events has afforded him the opportunity not just to watch, but to analyse. Hence, his observation about the Queen’s innate compassion for the family “spares”. When Harry brought home his prospective bride, Hardman emphasises that she demonstrated nothing but kindness.

“In private, the Queen was much more smiley, much more sparkly than the dreadful way she was portrayed in The Crown, particularly by Olivia Colman and Imelda Staunton,” says Hardman. “Her Paddington Bear tea party sketch for the Platinum Jubilee showed off that side of her, and she loved doing it.

“Yes, everyone was scared of her, but not because she might be angry – she wasn’t an ogre – they were just terrified of letting her down. The Queen invited Meghan on the royal train on a solo trip together, which was a rare honour for new family members, and gifted her a pair of diamond and pearl drop earrings to make her feel special.”

It’s a far cry from the frosty, formal atmosphere described by Meghan, who went on to complain that she was initially left floundering and bewildered by arcane etiquette. But those and even more hurtful accusations would come later, long after the pomp and pageantry of a 2018 Windsor wedding.

As Prince Harry and Meghan took their place alongside Prince William and Catherine, then the Duke and Duchess of Cambridge, these two modern and modernising couples seemed to embody a dynamic new spirit – until the cracks started to appear.

Reports of tears and tension over tiaras and flower girls’ tights, frayed tempers, and whispers of Meghan being difficult and demanding. Then, wilful obfuscation surrounding the birth of Archie, and Meghan’s claim in an interview that she was left lonely and depressed after her son’s birth. Then, an irreconcilable rift between the brothers, confirmed by Harry, who announced: “We’re on different paths.”

It’s inconceivable that the Queen was in the dark about this, but she was distracted by Prince Philip’s ailing health – by then he had retired to Sandringham – and the growing furore surrounding Andrew’s friendship with the convicted sex offender Jeffrey Epstein, culminating in his disastrous Newsnight interview.

Ever the pragmatist, she may have assumed that her grandsons would put duty first and lay their differences aside. Nothing could have been further from the truth.

“The Queen recognised what a modernising force Harry was,” says Hardman. “In March 2012, she sent him to Jamaica during her Diamond Jubilee celebrations at a time when there were growing rumbles of Republicanism.” Her grandson was the perfect choice – he hugged the country’s prime minister and managed to “beat” Usain Bolt on the track by shamelessly cheating – returning a hero who had helped shore up the monarchy.

Just months later, she bestowed on him another great honour. Having opened the London 2012 Olympic Games with her astonishing James Bond cameo, she then announced she was travelling to Balmoral and would not be present at the closure.

“Everyone assumed that privilege would go to Prince Charles, or at the very least Prince William. But no, it was Harry who got this huge pat on the back,” says Hardman. “It’s a truly global moment, watched by billions of people. And when she gave it to Harry, we all thought, “That’s an interesting move.’ And, of course, what she knew, what he knew and what none of us knew at the time, was that he was going to be sent to Afghanistan that autumn.”

Fast forward to married life, and January 2020 saw a unilateral announcement that Harry and Meghan were stepping back as senior royals. They told the media they would be moving to North America, but would nevertheless return to Britain regularly and continue to play a role in public life – without properly consulting the Palace.

“As far as the Queen was concerned, Harry and Meghan had everything handed to them but wanted to throw it all away… She was enormously upset and terribly sad about a wasted opportunity, but she acted decisively.” Crisis talks were convened at Sandringham, and Harry was told there would be no half-in, half-out role. He would no longer be permitted to use the HRH title.

Despite giving the couple a cooling-off period, she privately conceded that she didn’t think they would change their minds because “they took the dogs”. As ever, she kept her counsel until the couple’s shocking Oprah interview of March 2021, in which they levelled accusations of racism and hostility. The Palace’s crisp response was both trenchant and unforgettable: “Recollections may vary.”

“I can quite easily imagine The Queen saying it,” says Hardman. “She had a very direct manner, possibly because there was a tremendous flapping around her. While other people would be getting into a complete stew, she would keep calm and was always terribly practical.”

Hardman’s assiduously researched, richly entertaining book – which melds historical fact with social history and intimate observations with revealing anecdotes – includes insights from courtiers, friends, politicians and, in a major coup, Donald Trump.

The US president spoke warmly of the Queen, but more interesting was her take on him. “One senior member of her staff said that she had found him ‘charming, tall, tanned, big, courteous, mid-century’ – not at all how he had been portrayed,” writes Hardman. Although he notes that during her reign, she was accustomed to dealing with “the most appalling despots, monsters, crooks and dictators”, so it was no small wonder she was pleasantly surprised.

Elizabeth II: In Private. In Public is Hardman’s sixth book on the Royal family. Although his latest work just about steers the right side of hagiography, his great affection and admiration for Elizabeth are evident. “I’ve always been a royalist, without ever thinking about it,” he says.

Hardman was brought up in Hampshire – his father was a lawyer, while his mother raised the four children. Hardman was the eldest and, after attending Wellington, he read classics at Cambridge. “All I ever wanted was to become a journalist,” he says. “I wrote for the school newspaper, and at university, I started doing the occasional shift for The Telegraph’s diary column.”

Later, having joined the paper full-time, he found himself dispatched to cover a royal story in Klosters, simply because he was the only reporter who could ski. “It was very telling that in 1992 there wasn’t a royal reporter, because nobody was much interested.”

Married to Diana, 57, a former solicitor, the couple live in west London and have three children in their teens, who (so far) have done the decent thing by not embracing republicanism. “We all watch the Queen’s Speech on Christmas Day, and we tuned into the Jubilee celebrations, but nothing too over the top. I did dedicate one of my royal books to the children, but to the best of my knowledge, none of them has read it.”

A host of other events are due to mark the centenary of the late Queen’s birth later this month. The King will host a party for 100-year-olds who share the same birthday (April 21), details of Lord Foster’s Queen Elizabeth Memorial will be unveiled, and the Princess Royal will open the Queen Elizabeth II Garden in Regent’s Park.

The day will be a moment of celebration – but also reflection. Where does the House of Windsor go from here? It’s four years since the Second Elizabethan era drew to a close, with her 70-year reign being the unchanging backdrop to our lives. Her likeness – the most reproduced in the history of mankind – was on stamps and currency in Britain and the Commonwealth. And when the only monarch most of us had ever known died in September 2022 at the age of 96, it was as if the world tilted on its axis.

“The Queen felt timeless,” recalls Hardman, “regardless of what it said on her birth certificate, she was always this matriarchal figure and extraordinary presence, and we believed her shoes were impossible to fill.

“But thanks to the way in which she laid the groundwork, King Charles has completely confounded his critics. There was a preconception that he would reign as Prince Charles, meddling and crossing constitutional red lines. But no, it’s a situation analogous to a barrister becoming a judge – it’s the same person but in an entirely different role, requiring an entirely different approach.”

We may not have the same loving protectiveness towards Charles as we did his mother, but no one can dispute that his exercise of soft power has been masterful – not least in the way he and Queen Camilla have handled the US president.

Their state visit to the US will go ahead at the end of the month, despite Trump’s rancorous complaints about Sir Keir Starmer and the state of the Royal Navy, where it is hoped the King and Queen will smooth ruffled feathers and improve international relations.

“State visits have real impact,” says Hardman. “Charles is a major figure. When I met Trump, he immediately asked about the King and his health. Trump said: ‘He’s fantastic, and he has fought very hard. He’s a fighter… We’re close. I have a really good relationship with him.’”

“Charles is a great guy, and he’s grown so much in the last 10 years and especially over the last couple of years as King,” Trump has also remarked.

Transparency over private matters, not just illness, is not the standard palace strategy. To that end, Elizabeth II: In Private. In Public casts light on a number of fascinating royal debacles. In the course of the book, we learn of Andrew’s churlish behaviour, from yelling foul-mouthed abuse at a groomsman who reprimanded him for impatiently revving his car at horses or deliberately performing a handbrake turn so gravel cascaded down onto visitors’ cars. On another occasion, he punched the Master of the Household, Vice-Adml Sir Tony Johnstone-Burt, over the availability of a reception room.

“Even by Andrew’s standards, it caused astonishment,” writes Hardman. “When the matter did reach the Queen and a senior member of staff tried to downplay it to spare her blushes, she was unfazed. ‘Oh, I’m sure he did it,’ she replied. ‘That’s the sort of thing he does.’”

Andrew received a dressing down from Lord Peel, Lord Chamberlain and a senior officer of the royal household. Prince Philip, who by this point had retired from public life, was so cross that he wrote a letter of apology to the Master. The following year, Sir Tony received a knighthood. Should the Queen have stepped in and been more forceful?

Hardman sets the record straight: “People want to blame the Queen personally for the whole Andrew debacle, and say she indulged him because he was her favourite. But having spoken to lots of people who knew her well, it had nothing to do with favouritism – he was just the one she worried about far more than the others. They were strong, resilient self-starters whom she knew would be OK. But Andrew had cut a fairly lost figure since he left the Navy in 2001. He didn’t have a wife, and he didn’t have a role. He was… different.”

When pressed, Hardman reveals that his own encounters with Andrew diverged markedly from those he had with other royals: “Philip was fascinating and sharp – you really had to keep your wits about you. Ditto Charles. The Princess Royal is fabulous – I’ve followed her around and could barely keep up. William and Harry can effortlessly work a room, charming everyone and putting them at ease.”

“But Andrew just isn’t as intelligent or as articulate as the others,” adds Hardman. “As trade envoy, he was forever saying disobliging things and putting his foot in it. He’d be at a telecoms conference and suddenly come out with the line, ‘What is Orange?’”

For now, Andrew remains an exile in his own country, banished to a remote corner of the Sandringham Estate. Hardman reveals that it was the Queen who first wanted him out of his previous, inappropriately grand house, Royal Lodge in Windsor.

Having stripped him of his military titles and patronages, and compelled him to step back from public life, the Queen funded – according to Hardman, “loaned” – her son an alleged £12m for an out-of-court settlement after Virginia Giuffre brought a civil case against Andrew in New York, alleging he sexually assaulted her on three occasions when she was 17, something he has always denied.

“Everyone would agree the Newsnight interview was a disaster, but I think the Queen operated on the principle ‘innocent until proven guilty’, and when Andrew said he had cut all links with Epstein, she believed him,” says Hardman.

“But then all these emails emerged showing he had lied on air, lied to the country and lied to family. The King acted swiftly and decisively by removing the title he was born with. I think one of the blessings in all this is that the Queen wasn’t around to live through the final degradation, and the brutal disgrace of a royal having their ‘princedom’ taken away. That hasn’t happened since a descendant of Queen Victoria was a traitor in the First World War.”

Time will tell what other consequences Andrew will have to face for his actions – another ugly distinction is that he was the first royal to be arrested in 350 years. Meanwhile, the rest of the show must stay on the proverbial road, even if the roster of working royals is shrinking past the point of “slimmed down” to worryingly sparse.

“Particularly in the late 1980s, early 1990s, there was a sense of this huge, rather bloated, Edwardian-style family,” points out Hardman. “The annual photograph at the Trooping of the Colour would reveal a whole balcony full of lesser-spotted Windsors, obscure Kensingtons and various people even the editor of Majesty magazine probably struggled to identify. There would be so many that there wasn’t room for them all.

“None of them was a drain on the public purse – they were just cousins who came to a party once a year. But the optics weren’t good. Some courtiers gently suggested trimming the guest list back, but as far as the Queen was concerned, it was her party, and she would invite whom she liked.”

King Charles is rather more aware of how things look, but even he must be concerned at the dwindling frontline – no Harry and Meghan and no Princesses Beatrice and Eugenie due to the scandal surrounding their father.

“I think it’s a crying shame that when William takes the throne, he and Catherine are effectively looking at quite a significant spell on their own, carrying the whole show. Edward and Sophie, the Duke and Duchess of Edinburgh, are stepping up, and the Gloucesters too, but they’re all going to be of a certain age. The Princess Royal is turning 76 this year.

“I think William’s view is that we’re all going to just have to accept things will be different. We can’t suddenly summon up extra royals. We’re just going to have to get used to a different sort of business model, if you like.”

Historians will look back at the smooth transition between Elizabeth and Charles – there’s no reason to believe the next one will be any different. Success resides in managing expectations and adapting to survive.

“No one thinks that the monarchy will end with a mob storming the gates,” says Hardman. “If it ever does – and I hope that it doesn’t – it will be due to a loss of relevance. When people wake up one day wondering, ‘What’s the point of that?’, then the writing will be on the wall.”