A royal biographer said Prince Harry and Meghan are "putting pressure on the Queen" by attending Platinum Jubilee celebrations.
Angela Levin also claimed the visit is "badly timed", adding the couple are "trying to grab a few shots that will help them with their Netflix career".
However, royal commentator Emily Andrews contended the Queen's invitation will "hopefully be not only an olive branch", but also provide a "rapprochement within the Royal Family itself".
The comments follow the Sussexes being picked up by a Land Rover sent by the Queen after arriving by private jet from California for the four-day-long celebrations.
It is the first time that Harry and Meghan have been back in the UK together since they moved to the US over two years ago.
Follow all the Queen's Platinum Jubilee celebrations with our live blog here
Speaking on Good Morning Britain today, presenter Adil Ray asked: "If Her Majesty the Queen - and this is her day, her moment - if she is saying 'well I want all my family there, including Andrew, Meghan and Harry', who are we to deny that?"
The author responded: "Well I agree, she's the mother, the grandmother, she should make the decisions. She's 96, it's her day. But I don't think you can then influence all the population to feel exactly the same.
"We like the monarchy, I'm a really keen monarchist, and I think that it's badly timed. Harry and Meghan shouldn't come because they are putting more pressure on the Queen. She's got a very hard four days."
Mr Ray pointed out: "But the Queen would have welcomed them, so I don't know what the pressure would be. Maybe it's making it better for the Queen, maybe she feels more comfortable that they are there."
Ms Levin said: "No because she's not very well, and she's got mobility problems. I think that for someone of her age to have four full days, one after the other - "
The presenter interjected: "I don't think they forced themselves, they didn't say we're coming no matter what. They've been invited, they've been given full security, so the Queen has endorsed it. So how in any way is this going to be bad for the Queen?"
Ms Levin replied: "Well I think that the people who don't want them to come would agree with me that their behaviour has been appalling.
"Therefore they should be ashamed and not come at this time, and try to grab a few shots that will help them with their Netflix career."
A convoy of cars took Harry, Meghan and their two children Archie and Lilibet from Farnborough Airport to Frogmore Cottage, which used to be their home before they moved abroad.
On the decision by the Queen to send a car for the Sussexes, a Sun insider said: “The Queen believed it was the right thing to do.”
Harry and Meghan will not be on the Buckingham Palace balcony today after the Trooping of the Colour but will be involved in other festivities.
Royal correspondent Ms Andrews said it is "great" the public will get to see the pair, adding: "We saw in the Queen's statement today, she thanked communities, families, neighbours, there was a real focus on family.
"I think inviting Harry and Meghan to the Jubilee celebrations will be hopefully not only an olive branch from the Queen to them, but also maybe hopefully might provide a rapprochement within the Royal Family itself."
Meanwhile, reports have also suggested that tomorrow, the couple will finally formally introduce Lilibet to the Queen on her first birthday at a possible party.
On Saturday, it is the Epsom Derby - an event that the Queen often attends - but it has been reported she may no longer be planning on going.
The 96-year-old monarch, who has been suffering mobility problems, is said to want to pace herself during her milestone celebrations, with her daughter Princess Anne representing her at the racecourse instead, The Sunday Times reported.
The Sussexes may also attend the Jubilee Pageant, which is taking place on Sunday, although this has not been confirmed.
