Kwasi Kwarteng claims the "general direction" of his economy-crashing mini-budget had been "right", but admitted he does hold regrets about his brief time as chancellor.
Despite tanking the pound's value in the autumn with unfunded tax cuts, Mr Kwarteng said his political beliefs had not changed since being dismissed.
He said: "I don't think you get to prosperity by high taxes."
The former Chancellor of the Exchequer said it was "regrettable" that pharmaceutical giant AstraZeneca had chosen to build a £320million factory in the Republic of Ireland rather than the UK, with the firm citing Britain's higher tax regime as a factor.
He argued that losing investment was a "risk" that countries with high business taxes face, with corporation tax due to rise in the UK from 19% to 25% in April.
In his first live interview since being sacked from No 11 by former prime minister Liz Truss, Mr Kwarteng defended the "general direction" of his quickly reversed mini-budget.
His £45billion package of unfunded tax reductions, the largest seen for half a century, panicked the markets and sent sterling tumbling, while also pushing up mortgage rates.
Mr Kwarteng was sacked soon after, with Ms Truss exiting Downing Street only six days later, making her the shortest-serving prime minister in British history.
"It was a turbulent time," Mr Kwarteng told Piers Morgan Uncensored programme presenters Isabel Oakeshott and Richard Tice.
"We can laugh about it but we were trying to do a serious thing, we tried to reduce the tax burden on this country, and we were caught in a real firestorm. I think it is a shame what happened.
"I do regret some of the things we did in terms of the speed, but I think the general direction in terms of lowering taxes was right."
With the Budget due to be given next month, the Tory MP refused to criticise the handling of the economy by his successor Jeremy Hunt.
The former foreign secretary was appointed by Ms Truss and kept on in 11 Downing Street by Rishi Sunak after he became prime minister.
Mr Kwarteng said the pair were "doing a good job" of stabilising the markets and highlighted that the prime minister and chancellor had talked about wanting to reduce taxes "when it is appropriate to do so".
"The question is, when is that?" Mr Kwarteng said.
The former cabinet minister said he stood by his political stance that high taxes can impede growth.
He told the programme on Thursday: "I've always been very clear that I don't think you get to prosperity by high taxes, I don't think that's the route to do it.
"There is always a risk when you put taxes up that people who generate wealth decide to up sticks and go somewhere else, and that's what Pascal Soriot, the head of AstraZeneca, ... he made that very clear in his statement."
Mr Kwarteng said the mood in the Sunak administration was "let's try and stabilise things and not get too excitable or agitate the markets".
He described Mr Hunt as having a "very orthodox Treasury ideology".
He added: "It is not about being boring or exciting. It is about an approach to economic management. And I think what he is doing is trying to stabilise the markets."
Vote Leave campaigner Mr Kwarteng also said there was a "risk" Brexit could be reversed.
His comments came after Eurosceptic anger was stoked following a so-called "secret" summit, which was reportedly held earlier this month to discuss how to make Brexit work better.
"They are not stupid, they are not going to say, 'oh let's rejoin the EU'," he continued. "But they'll dress it up in some form of words, which essentially means it will be the EU, on worse terms."
Mr Kwarteng's intervention comes only hours before Ms Truss, who he was once closely allied with, was due to give a speech in Japan where she was expected to call for a tougher approach to dealing with China.
Earlier this month, in her first interview since being forced out, Ms Truss vowed to continue making the case for domestic tax cuts.
