Indigenous leader Noel Pearson has called on Labor to "respect its base" and push ahead with a referendum on recognition if it wins the federal election.
Mr Pearson also revealed at a Judith Neilson Institute event in Sydney on Wednesday a conversation he had with Malcolm Turnbull, in which the then cabinet minister supported an Indigenous Voice to Parliament.
Discussing his new book, Mission, with former prime minister Paul Keating, Mr Pearson said it was well past time to recognise Indigenous people in the constitution.
"We've had four parliamentary processes, three public inquiries - there has not been a public policy issue that has been as protracted and as involved as that to do with constitutional recognition of Indigenous Australians," he said.
He said he feared Labor under Anthony Albanese was "in danger" of not progressing the change.
"The blackfellas are the base of the Labor Party," he said.
"There's probably no other group in Australia that more faithfully votes for Labor than Aboriginal and Torres Strait Islanders, and yet the Labor Party's respect for that base is often found wanting.
"They have abandoned us in the past, like Bob (Hawke) did with national land rights in the 1980s. At the drop of a hat they'd abandon us."
Mr Albanese has pledged to create a treaty-making Makarrata commission and put the Indigenous recognition question to a referendum in his first term.
Mr Pearson also revealed he went to Mr Turnbull in 2015 with the idea of an Indigenous Voice in the constitution.
"He says, 'this is a very sensible idea, the Voice' ... a very sensible compromise," Mr Pearson said.
"He said it was a better idea than putting a clause prohibiting racial discrimination in the constitution."
However, Mr Turnbull went on to argue as prime minister such a move was impractical because it would represent a "third chamber" of parliament.
"So my respect for Malcolm completely plummeted after 2017 when he took the position he did - it was an outrageous lie, an outrageous misrepresentation of the Voice idea, but thankfully the idea has survived him," Mr Pearson said.
He said he remained disappointed "the conservatives haven't risen to the occasion", as the Voice was a "necessary institution".
Last week's federal budget set aside $160 million in the contingency reserve for a possible referendum.
