Fry worked at the team in its Benetton days from 1987 to 1993, before embarking on a career that encompassed two spells at McLaren, and one at Ferrari.
He returned to the Enstone camp in 2020, but after contributing to the last three cars he decided to leave earlier this year.
By a coincidence of timing, his move to Williams became public on the same day that the sudden departures of team principal Otmar Szafnauer and sporting director Alan Permane were announced.
After a short gardening leave, he began work at Grove earlier this month.
Speaking for the first time since he left Alpine, Fry said the team made good progress during his stint there, but he sensed that further steps would be harder.
"I look back at the first three years I was there, and we improved Enstone, dramatically," he said when asked by Motorsport.com to explain his move. "Year-on-year, we built a better car.
"If you put the three cars next to each other, each one was a massive step. It's a credit for everyone there, the various teams were collaborating a huge amount better. I think everyone there should be proud of what we achieved for those three years.
"I guess I'd gone back there [thinking] to go back to the place you started your career and try and rebuild it. And I think we did really well. From a distant fifth, we were a solid fourth.
"But I didn't feel there was the enthusiasm or the drive to move forward beyond fourth.
"I decided at the start of March that I want to be pushing things forward, I don't just want to sit there and not be able to do things. So for me, that was time to stop and move on, really.?
"It's one of those things, I think as a company, they weren't almost set up to push hard enough, you can say you want to be first. But the difference between saying it and achieving it, is monumental, as we all know."
Asked to elaborate on what was lacking, he indicated that he was referencing parent company Renault rather than the race team.
"Enstone as such, the destiny that we were in charge of, we could control," he said. "And I think we did a good job.
"I'm not so sure that Otmar got a fair chance at fixing the place, because to some degree I think metaphorically, your hands are tied, I guess.
"But as I say, I think everyone there should be proud of what we achieved in those first three years.
"It's always a shame walking away from things. But I think for me, I'd taken them as far as I could. And it was time for me to put my feet up and sit in my garden!"
Fry said that after he had decided to leave Alpine, it took Williams team principal James Vowles two months to convince him to join.
"James had been talking to me for a little while, and it wasn't until another couple of months after that that I decided to come here," he said.
"I guess the thing that excites me about this opportunity is the board is fully on board with what it's going to take to move this place forward. They're willing to invest what it takes and support us in building a team.
"And again, it's a nice thing isn't it, to rebuild an old British icon. It's a bit like my romantic view of going back to Benetton, to rebuild them. So it's another exciting prospect."
He added: "James is pushing hard to try and improve this place. As I said the board is fully behind him moving the pace forward, and that's the thing that excites me, we're not going to be limited in what we can achieve.
"We just got to do the best we can in the time and move things forward."
