It's probably the most instantly recognisable Scottish site from Outlander. Midhope Castle lies just 30 miles outside of Edinburgh and is the real-life location that is used to represent Broch Tuarach (north-facing tower in Gaelic).
Showrunner Ronald D. Moore said that this 16th-century townhouse fits the "mental image" that they had of Lallybroch (meaning lazy tower according to the Outlander Wiki). It was first used as an exterior film setting during both Season 1 and 2 of the hit show.
It was one of the first locations scouted as it was needed for the flashback scenes where Jamie (Sam Heughan) first faces Black Jack Randall (Tobias Menzies), it's since been used for filming several times and is even set to return in season 7 with filming having recently taken place there.