Ever since the advent of agentic AI, security researchers have been yelling from the top of their lungs about how it's a bad idea to grant user-level permissions to an LLM — for all purposes, a program with non-deterministic outputs and inconsistent handling of inputs. A research paper on HalluSquatting from researchers at Tel Aviv University, Technion, and Intuit, shows how easily one can fool modern AI bots and harness them into a massive army of AI agents, with the research showing that agents can hallucinate potentially malicious code repositories up to 85% of the time.
The mechanism for HalluSquatting (aka "adversarial hallucination squatting") is surprisingly simple, and takes advantage of the fact that when met with unfamiliar terms, bots will not know they're incorrect and hallucinate a "correct" answer. Adding to that, the methods the bots use to come up with said answer are predictable, for example, owner/repository or toolname/toolname GitHub URLs. This is different than just standard typo-squatting, as it exploits the hallucination mechanism itself.
