From Thursday's decision in In the Matter of Accounting by Weber, decided by Saratoga County (N.Y.) Surrogate's Court judge Jonathan G. Schopf; the expert was opining on damages in a financial dispute:
Use of Artificial Intelligence
Although the Court has found [proposed expert witness Charles Ranson's] testimony and opinion not credible [see below -EV]…, a portion of his testimony bears further and separate discussion as it relates to an emerging issue that trial courts are beginning to grapple with and for which it does not appear that a bright-line rule exists.
Specifically, the testimony revealed that Mr. Ranson relied on Microsoft Copilot, a large language model generative artificial intelligence chatbot, in cross-checking his calculations. Despite his reliance on artificial intelligence, Mr. Ranson could not recall what input or prompt he used to assist him with the Supplemental Damages Report. He also could not state what sources Copilot relied upon and could not explain any details about how Copilot works or how it arrives at a given output. There was no testimony on whether these Copilot calculations considered any fund fees or tax implications.
The Court has no objective understanding as to how Copilot works, and none was elicited as part of the testimony. To illustrate the concern with this, the Court entered the following prompt into Microsoft Copilot on its Unified Court System (UCS) issued computer: "Can you calculate the value of $250,000 invested in the Vanguard Balanced Index Fund from December 31, 2004 through January 31, 2021?" and it returned a value of $949,070.97—a number different than Mr. Ranson's. Upon running this same query on two (2) additional UCS computers, it returned values of $948,209.63 and a little more than $951,000.00, respectively. While these resulting variations are not large, the fact there are variations at all calls into question the reliability and accuracy of Copilot to generate evidence to be relied upon in a court proceeding.
Interestingly, when asked the following question: "are you accurate", Copilot generated the following answer: "I aim to be accurate within the data I've been trained on and the information I can find for you. That said, my accuracy is only as good as my sources so for critical matters, it's always wise to verify.
