Online game chats are notorious for vulgar, offensive, and even criminal behavior. Even if only a tiny percentage, the many millions of hours of chat can accumulate a lot of toxic interactions in a way that's a problem for players and video game companies, especially when it involves kids. Roblox has a lot of experience dealing with that aspect of gaming and has used AI to create a whole system to enforce safety rules among its more than 100 million mostly young daily users, Sentinel. Now, it's open-sourcing Sentinel, offering the AI and its capacity for identifying grooming and other dangerous behavior in chat before it escalates for free to any platform.
This isn’t just a profanity filter that gets triggered when someone types a curse word. Roblox has always had that. Sentinel is built to watch patterns over time. It can track how conversations evolve, looking for subtle signs that someone is trying to build trust with a kid in potentially problematic ways. For instance, it might flag a long conversation where an adult-sounding player is just a little too interested in a kid’s personal life.
