Media
1 item
Description
Athena is a fully open-source AI detector trained on 25,180 samples of human text and 35,254 samples of AI-generated text from a… Athena is a fully open-source AI detector trained on 25,180 samples of human text and 35,254 samples of AI-generated text from a variety of large language models. It started as a research project looking into whether subtle patterns occurring in AI-generated text could be used to estimate a probability that a text was AI generated. Athena was tested on six datasets, covering the topics: politics, sports, culture, lifestyle, opinion and science, with a total of 49,480 text samples ranging from about 100 words to about 400 words in length. Athena achieved an average sensitivity of 98.0% (i.e. 98.0% of human texts were correctly identified), and a recall of 98.7% on text produced by GPT (i.e. 98.7% of GPT-generated texts were correctly identified), 95.9% on Gemini, 94.6% on Deepseek-v3, and 93.7% Claude Haiku, and results ranged for Claude Sonnet, Deepseek-r1, Llama and Mistral. The results of the model evaluation are available on an interactive platform at the web portal: https://birefringent-ai.com/athena, which also contains a short course on large language models and the mechanisms behind AI text generation. For a deep dive into Athena's creation and testing, take a look at my articles on medium.com: https://medium.com/@tomrodolfolee/building-an-ai-detector-from-scratch-part-i-db72bc2bdadb. The full source, train and test data are available at https://github.com/tommyliphysics/athena-source