o1-Preview Achieves Expert-Level Ontology Modeling

OpenAI’s o1-preview model with the Ontogenia prompting technique achieved 96-100% adequate competency question modeling in expert evaluation, matching or exceeding the quality expected from trained ontology engineers.

For comparison:

• Novice student submissions: 73-80% adequate modeling
• GPT-4 with same techniques: 88-93%
• Llama-3.1-405B: 60-86%

The gap between o1 and GPT-4 was most pronounced on complex patterns (reification, restrictions) where reasoning chains matter most. The gap between commercial and open-source models (o1 vs Llama) was significant enough that the authors concluded “it may still not be possible to create an ontology engineering co-pilot using open models.”

Related: [None yet]