Hybrid Knowledge Elicitation

The Framework

A pipeline for extracting and structuring expert knowledge that assigns tasks based on what AI and humans each do well:

AI handles: Data collection (interviews), initial structuring, consistency enforcement Humans handle: Semantic interpretation, hierarchical modeling, validation, tacit knowledge encoding

Why This Matters

Traditional knowledge elicitation is expensive, expert time, knowledge engineer time, coordination overhead. Fully automated approaches are faster but produce shallower, less accurate structures. The hybrid approach captures most of the efficiency gains while preserving quality.

The underlying insight: LLMs operate in the explicit knowledge domain (Combination in SECI terms), while ontology construction requires tacit-to-explicit conversion (Externalization). These are different cognitive operations that map to different agents in the pipeline.

The Components

Collection Phase (AI-led):

• 3.5x faster than human interviews
• More structured, consistent outputs
• Stays on topic without prompting
• Captures what experts can articulate

Structuring Phase (Human-led):

• Better hierarchical alignment
• Captures more instances (not just types)
• Preserves domain nuance
• Validates against tacit understanding

Validation Phase:

• Check for hallucinated content (correct but untraceable)
• Competency questions test coverage
• Compare against domain expertise

When to Use

• Knowledge capture projects with time/budget constraints
• Domains with accessible documentation (AI has training data)
• Tasks requiring formal representation (ontologies, knowledge graphs)
• Situations where you need both speed and accuracy

Limitations

• AI may systematically omit concepts due to content moderation
• Open-ended collection misses topics not explicitly raised
• Requires human expertise for the high-value structuring work
• Hallucination risk means outputs need validation
• Works best when AI has relevant training data

Implications for Practice

The framework suggests a shift in how to allocate expensive human expertise: focus humans on interpretation and structuring rather than collection. This doesn’t eliminate the need for experts, it redirects them to higher-leverage tasks.

Related: 06-molecule—seci-framework, 06-molecule—knowledge-graph-construction