LLM Cognitive Bias Auditing

knowledge skill

Detects and classifies cognitive biases in LLM-generated text by analyzing reasoning patterns, logical fallacies, and heuristic-driven errors. Produces a structured audit report listing bias types, evidence, and severity scores.

Worked examples

• {"input": {"text": "The AI said 80% of users prefer Product X because it's the most popular. However, the survey only included 50 people from one city.", "confidence_threshold": 0.8}, "output": {"audit_report": {"bias_types": ["survivorship_bias", "small_sample_size"], "evidence": ["Claim based on 50-person survey", "No comparison group provided"], "severity": 0.85}, "summary_stats": {"total_biases": 2, "avg_severity": 0.85}, "flagged_passages": ["80% of users prefer Product X"]}}
• {"input": {"text": "Stock prices always rise in January because of the January Effect. This year, the S&P 500 gained 3% in January.", "bias_taxonomy": "custom_taxonomy.json"}, "output": {"audit_report": {"bias_types": ["confirmation_bias", "hasty_generalization"], "evidence": ["Single data point cited", "Ignores contrary historical data"], "severity": 0.92}, "summary_stats": {"total_biases": 2, "avg_severity": 0.92}, "flagged_passages": ["Stock prices always rise in January"]}}

Input

• text: string
• bias_taxonomy: string (optional, path to custom taxonomy file)
• confidence_threshold: float (optional, default 0.7)

Output

• audit_report: object
• summary_stats: object
• flagged_passages: array

Details

• Skill type: knowledge skill
• Safety level: safe_public_research
• Version: 1.0.0