Case Study: Intelligent Document Classification¶

Case Study

Executive Summary¶

A service delivery agency implemented an AI-powered document classification system to automatically categorize incoming correspondence, reducing manual triage time by 65% and improving response times to citizens by 40%.

The Challenge¶

Situation¶

• 500,000+ pieces of correspondence received annually
• Mix of emails, letters, forms, and scanned documents
• 15 different categories requiring specialized handling
• Average 3-day delay in initial triage
• High workload on frontline staff

Problems¶

• Manual classification was time-consuming and inconsistent
• Urgent matters were sometimes delayed
• Staff frustration from repetitive categorization work
• Citizen complaints about response times
• Difficulty tracking correspondence patterns

Business Impact¶

• 12 FTE dedicated to triage function
• Citizen satisfaction scores declining
• Ministerial complaints about delays
• Staff turnover in correspondence team

The Solution¶

AI Approach¶

Model Type: Multi-class text classification Architecture: Fine-tuned transformer (DistilBERT) Integration: Email gateway and document management system

System Design¶

flowchart LR
    subgraph IN["<strong>Input</strong>"]
        I1[Email/Letter]
        I2[Scanned Document]
    end

    subgraph OCR["<strong>OCR/Text Extraction</strong>"]
        O1[Clean Text]
    end

    subgraph CLASS["<strong>AI Classification</strong>"]
        C1[15 Category Classifier]
        C2[Confidence Score]
    end

    subgraph ROUTE["<strong>Routing</strong>"]
        R1[Queues by category]
    end

    IN --> OCR --> CLASS --> ROUTE
    CLASS --> HIGH[✓ High confidence<br/>Auto-route]
    CLASS --> LOW[? Low confidence<br/>Human review]

    style IN fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style OCR fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style CLASS fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style ROUTE fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style HIGH fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
    style LOW fill:#ffcc80,stroke:#ef6c00,stroke-width:2px

Categories Classified¶

1. New claims
2. Claim updates
3. Change of circumstances
4. Payment enquiries
5. Complaints
6. Appeals
7. Document submissions
8. General enquiries
9. Feedback (positive)
10. Urgent/hardship
11. Fraud reports
12. Death notifications
13. Address changes
14. Representative requests
15. Other/Unclear

Key Design Decisions¶

Implementation¶

Timeline¶

Team¶

Data Preparation¶

Training Data: - 100,000 labeled historical documents - Manual quality review of 10,000 samples - Inter-rater reliability testing (92% agreement) - Augmentation for underrepresented categories

Challenges: - Historical labels were inconsistent - Some categories had limited samples - OCR quality varied for scanned documents

Solutions: - Re-labeled 20,000 documents with new taxonomy - Synthetic oversampling for rare categories - OCR preprocessing pipeline with quality scoring

Results¶

Performance Metrics¶

Business Impact¶

Accuracy by Category¶

Challenges and Lessons Learned¶

Challenge 1: Data Quality¶

Issue: Historical labels were inconsistent across teams Solution: Invested 8 weeks in data cleaning and relabeling Lesson: Budget significant time for data preparation (30% of project)

Challenge 2: Staff Concerns¶

Issue: Staff worried about job security Solution: Clear communication that AI handles triage, not decisions Lesson: Involve staff early, show how AI helps not replaces

Challenge 3: Edge Cases¶

Issue: Multi-topic correspondence confused the model Solution: Added multi-label capability and human review routing Lesson: Design for edge cases from the start

Challenge 4: Model Drift¶

Issue: Accuracy dropped 5% after 3 months Solution: Implemented monthly retraining with recent data Lesson: Plan for ongoing model maintenance

Challenge 5: Privacy Concerns¶

Issue: Questions about AI reading citizen correspondence Solution: Clear privacy documentation, data minimization Lesson: Address privacy proactively with stakeholders

Governance and Compliance¶

Governance Structure¶

• Executive sponsor: Director of Service Delivery
• Governance board oversight: Quarterly review
• Ethics review: Completed prior to deployment
• Risk tier: Tier 2 (Medium)

Compliance Measures¶

• Privacy Impact Assessment: Completed
• Security assessment: Completed
• Model card: Published internally
• Audit trail: All classifications logged
• Human oversight: Staff can override any classification

Fairness Considerations¶

• Tested for geographic bias: None detected
• Tested for language patterns: Minor adjustments made
• Complaint routing accuracy equal across demographics

Technical Details¶

Model Specifications¶

• Base model: DistilBERT (66M parameters)
• Fine-tuned on: Agency-specific data
• Input: Document text (max 512 tokens)
• Output: 15 class probabilities
• Confidence calibration: Platt scaling

Infrastructure¶

• Training: AWS SageMaker
• Serving: API on agency cloud (AWS)
• Throughput: 1,000 documents/minute
• Latency: <500ms per document
• Availability: 99.9%

Integration Points¶

• Email gateway (incoming mail)
• Document management system
• Case management system
• Reporting dashboard

Recommendations for Similar Projects¶

Do¶

• Invest heavily in data quality
• Engage staff and unions early
• Design for human oversight
• Plan for ongoing retraining
• Start with pilot before full rollout
• Measure and communicate benefits

Don't¶

• Underestimate data preparation time
• Deploy without monitoring
• Ignore edge cases
• Skip change management
• Assume one-time model training is sufficient

Cost-Benefit Summary¶

Costs (First Year)¶

Ongoing Costs (Annual)¶

Benefits (Annual)¶

ROI: 147% (Year 1) | Payback: 8 months¶

Contact¶

For more information about this case study, contact the AI Toolkit team.

Related documents: AI Use Case Template | ROI Calculator