Case Study: AI-Assisted Grants Assessment¶

Case Study

Executive Summary¶

A federal grants administration agency implemented an AI-assisted assessment system to support human assessors in evaluating grant applications. The system improved assessment consistency by 40%, reduced processing time by 50%, and enabled faster funding decisions while maintaining full human oversight.

The Challenge¶

Situation¶

• 25,000+ grant applications annually across 15 programs
• $500M in annual grants administered
• 120 assessors across multiple locations
• 8-12 week average assessment time
• Inconsistent assessment quality across assessors

Problems¶

• Assessment variability between assessors
• Long processing times delayed funding
• Assessors spent excessive time on administrative tasks
• Difficulty identifying high-potential applications quickly
• Limited capacity for thorough due diligence

Business Impact¶

• Applicant satisfaction declining
• Ministerial pressure on processing times
• Concerns about assessment fairness
• Staff overwhelmed during peak periods
• Audit findings on consistency issues

The Solution¶

AI Approach¶

Model Type: Multi-component system (NLP + scoring + anomaly detection) Architecture: Transformer-based text analysis + rule-based scoring Integration: Grants management system

System Design¶

flowchart LR
    subgraph APP["<strong>Application</strong>"]
        A1[Documents]
        A2[Budget]
        A3[Attachments]
        A4[History]
    end

    subgraph DOC["<strong>Document Processing</strong>"]
        D1[Extract Sections]
        D2[Parse Numbers]
        D3[Validate Complete]
    end

    subgraph AI["<strong>AI Analysis</strong>"]
        AI1[Criteria Matching]
        AI2[Risk Flags]
        AI3[Similar Apps]
    end

    subgraph OUT["<strong>Assessor Interface</strong>"]
        O1[Assessment Workbench]
        O2[AI Insights]
        O3[Suggested Questions]
    end

    APP --> DOC --> AI --> OUT
    OUT --> DEC[Human Decision<br/>Required]

    style APP fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style DOC fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style AI fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style OUT fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style DEC fill:#ffcc80,stroke:#ef6c00,stroke-width:2px

AI Components¶

1. Document Processing - OCR for scanned documents - Section extraction and parsing - Budget parsing and validation - Completeness checking

2. Criteria Matching - NLP analysis of application against criteria - Evidence extraction for each criterion - Strength/weakness identification - Gap detection

3. Risk Flagging - Financial viability indicators - Applicant history analysis - Budget reasonableness checks - Duplication detection

4. Similar Application Matching - Find similar historical applications - Show outcomes of similar applications - Identify potential duplicates

Key Design Principles¶

Implementation¶

Timeline¶

Team¶

Model Training¶

Data Sources: - 75,000 historical applications (5 years) - Assessment reports and decisions - Criteria guidelines for each program - Financial reports of funded applicants

Labeling: - Successful/unsuccessful decisions - Assessor scores by criterion - Risk flags from historical reviews - Due diligence outcomes

Validation: - Expert assessor review of AI outputs - A/B testing with assessor panels - Accuracy testing against historical decisions

Results¶

Assessment Quality¶

Efficiency Gains¶

Quality Indicators¶

Fairness Outcomes¶

Challenges and Lessons Learned¶

Challenge 1: Assessor Concerns¶

Issue: Assessors worried AI would replace them Solution: - Clear communication: AI assists, humans decide - Assessors can override/ignore AI suggestions - AI handles admin, assessors focus on judgment Lesson: Position AI as tool, not replacement

Challenge 2: Varied Program Criteria¶

Issue: Different programs had different criteria styles Solution: - Program-specific criteria models - Common framework with program adaptations - Easy update mechanism for criteria changes Lesson: Build flexible architecture for variety

Challenge 3: Historical Bias¶

Issue: Historical decisions might embed bias Solution: - Fairness testing across applicant types - Removed biased features (e.g., organization name) - Human decision required for all outcomes Lesson: AI surfaces insights, doesn't make decisions

Challenge 4: Explainability for Applicants¶

Issue: Applicants wanted to understand assessments Solution: - AI evidence used in feedback letters - Clear mapping to criteria - Improvement suggestions generated Lesson: AI can improve applicant communication

Challenge 5: Gaming Prevention¶

Issue: Applicants might optimize for AI rather than quality Solution: - AI criteria matching not visible to applicants - Human judgment required for approval - Regular model updates Lesson: Keep some AI logic confidential

Governance and Compliance¶

Governance Structure¶

• Executive sponsor: Branch Head, Grants Administration
• Program governance: Program managers committee
• Ethics oversight: Ethics committee review
• Risk tier: Tier 3 (High) - Affects funding decisions

Human Oversight Requirements¶

Compliance Measures¶

• Grants CPGs (Commonwealth Grants Rules and Guidelines)
• Public Governance Framework
• Anti-discrimination legislation
• Privacy Act (applicant data)
• Administrative law (fair process)

Transparency¶

To Applicants: - Notification that AI assists assessment - Human makes all decisions - Right to appeal - Feedback includes evidence-based reasoning

To Assessors: - Full visibility of AI reasoning - Ability to override any AI output - Training on AI capabilities and limitations

Technical Details¶

AI Components¶

Document Processing: - OCR: Tesseract + Azure Form Recognizer - Section extraction: Custom NER model - Budget parsing: Rule-based + ML validation

Criteria Matching: - Base model: DistilBERT fine-tuned - Evidence extraction: Named entity recognition - Semantic similarity: Sentence transformers - Coverage analysis: Rule-based

Risk Flagging: - Financial risk: Gradient Boosted Trees - History analysis: Database queries + rules - Anomaly detection: Isolation Forest

Similar Applications: - Embedding: Sentence transformers - Search: Approximate nearest neighbors (FAISS) - Threshold: Human-tuned similarity cutoff

Infrastructure¶

• Training: AWS SageMaker
• Serving: Agency cloud (AWS)
• Integration: API to grants management system
• Storage: Application data in existing system
• Monitoring: Custom dashboard

Performance¶

• Document processing: <2 minutes per application
• Analysis: <30 seconds per application
• Availability: 99.5%
• Model refresh: Quarterly

Recommendations for Similar Projects¶

Do¶

• Design for human decision-making throughout
• Involve assessors in design and testing
• Build explainability from the start
• Test for fairness across applicant types
• Maintain complete audit trail
• Plan for criteria changes

Don't¶

• Let AI make funding decisions
• Ignore assessor concerns
• Rely solely on historical data patterns
• Reveal AI details to applicants (gaming risk)
• Skip administrative law review
• Assume one model fits all programs

Cost-Benefit Summary¶

Costs (First Year)¶

Ongoing Costs (Annual)¶

Benefits (Annual)¶

ROI: 124% | Payback: 7 months¶

Contact¶

For more information about this case study, contact the AI Toolkit team.

Related documents: AI Governance Framework | How to Explainability