How to Conduct AI Vendor Evaluation¶

Ready to Use

Purpose¶

This guide provides a structured approach to evaluating AI vendors and solutions for government procurement, ensuring compliance, value for money, and fit for purpose.

When to Use This Guide¶

Use this evaluation process when: - Procuring AI/ML software or platforms - Engaging AI solution providers - Evaluating AI-as-a-Service offerings - Assessing AI consulting partners - Comparing build vs buy options

Evaluation Framework Overview¶

flowchart TB
    EVAL["<strong>AI VENDOR EVALUATION</strong>"] --> TECH & BIZ & COMP & VIAB

    subgraph TECH["<strong>TECHNICAL<br/>CAPABILITY</strong>"]
        T1[Accuracy]
        T2[Scalability]
        T3[Integration]
        T4[MLOps]
    end

    subgraph BIZ["<strong>BUSINESS<br/>VALUE</strong>"]
        B1[ROI]
        B2[TCO]
        B3[Flexibility]
        B4[Time value]
    end

    subgraph COMP["<strong>COMPLIANCE<br/>& SECURITY</strong>"]
        C1[Privacy]
        C2[Security]
        C3[Ethics]
        C4[Compliance]
    end

    subgraph VIAB["<strong>VENDOR<br/>VIABILITY</strong>"]
        V1[Stability]
        V2[References]
        V3[Support]
        V4[Roadmap]
    end

    style EVAL fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style TECH fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style BIZ fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style COMP fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style VIAB fill:#e0f2f1,stroke:#00796b,stroke-width:2px

Step 1: Define Requirements¶

Requirements Gathering Template¶

## AI Solution Requirements

### Business Requirements
| # | Requirement | Priority | Acceptance Criteria |
|---|-------------|----------|---------------------|
| B1 | | Must/Should/Could | |
| B2 | | | |

### Functional Requirements
| # | Requirement | Priority | Acceptance Criteria |
|---|-------------|----------|---------------------|
| F1 | | Must/Should/Could | |
| F2 | | | |

### Non-Functional Requirements
| # | Requirement | Target | Minimum |
|---|-------------|--------|---------|
| Performance | Response time | <500ms | <2s |
| Availability | Uptime | 99.9% | 99.5% |
| Scalability | Concurrent users | 1000 | 500 |
| Accuracy | Model accuracy | 95% | 90% |

### Compliance Requirements
| # | Requirement | Mandatory |
|---|-------------|-----------|
| Privacy Act compliance | Yes |
| PSPF alignment | Yes |
| Data sovereignty (Australian) | Yes |
| Accessibility (WCAG 2.1) | Yes |

### Integration Requirements
| System | Integration Type | Criticality |
|--------|-----------------|-------------|
| | API/File/Event | High/Med/Low |

Step 2: Vendor Shortlisting¶

Initial Screening Criteria¶

Market Scan Approach¶

1. Request for Information (RFI): Broad market scan
2. Industry research: Analyst reports, peer reviews
3. Existing panel suppliers: Check whole-of-government panels
4. Peer agencies: What solutions are other agencies using?

Step 3: Evaluation Criteria¶

Scoring Matrix¶

Scoring Scale¶

Step 4: Technical Evaluation¶

Capability Assessment¶

## Technical Capability Assessment

### Model Performance
| Test | Vendor A | Vendor B | Vendor C |
|------|----------|----------|----------|
| Accuracy on test set | | | |
| Precision | | | |
| Recall | | | |
| F1 Score | | | |
| Performance on edge cases | | | |

### Scalability Testing
| Test | Target | Vendor A | Vendor B | Vendor C |
|------|--------|----------|----------|----------|
| Max concurrent requests | 500 | | | |
| Response time under load | <1s | | | |
| Auto-scaling capability | Yes | | | |
| Peak capacity | 2x normal | | | |

### Integration Assessment
| Aspect | Vendor A | Vendor B | Vendor C |
|--------|----------|----------|----------|
| API documentation quality | | | |
| Authentication options | | | |
| Existing connectors | | | |
| Webhook/event support | | | |
| Data format flexibility | | | |

Proof of Concept (PoC)¶

When to conduct a PoC: - High-value procurement (>$500k) - Novel technology - Critical system integration - Top 2-3 shortlisted vendors

PoC Evaluation Checklist: - [ ] Use realistic data (anonymized if needed) - [ ] Test actual use cases - [ ] Evaluate with real users - [ ] Measure against defined metrics - [ ] Document implementation effort - [ ] Assess vendor support quality

Step 5: Commercial Evaluation¶

Total Cost of Ownership¶

## 5-Year TCO Comparison

### One-Time Costs
| Cost Item | Vendor A | Vendor B | Vendor C |
|-----------|----------|----------|----------|
| Licensing (initial) | | | |
| Implementation services | | | |
| Integration development | | | |
| Training | | | |
| Data migration | | | |
| **Subtotal** | | | |

### Annual Recurring Costs
| Cost Item | Vendor A | Vendor B | Vendor C |
|-----------|----------|----------|----------|
| Subscription/license | | | |
| Support & maintenance | | | |
| Infrastructure | | | |
| Internal operations | | | |
| **Subtotal (Annual)** | | | |

### 5-Year TCO Summary
| | Vendor A | Vendor B | Vendor C |
|---|----------|----------|----------|
| One-time costs | | | |
| Year 1 recurring | | | |
| Year 2-5 recurring | | | |
| **Total 5-Year TCO** | | | |

Commercial Terms Checklist¶

• Pricing model clear and predictable
• Volume discounts available
• Exit terms reasonable (data export, transition)
• Price protection clauses
• Renewal terms defined
• Liability and indemnity acceptable
• IP ownership clear

Step 6: Compliance & Security Evaluation¶

Privacy Assessment¶

Security Assessment¶

AI Ethics Assessment¶

Step 7: Vendor Due Diligence¶

Financial Assessment¶

Reference Checks¶

Questions for references: 1. How long have you used this solution? 2. What problem did it solve? 3. How was the implementation experience? 4. How responsive is vendor support? 5. Have you experienced any issues? 6. Would you recommend them? Why/why not? 7. What would you do differently?

Reference Check Template:

## Reference Check: [Vendor]

Reference: [Organization]
Contact: [Name, Role]
Date: [Date]

### Implementation
- Duration: [X months]
- On time/budget: [Yes/No]
- Challenges: [Description]

### Operations
- Uptime experience: [%]
- Support quality: [1-5]
- Issue resolution: [Fast/Slow]

### Overall
- Satisfaction: [1-5]
- Recommendation: [Yes/No]
- Key advice: [Quote]

Step 8: Final Evaluation¶

Evaluation Summary Template¶

## Vendor Evaluation Summary

### Scores

| Category (Weight) | Vendor A | Vendor B | Vendor C |
|-------------------|----------|----------|----------|
| Technical (35%) | | | |
| Business Value (25%) | | | |
| Compliance (25%) | | | |
| Vendor Viability (15%) | | | |
| **Weighted Total** | | | |
| **Rank** | | | |

### Strengths & Weaknesses

#### Vendor A
Strengths:
-
Weaknesses:
-

#### Vendor B
Strengths:
-
Weaknesses:
-

#### Vendor C
Strengths:
-
Weaknesses:
-

### Recommendation
[Clear recommendation with justification]

### Risks and Mitigations
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| | | | |

### Negotiation Points
1. [Key point to negotiate]
2. [Key point to negotiate]

Quick Reference: Evaluation Checklist¶

Pre-Evaluation¶

• Requirements documented and approved
• Evaluation criteria weighted and agreed
• Evaluation panel assembled
• Conflict of interest declarations signed
• Probity plan in place

During Evaluation¶

• All vendors assessed against same criteria
• Scores independently assigned then consolidated
• Technical testing completed
• Reference checks completed
• Commercial terms reviewed

Post-Evaluation¶

• Evaluation report drafted
• Recommendation documented
• Value for money assessment completed
• Risk assessment completed
• Approval obtained

Common Pitfalls to Avoid¶

Resources¶

Government Procurement Guidance¶

• Digital Sourcing Framework
• Commonwealth Procurement Rules
• ICT Procurement Framework

Evaluation Templates¶

• ROI Calculator
• TCO Calculator
• Security Assessment