Skip to content

Inherited an AI System

You've just taken responsibility for an AI system you didn't build. This journey helps you understand what you've got and assess its state.

Step 1
Discover
Step 2
Document
Step 3
Assess
Step 4
Decide

Step 1: Discover What You Have

Before you can manage it, you need to understand it.

Technical Discovery

Find answers to:

Question Where to Look
What does the model do? Documentation, code comments
What data was it trained on? Training scripts, data pipelines
When was it last updated? Git history, deployment logs
Who built it? Code ownership, commit history
What are its dependencies? Requirements files, infrastructure
How is it deployed? CI/CD config, infrastructure code

Find answers to:

Question Where to Look
Which model/API is used? Code, config files, vendor contracts
What prompts are used? Prompt templates, system messages
What guardrails exist? Content filters, validation logic
What data is sent to the API? Integration code, data flows
What are the costs? Billing records, usage dashboards
What's the vendor contract? Procurement records

Operational Discovery

  • Who uses this system?
  • What decisions does it influence?
  • What happens if it's wrong?
  • What happens if it goes down?
  • Who currently supports it?

Step 2: Document What You Find

Create the documentation that should have existed.

Essential Documentation

Template: Model Cards Guide

Create or update:

  1. System overview - What it does, why it exists
  2. Technical architecture - Components, data flows, dependencies
  3. Operational runbook - How to monitor, restart, troubleshoot
  4. Known limitations - What it can't do, known issues
  5. Contact list - Who knows what about this system

Questions for Previous Owners

If you can reach them:

  • What were the original requirements?
  • What trade-offs were made?
  • What's the scariest part of this system?
  • What would you do differently?
  • What's the technical debt?
  • Are there any "don't touch" areas?

Step 3: Assess the Risk

Now that you understand it, evaluate its state.

Health Check

Run these assessments:

  • Performance check - Is accuracy still acceptable?
  • Bias check - Check for Bias journey
  • Data drift check - Has input data distribution changed?
  • Dependency audit - Are libraries up to date? Vulnerabilities?
  • Security review - Security Assessment

Run these assessments:

  • Output quality review - Sample recent outputs
  • Guardrail test - Do safety measures work?
  • Cost analysis - Is spend reasonable?
  • Vendor review - Is contract current? Terms acceptable?
  • Data handling audit - What's being sent to vendor?

Risk Assessment

Template: Risk Register

Identify risks in these categories:

Category Questions
Operational What if it fails? Who's impacted?
Technical What's the technical debt? Bus factor?
Compliance Does it meet current requirements?
Reputational What if it makes a mistake publicly?
Financial What does it cost? What if costs spike?

Red Flags

Warning signs

  • No documentation exists
  • Original team is gone
  • No one understands how it works
  • It hasn't been updated in years
  • "Don't touch it, it just works"
  • No monitoring or alerting
  • Unknown data sources
  • Unclear ownership

Step 4: Decide What to Do

Based on your assessment, choose a path.

Option A: Maintain As-Is

When appropriate: - System is stable and low-risk - Documentation is adequate - No major compliance gaps - Resources are limited

Actions: - Document what you learned - Set up monitoring if missing - Establish regular review cadence - Create succession plan

Option B: Remediate and Improve

When appropriate: - Fixable issues identified - System provides value - Resources available for improvement

Actions: - Prioritise fixes by risk - Create improvement roadmap - Allocate ongoing maintenance capacity - Consider Improve a Struggling Model journey

Option C: Replace or Rebuild

When appropriate: - Fundamental issues exist - Technical debt too high - Better alternatives available - Compliance requirements can't be met

Actions: - Build business case for replacement - Plan migration strategy - Consider Assess a New Opportunity for replacement

Option D: Shut Down

When appropriate: - Risk outweighs benefit - No longer serves business need - Can't be made compliant - Better to stop than continue

Actions: - Follow Shut Down an AI System journey