AI Tool Risk Assessment: A Step-by-Step Guide
Evaluate any AI tool in about 30 minutes: data handling, account control, contract terms, output quality and cost, scored on a simple sheet you can reuse.
This guide gives you a repeatable way to assess a single AI tool before you approve it for company use, or to review one your team already relies on. It takes about 30 minutes per tool once you have the vendor's documentation at hand.
The method is a scoring sheet with five areas. Each area has a few questions; each question gets a score. At the end, the total and the lowest individual scores tell you whether to approve the tool, approve it with conditions, or say no.
Before you start
Collect these for the tool you are assessing:
- the plan or tier you would actually use (free, team, business, enterprise),
- the terms of service and privacy policy for that plan,
- the vendor's security or trust page, if they have one,
- a short description of how your team would use the tool and what data would go into it.
The last point matters most. The same tool can be low risk for drafting social media posts and high risk for summarizing customer contracts. Always assess a tool for a specific use, not in general.
How to score
Score every question on a simple scale:
| Score | Meaning |
|---|---|
| 2 | Clearly good. Documented and verified. |
| 1 | Acceptable, or good but only partly documented. |
| 0 | Unclear. You could not find the answer. |
| -2 | Clearly a problem. |
Note that unclear scores zero, not one. If a vendor does not tell you how they handle your data, you should not assume the best.
Area 1: Data handling
- Training: Does the vendor use your inputs to train or improve its models? Can you turn this off for your plan, and is it off?
- Retention: How long are inputs and outputs stored? Can you delete them?
- Location: Where is data stored and processed? Does that match your obligations to customers?
- Access: Who at the vendor can see your data, and under what circumstances?
- Fit for your data: Given what you plan to put into the tool, are the answers above good enough?
Area 2: Account control
- Business accounts: Can the company own the accounts, rather than individual employees?
- Single sign-on: Can users log in through your company identity provider?
- Admin visibility: Can an administrator see users, usage and connected integrations?
- Offboarding: When someone leaves, can you remove access and keep the company's data?
Area 3: Contract and terms
- Confidentiality: Do the terms treat your inputs as confidential?
- Data processing agreement: Is one available if you will process personal data?
- Output ownership: Do you own what the tool produces?
- Changes to terms: Will you be notified before terms change, or can they change silently?
- Customer obligations: Does using this tool conflict with anything you promised your own customers?
Area 4: Output quality
- Accuracy for your use: Test it on five to ten realistic examples from your own work. How often is the output correct without editing?
- Failure mode: When it is wrong, is it obviously wrong, or wrong in a way that looks right?
- Review step: Is there a clear point where a person checks the output before it is used?
- Accountability: Is it clear who is responsible when the output contains an error?
The test with your own examples is the most valuable part of this area. Vendor demos are chosen to look good; your own data is not.
Area 5: Cost and dependency
- Predictable cost: Is pricing per seat, per use, or both? Can you estimate the monthly cost at realistic usage?
- Overlap: Does another approved tool already do the same job?
- Exit: If you stopped using the tool, could you export your data and work, and continue elsewhere?
- Vendor stability: Is the vendor established enough that the tool is likely to exist in two years?
Reading the result
Add up the scores in each area and overall. Then look at two things:
1. Any -2 in Area 1 or Area 3 is a stop for the planned use, regardless of the total. A great tool that trains on your customer contracts is still not acceptable for customer contracts. You can still approve it for a different, lower-risk use and reassess.
2. The overall picture decides the rest:
| Result | Decision |
|---|---|
| No -2 scores, few zeros | Approve for the assessed use |
| No -2 scores, several zeros | Approve with conditions: get the missing answers from the vendor, restrict which data may be used, or start with a small pilot group |
| One or more -2 scores | Do not approve for this use, or only after the problem is resolved |
Write the decision down along with the date, the plan assessed and the use it covers. Tools and their terms change often, so set a date to reassess, typically every six to twelve months, or immediately when the vendor changes its terms or you change how you use the tool.
Keep a register
Once you have assessed a few tools, keep the results in a single list: tool, approved uses, data allowed, owner, cost, review date. That register is the simplest possible AI policy, and it answers the question every company eventually gets from a customer or auditor: which AI tools do you use, and what do you put into them?
For the broader rollout picture, including how to find tools your team is already using, see Before You Roll Out AI Tools. To check the quality of output from a tool already in use, apply the method from How to Run an IT Spot Check.