As AI adoption grows, so do risks of compromised or backdoored models. A security audit tool could automatically scan models for signs of tampering, unauthorized modifications, or suspicious behavior patterns.
This would be a SaaS product targeting AI teams in enterprises, especially those using third-party models. The service would analyze model weights, architectures, and inference behaviors to detect anomalies.
The first version could simply check for known attack signatures in popular model formats. Over time, it could incorporate more sophisticated behavioral analysis.
The biggest risk is false positives - incorrectly flagging legitimate models as compromised could damage customer trust and adoption.