Can Businesses Trust AI? Security Risks and Emerging Guardrails
Summary
The article examines AI risks including hallucinations, bias, sycophancy and model collapse, and describes emerging defenses such as data provenance, lifecycle guardrails, red teaming and prompt-injection controls.
Key points
- The article says LLM responses are probabilistic and can reflect false or biased training data, making errors and hallucinations difficult to identify.
- Ilia Shumailov describes model collapse as a risk when models are trained on data produced by earlier models, potentially compounding errors.
- The article identifies cybersecurity, operational and reputational risks when businesses deploy AI without adequate safeguards.
- Kamiwaza AI's Krti Tallam advocates tracking data provenance and building guardrails into the AI lifecycle rather than relying on policy alone.
- DeepKeep describes external controls including automated red teaming, prompt-injection protections, PII leakage prevention, and drift and bias detection.
- AI Sequrity says its Sequrity Control converts inputs to text to neutralize hidden prompts; the article does not independently verify the company's claim that it eliminates indirect prompt engineering.
Article Details
- Topic
- Risks of trusting current AI systems and the development of security guardrails to mitigate them
People
Dr. Cheng ZhangAI Sequrity co-founder and researcher involved in AI security frameworks and agentic workflows.Dr. Yiren ZhaoAI Sequrity co-founder who works on secure agentic flows; also leads the DeepWok Lab.Ilia ShumailovAI scientist who coined the term “model collapse” and explained its causes and effects; he also founded AI Sequrity.Jim CardenRetired FBI detective and cybercrime investigator who wrote and distributed a public safety warning about AI sycophancy, drawing on his own experience using an AI chatbot.Krti TallamAI security researcher and Kamiwaza AI technical staff member who discussed data provenance and engineering guardrails into the AI software lifecycle.Raz LapidDeepKeep chief scientist who discussed AI model risks and the need for stricter control and monitoring of Grok.Yossi AltevetDeepKeep co-founder and CTO who discussed the firm’s approach to mitigating AI risks.
Vendors
AI SequrityAI Sequrity is a firm founded by Ilia Shumailov (formerly a research scientist at Google Deepmind and author of the model collapse paper discussed earlier); Dr.DeepKeepDeepKeep, a firm founded in 2021, similarly believes in an engineering solution to the problems of AI – but it does so from outside the AI.Kamiwaza AIKrti Tallam, senior member of the technical staff at Kamiwaza AI.
Products
ChatGPTcites the recently reported incident of an acting director of CISA uploading sensitive data to a public chatGPT – which retained the sensitive data for ongoing model training.GrokAsked to comment on Defense Secretary Hegseth’s January 2026 announcement that the Pentagon would be using Grok, Lapid said, “Assuming this is being used locally and not as part of X, Grok has an advantage as it’s lessSequrity ControlIts Sequrity Control solution eliminates hidden prompts by converting all inputs to text, so a prompt hidden within another document becomes simple text.