CD Consulting R&D
KB-RISK · Entry 4 · Published 2026-09-15 · Version 1
Twenty-one risks of working with AI agents: a field register, and the case of the pre-filled prompt
After two months of running several AI coding sessions under a written governance, a one-person squad has logged twenty-one distinct risks. Most are not about bad code: they are about the memory of decisions. One of them, the next prompt suggested by the tool itself, deserves a close look.
If you let an AI assistant act on your instructions, know that the next instruction in your input box may not be yours.
Contents
- 1. Where this comes from
- 2. The register at a glance
- 3. What the register teaches
- 4. R-009 in detail: the pre-filled prompt
- 5. A research track: how other assistants do it
- 6. Takeaways for practitioners
- 7. Sources
1. Where this comes from
Pour lire cette note en français : version française.
The setting is deliberately small: one human decision-maker (called the owner below), and a handful of AI sessions running in a desktop coding assistant (Claude Code, with Claude models of several sizes: Fable, Opus, Sonnet, Haiku). The sessions write code, documentation, analyses and governance records. What makes the setting unusual is the discipline around it:
- A charter of twelve rules binds every session: no engaging action without an explicit mandate; one file has one writer; every timestamp is read from an instrument, never estimated; a decision counts only when the owner types an explicit gate formula (typically
go <verb> <object>) in the session's own window; ambiguous requests trigger a question, one at a time, never a silent choice. - Written registers are the memory. A backlog, a change log, a risk log and an audit register are the only authoritative state. A session's window is disposable; the registers are not. Any commit that records a decision carries a fixed prefix so that the decision trail can be audited mechanically from the version history.
- A supervisor session (long-lived, itself bounded by rules on context compaction) drafts mandates, audits executions against transcripts, and keeps the registers. It has no power to decide: it measures, frames, and writes what the owner has gated.
The risk log was created on 14 July 2026, after a governance incident, and had twenty-one entries on 15 September 2026. Each entry carries a gravity (Low / Medium / High), a probability with the active controls in place, the documented occurrences, the preventive controls, the detection controls and a status. A risk is closed only on the owner's gate.
2. The register at a glance
The twenty-one entries fall into seven families. The one-line wording below is a paraphrase; gravity and probability are those of the register on 15 September 2026.
| Family | ID | Risk (one line) | Gravity / Prob. | Main control | Status |
|---|---|---|---|---|---|
| A. Integrity of decisions | R-001 | Recording a decision that was never pronounced (the agent reads a direction as a gate and writes "decided") | High / Medium | Rule 9: a gate is an explicit formula; commit prefix audited from history | Active |
| R-009 | Pseudo-gate through a pre-filled prompt suggested by the interface and sent by mistake | High / Low-Medium | Echo-confirmation of any abnormal gate; formulas carry their own bounds | Active, 5 occurrences | |
| R-010 | Laundering a gate through relay: a session treats a reported authority as a gate | High / Low | A gate is never second-hand; reported gates are echo-confirmed | Active | |
| R-013 | Autonomous arbitration of an ambiguous request (the agent picks a costly or deviant path instead of asking) | High / Medium | Rule 12: elicitation, one question at a time; bounds before cost | Partial | |
| B. Truth and memory | R-003 | Temporal hallucination: an estimated date or time contaminates journals | High / Low | Instrumental timestamps only | Controlled |
| R-004 | False premise leading to unrequested initiative | High / Low-Medium | Explicit mandate; situated diagnosis | Active | |
| R-015 | Cognitive drift of a long-lived session across context compactions | Medium / Certain over time | Cap of two compactions per cycle; registers-first doctrine; restitution of known / inferred / lost at each resumption | Open, observed 51 days | |
| R-020 | A copied derivative silently diverges from its source | Medium / High | Derivatives are never masters; scripted drift detectors | Open | |
| C. Concurrency and lifecycle | R-002 | Concurrent writes by several sessions on a shared artefact (loss, resurrection, mis-attribution) | Medium / Medium | One file, one writer; path-scoped commits; diff-size check before commit | Reinforced |
| R-011 | "Zombie" window surviving its mandate, with live write permissions | High / Medium | Graceful exit template: declared relinquishment, permissions extinguished | Active | |
| D. Model and platform behaviour | R-007 | Capability variance between models: a safeguard based on one model's judgement fails on another | Medium / High | Bright-line rules usable by any model; gates in the process, not in judgement | Doctrine |
| R-012 | Silent substitution of the model during a session (provider-side safety fallback), invisible from the inside | Medium / Medium | Cross-confirmation at entry; the transcript is the only instrument of calibre | Active | |
| E. Automation and permissions | R-005 | Silent freeze of a scheduled routine waiting for a permission | Medium / Medium | Hardened allow-list; canonical commands; re-arm rule | Partial |
| R-014 | Non-persistent authorisation on scheduled routines ("always allow" dies with the session) | Medium / Medium | Copy step moved out of the app layer to a native OS task | Active on copy, partial on generation | |
| R-016 | Non-minimal permission surface (policy mixed with an auto-fed journal of approvals) | High / Medium | Staged removal, tested on real runs; hash-proved inertia of settings | Open | |
| F. Third parties and money | R-006 | Dependency on a third party outside control (registrar, CA, host) | Medium / High | Portable architecture; multi-resolver diagnostics | Ongoing |
| R-017 | Spend triggered by an unbounded inbound request | High / Medium | Spending cap at the provider console; input validation; honeypot | Open, mitigated | |
| R-018 | Service cut-off by exhaustion of a platform quota | High / Low | Paid tiers subscribed; no auto-recharge (a chosen failure mode) | Open | |
| R-021 | Exit cost of a platform dependency, installed by architecture | Medium / High | One-function adapters; register of exit costs | Open | |
| G. Instruments that stop instrumenting | R-019 | A control kept red after its expected result changed | Medium / High | Amend the control in the same gesture as the change | Closed |
| R-008 | Test artefacts in invisible asynchronous queues resurfacing days later | Medium / Medium | Generation timestamp in every produced message; [TEST] marker | Active |
3. What the register teaches
Five cross-cutting lessons come out of two months of entries.
- The first thing at risk is the record, not the files. Four of the six "High" risks (R-001, R-009, R-010, R-013) describe the same failure from different angles: a decision appears in the register that the human never made, or was made by the wrong hand. Code can be reverted; a false memory of governance propagates.
- Bright lines beat judgement. A safeguard that relies on a model's fine reading of context holds for one model and fails for another (R-007). The rules that survived are those a script could check: is the timestamp instrumental, does the commit touch only the declared paths, was the gate formula typed by the owner in this window, is the question present in the transcript.
- Declare nothing, measure everything. A session cannot see its own model substitution (R-012), cannot see what it forgot after a compaction (R-015), and will confidently estimate a time (R-003). The transcript, the version history and the clock are the instruments; self-declaration is a starting point for verification, never a proof.
- Automation multiplies the permission surface. Scheduled routines that freeze on a permission prompt (R-005, R-014) and allow-lists that grow as journals of past approvals (R-016) show the same thing: every right is justified in isolation, the sum creates the exposure.
- Money is the only damage a fix does not repair. R-017, R-018 and R-021 were all raised before any loss, on documentation reading, and each was answered with a hard bound chosen in advance (a spending cap, a refused auto-recharge, an adapter) rather than with vigilance.
4. R-009 in detail: the pre-filled prompt
4.1 The mechanism, as documented
The coding assistant used by the squad shows, after each of its answers, a greyed-out suggestion of what the user might type next. The vendor documentation (interactive mode reference, read on 15 September 2026) states it plainly:
> "After Claude responds, Claude Code can suggest your next prompt based on your conversation history, such as a follow-up step from a multi-part request or a natural continuation of your workflow."
> "Press Tab or Right arrow to place the suggestion in the prompt input, then Enter to submit. Start typing to dismiss it."
> "Claude Code generates each of these next-prompt suggestions with a background request that reuses the conversation's prompt cache, so the additional cost is minimal."
Three properties follow, two of them measured by the squad:
- The text is produced by a separate background model call fed with the conversation so far. It is not written by the session the user is talking to, and the session has no knowledge of it.
- An unsent suggestion leaves no trace in the transcript. The squad verified this on the fifth occurrence: the suggested formula appeared nowhere in the session's log until the owner quoted it in a message.
- A sent suggestion is indistinguishable from typed text. Once the user presses Enter, the session receives it as the user's own words.
The feature is on in a standard session. The documentation lists three ways to turn it off: a toggle in the /config panel, the setting promptSuggestionEnabled: false in a settings file, or the environment variable CLAUDE_CODE_ENABLE_PROMPT_SUGGESTION=false, which takes precedence and can be enforced organisation-wide through managed settings.
4.2 Why it is a governance risk rather than a usability nuisance
In this squad, a decision is a short formula typed by the owner: go push, go T2, go commit journal. The suggestion generator has read hundreds of such formulas in the conversation history. It therefore produces text in exactly that shape. Two keystrokes (Tab, Enter) turn a machine-generated guess into a human decision that the session will execute and, worse, record as a gate in the change log.
The register is explicit that execution is the lesser danger: sessions work under written mandates that bound what they may touch. The greater danger is the fictitious gate: an entry in the decision register attributing to the owner an instruction he never conceived. That is the same damage as R-001 (over-inference by the model), with the cause moved from the model to the interface.
4.3 The five occurrences
| # | Date (2026) | Suggested text (verbatim) | What made it dangerous | Outcome |
|---|---|---|---|---|
| 1 | 18 July | « go, lance aussi Sonnet 054 et 055 » (start two more sessions) | A launch order in gate form, never discussed | Spotted by the owner, not sent |
| 2 | 19 July | « Opus soumet son plan d'ensemble, go pour O4 » | A gate anticipated and aimed at the wrong window | Intercepted; R-009 created the same day |
| 3 | 26 July | « supprime la branche analyse-blade-runner » (delete a branch, after a merge) | A destructive action pre-written by the tool | Not sent; the owner asked the supervisor first |
| 4 | 27 July, 12:15 | "gate AH : compact memory now" (an unannounced test typed by the owner) | Abnormal gate: foreign language, object never discussed, triply ambiguous | The supervisor executed without echo-confirmation and was interrupted at the first tool call: the control had failed on the agent side |
| 5 | 15 September, before 06:56 | « go vidange _scratch » (empty the scratch folder) | The generator condensed a to-do item that read "empty the scratch folder, except sensitive files" and dropped the exclusion | Spotted by the owner, not sent; fourth successful interception |
4.4 What the fifth occurrence added
- Scope widening. The generator does not copy an instruction; it summarises one. In doing so it kept the verb and the object and lost the restriction. The suggested formula covered six files holding secrets that the to-do list explicitly excluded and that only the owner may handle.
- Format mimicry. In an assistant session working under an open mandate, the owner deposits tasks as short
go Xformulas. A suggestion imitates that format exactly; "abnormal" is harder to detect when every real gate is also three words long. - The agent feeds the generator. The sessions' own reports end with ready-to-copy gate formulas ("next gate:
go commit journal"). Those formulas are in the history the background call reads. The convenience offered to the human is also the training material of the suggestion.
4.5 Controls in force
- Echo-confirmation. Any gate that is abnormal in its context (anticipated, out of its window, inconsistent with the known state, not matching a formula the session itself just proposed) is reformulated and confirmed with the owner before any action and before any register entry. After occurrence 4, this rule was extended to the agent's internal operations (its own memory, its own context).
- Bounds inside the formula. Since 15 September, every gate formula an agent proposes carries its restriction in the text itself (
go empty scratch except sensitive), so that a summariser cannot drop it without changing the words. - No formula for sensitive material. Agents propose no gate at all for actions on secrets or on irreversible deletions; those are described in prose and left to the owner's wording.
- Mechanical audit of the decision trail. Every commit recording a decision carries a reserved prefix; the supervisor checks, from the version history against the transcript, that the gated event actually happened in the owner's window.
- Unannounced tests by the owner. Occurrence 4 was one; it is the only occurrence where the control failed, and the failure was on the agent's side, which is where the tests are aimed.
- The switch. Disabling the feature at the settings level is available and is a decision of the owner alone: it removes the risk at the cost of a genuine convenience, and it is enforceable across an organisation.
4.6 Residual risk
With the controls above, the register rates the probability as Low and the gravity as High. Four interceptions out of five were made by the human, not by the agent: the control that works is the owner's habit of reading the input box before pressing Enter. That is a thin line, and it is the reason the switch is listed among the controls rather than dismissed.
5. A research track: how other assistants do it
Every mainstream assistant now proposes follow-ups. What the squad wanted to know, and could only partly find on 15 September 2026, is whether the same three properties hold elsewhere: generated by a separate call from the history, invisible in the logs until sent, indistinguishable once sent. The table records what public documentation says, and what it does not.
| Product | What is documented | Where | Control for the user or admin | Open questions |
|---|---|---|---|---|
| Claude Code (Anthropic) | Next-prompt suggestion generated after each answer by a background request reusing the prompt cache, from the conversation history; Tab or Right arrow to accept, Enter to send; skipped in plan mode, after an error, near the usage limit, in teammates' sessions | Official interactive-mode reference and settings reference | /config toggle; promptSuggestionEnabled: false; environment variable with precedence; managed settings for an organisation | Whether an accepted suggestion is marked in telemetry; whether the generator sees tool results or only messages |
| ChatGPT (OpenAI) | Clickable follow-up suggestion bubbles after answers, on web and mobile, described by users since May 2025; a web toggle is reported by users and contested by others ("a placebo button"); a "Follow-up behaviour" setting is reported in the desktop app | Community forum threads; no help-centre article found describing how they are generated | Reported, unverified | Official description of the generation; reliability of the toggle; whether follow-ups can carry actions in agentic modes |
| Microsoft 365 Copilot and family | Starter prompts on an agent's welcome page, up to ten, admin-configurable and auto-generated from the agent's description (Copilot Studio); in Dynamics 365 Finance and Operations, the Copilot sidecar "generates and displays up to three questions … based on the existing conversation so far" and "avoids repeating questions you have already asked" (Responsible AI FAQ, page reached through search, to be re-read); in Dynamics 365 Contact Center, prompts refreshed every third message | Microsoft Learn (Copilot Studio, Dynamics 365); the support page "About Copilot Chat responses" says nothing about follow-ups | Admin-configurable for starter prompts; not found for post-answer follow-ups in Copilot Chat | Whether Microsoft 365 Copilot Chat itself generates post-answer follow-ups, from what, and whether a tenant can turn them off |
| Gemini (Google) | Users report suggestions offered "at the end of every prompt" that they cannot turn off (Gemini Apps community thread, 2026); Workspace administrators can switch "Ask Gemini" on or off per application | Community thread; Workspace admin help | Not found for the suggestions themselves | Whether a dedicated setting exists; how suggestions are generated; whether they are logged |
Questions worth investigating, in order:
- Source of generation. Is the suggestion produced by the same model call as the answer, by a separate call, or by a lightweight local heuristic? The answer determines what the suggestion can "know" and whether it can leak context.
- Trace. Does an accepted suggestion leave a mark (a flag in the export, a telemetry event) that distinguishes it from typed text? For any organisation that audits decisions from chat logs, this is the decisive property.
- Administrative control. Can a tenant or an organisation disable the feature centrally? Claude Code documents it; the others do not, as far as the squad could find.
- Actions, not questions. In agentic products (coding assistants, Copilot agents, browser agents) a follow-up can name an action rather than a question. Occurrence 3 above was a deletion. Which products allow the generator to propose irreversible actions, and with what safeguards?
- Summarisation loss. Occurrence 5 shows a generator dropping a restriction while summarising. A simple test protocol: write a to-do item with an explicit exclusion, let the assistant discuss it, and read the suggestion.
Method: official documentation first (transparency notes, admin references), then a reproducible test conversation per product, then a reading of the exported log. Community reports are useful as leads, not as facts.
6. Takeaways for practitioners
- Decide, in writing, what counts as a decision. If it is a formula, know that the tool has read that formula many times.
- Treat the input box as part of the attack surface: what is in it before you press Enter may not be yours.
- Ask for echo-confirmation of any instruction that does not match what you just discussed; it costs one exchange and catches the fictitious gate before it reaches the register.
- Put the restriction inside the instruction, not beside it. Summaries keep verbs and objects and lose exceptions.
- Prefer rules a script could check over rules a model must interpret; they survive a change of model, of tool and of mood.
- Know where the switch is. Convenience features are worth keeping only where the register can prove afterwards who decided.
7. Sources
All read on 15 September 2026. Community threads are cited as leads, not as facts.
- S1 Anthropic, Claude Code documentation: Interactive mode, section "Prompt suggestions" and "Turn prompt suggestions off": https://code.claude.com/docs/en/interactive-mode (read 15 September 2026).
- S2 Anthropic, Claude Code documentation: Settings: https://code.claude.com/docs/en/settings (read 15 September 2026).
- S3 OpenAI Developer Community, "Disable or Customize the 'Follow Up' Suggestions" (May to September 2025): https://community.openai.com/t/disable-or-customize-the-follow-up-suggestions/1254246 ; "How to disable prompt suggestions": https://community.openai.com/t/how-to-disable-prompt-suggestions/316524 ; "Feature Request: Setting to Disable Follow-up Suggestions": https://community.openai.com/t/feature-request-setting-to-disable-follow-up-suggestions/1396688.
- S4 Microsoft Learn, Configure suggested prompts for Teams and Microsoft 365 (Copilot Studio): https://learn.microsoft.com/en-us/microsoft-copilot-studio/configure-starter-prompts.
- S5 Microsoft Learn, Responsible AI FAQ for Follow-up questions in the Copilot sidecar (preview), Dynamics 365 Finance and Operations: https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/fin-ops/copilot/faq-copilot-suggested-questions (reached through search on 15 September 2026; the direct fetch redirected to the Responsible AI overview, to be re-read).
- S6 Microsoft Learn, Use Copilot-generated prompts in your CRM solutions (Dynamics 365 Contact Center): https://learn.microsoft.com/en-us/dynamics365/release-plan/2024wave2/service/dynamics365-contact-center/use-copilot-generated-prompts-crm-solutions.
- S7 Microsoft Support, About Copilot Chat responses: https://support.microsoft.com/en-us/microsoft-365-copilot/about-copilot-chat-responses (no mention of follow-up suggestions on 15 September 2026).
- S8 Google, Gemini Apps Community, "I can't stop Gemini from offering suggestions which I don't want at the end of every prompt": https://support.google.com/gemini/thread/411870292 ; Google Workspace Admin Help, Turn Ask Gemini on or off for Meet: https://knowledge.workspace.google.com/admin/meet/turn-ask-gemini-on-or-off-for-meet.
- S9 The squad's own risk register, change log and session transcripts (internal, not published).