CODACON Blog

Quebec Law 25 and Generative AI: What to Settle Before Production

2026-10-19T00:00:00.000Z

If you run a Quebec company, you are probably its person in charge of the protection of personal information. Law 25 gives that role to the person with the highest authority by default, whether or not anyone told you (section 3.1).

Now read the list of failures that can draw an administrative monetary penalty (section 90.1). Two of the five that apply to most companies are not legal paperwork. They are engineering: failing to take reasonable security measures, and failing to handle decisions made by automated processing. Generative AI touches both.

Law 25 was not written with large language models in mind. It applies to them anyway, and the regulator has already said so in writing. Here is where Law 25 and generative AI meet, what each obligation asks of an AI system, and what the Commission d'accès à l'information has actually decided.

What Law 25 is

Law 25 is the common name for An Act to modernize legislative provisions as regards the protection of personal information (S.Q. 2021, c. 25). It amended Quebec's private sector privacy act and its public sector equivalent. It was phased in mainly between September 2022 and September 2024. Everything below is in force today.

The regulator is the Commission d'accès à l'information (CAI). Administrative monetary penalties reach $10 million or 2% of worldwide turnover, whichever is greater (section 90.12). Penal fines reach $25 million or 4%, doubled for a subsequent offence (sections 91 and 92.1). Individuals can also sue: where an unlawful infringement is intentional or results from gross fault, the court must award punitive damages of at least $1,000 (section 93.1).

Section references below are to the private sector act. We are security practitioners, not lawyers. This post explains how the obligations translate into AI architecture and controls; your counsel decides how they apply to you.

What the record shows so far

We reviewed the CAI's published decisions, its annual and five-year reports, and the joint investigations it conducts with the federal Privacy Commissioner. As of October 2026:

No published administrative monetary penalty. The CAI has had the power since September 2023. Its decisions page and annual reports show orders, not fines. Be wary of blog posts listing "real" Law 25 fines: the ones we traced were composites.
No decision interpreting section 12.1, the automated decision rule. The CAI has issued guidance and law reform recommendations, not rulings.
Plenty of orders on AI and biometrics. The CAI has ordered organizations to stop systems, destroy data, notify people, and report on algorithmic bias. Several of those decisions are cited below.
Volume is real. The CAI received 514 confidentiality incident notices in 2024–2025, 80% from the private sector (annual report 2024–2025).

The absence of fines is not a reason to wait. Orders to shut down a system after it is built are expensive, and class actions do not need a CAI decision to proceed.

Read the penalty provisions as a scope

Section 90.1 lists what can draw an administrative monetary penalty for most companies: failing to inform people about collection, collecting, using, communicating, keeping, or destroying personal information unlawfully, failing to report a confidentiality incident, failing to take the security measures required by section 10, and failing to inform a person of an automated decision or let them submit observations under section 12.1. A sixth ground applies only to personal information agents.

The act also tells you how penalties and fines are weighed. The CAI must first send a notice of non-compliance and let you respond (sections 90.3 and 90.4). At any time after a failure, you can offer an undertaking to remedy it; if the CAI accepts it and you comply, no penalty can be imposed for those acts (section 90.1). Its criteria include the measures you took to remedy the failure and your cooperation (section 90.2). For penal fines, a judge weighs whether the offence was foreseeable, whether you failed to follow recommendations or warnings to prevent it, and whether you failed to take reasonable preventive measures (section 92.3).

Two consequences follow. A security assessment whose findings sit in a drawer is worse than none: it proves the risk was foreseeable. And when a notice arrives, a credible, sequenced remediation plan is what an undertaking is built on.

Not sure which of your AI systems this reaches? Book a 30-minute call. Bring one system; you leave knowing which obligations below apply to it and your top three gaps.

1. A privacy impact assessment before you build

What the law says. Any project to acquire, develop, or overhaul an information system or electronic service delivery system involving personal information requires a privacy impact assessment (section 3.3). The person in charge of the protection of personal information must be consulted from the outset of the project.

What it means for AI. A RAG assistant, a copilot connected to your CRM, or an agent with access to a mailbox is exactly that kind of project. The assessment has to cover what the model can read, not just what users type. A retrieval index built over a shared drive inherits every piece of personal information on that drive.

On the record. In Centre de services scolaire du Val-des-Cerfs (November 2022), a school service centre built a machine learning tool to predict grade 6 dropout risk. The CAI held that the predictions were new personal information, so producing them was a collection, and that a privacy impact assessment should have been done before deployment. In its 2026 five-year report, the CAI recommends that assessments of AI systems making decisions about people include an algorithmic impact analysis. The CAI's PIA guide is the starting template.

Warning sign. The PIA is drafted after the pilot, describes the vendor's product page, and never mentions embeddings, logs, model outputs, or tool access.

2. Sending personal information outside Quebec

What the law says. Before communicating personal information outside Quebec, an organization must conduct a privacy impact assessment covering the information's sensitivity, the purposes, the protection measures including contractual ones, and the legal framework of the destination (section 17). The information may go only if it would receive adequate protection, and the transfer needs a written agreement that reflects the assessment. The same applies when you entrust a provider outside Quebec with collecting, using, communicating, or keeping the information on your behalf. Wherever the provider is, section 18.3 requires a written contract that specifies its protection measures and bars it from keeping the information after the contract ends. The provider must notify your person in charge without delay of any violation or attempted violation of confidentiality, and let them verify compliance. Few standard AI vendor terms say that.

What it means for AI. Most hosted model APIs process prompts outside Quebec, often outside Canada. Every prompt carrying personal information is a communication outside Quebec. So is a cloud vector database in another region, a managed logging service, or an evaluation tool that stores conversation samples.

On the record. The reach runs both ways. In the joint investigation of OpenAI (May 2026), the CAI found that Quebec's act applied to OpenAI even though it had no establishment or employees in Quebec. In Clearview AI (December 2021), it ordered a US company to stop collecting Quebecers' facial images. Your foreign provider is within the CAI's reach. So are you, for choosing it. We found no decision yet that turns on section 17 itself.

Warning sign. Nobody can say which region processes inference, or whether the "data residency" option covers prompts, completions, logs, and abuse monitoring, or only data at rest.

3. Decisions made by automated processing

What the law says. An organization that uses personal information to render a decision based exclusively on automated processing must inform the person no later than when it informs them of the decision (section 12.1). On request, it must disclose the personal information used, the reasons and the principal factors and parameters that led to the decision, and the right to have the information corrected. The person must be able to submit observations to a staff member in a position to review the decision.

What it means for AI. Screening job applicants, triaging insurance claims, scoring credit, or flagging accounts for closure with a model all qualify when no human meaningfully reviews the outcome. The CAI's guidance gives credit decisions, financial aid assessments, and prioritizing a request as examples. "Principal factors and parameters" is hard to produce from an LLM after the fact. It has to be designed in: logged inputs, retrieved sources, and a decision record a reviewer can read. Section 11 adds a floor: information used to make a decision about a person must be accurate when used and kept for at least one year after the decision. A model that decides today needs a decision record that still exists next October.

On the record. No decision interprets section 12.1 yet, but the CAI has shown what algorithmic accountability looks like to it. In its April 2026 Metro decision on facial recognition in grocery and pharmacy stores, it ordered the company to report every six months for two years on the systems' effectiveness, false positives, and algorithmic bias. Expect the same questions about any model that decides things about people.

Warning sign. The human in the loop approves 99% of model recommendations in under ten seconds. That is automated processing with a signature.

4. Profiling and privacy by default

What the law says. An organization that collects personal information using technology with functions to identify, locate, or profile a person must inform them beforehand and explain how to activate those functions (section 8.1). Technological products and services offered to the public must apply the highest level of confidentiality by default, without any action by the user (section 9.1). Browser cookies are excluded.

What it means for AI. Personalization, sentiment analysis, and behavioural inference built into a customer-facing assistant are profiling. Training on user conversations, and conversation memory across sessions, are privacy settings. The feature ships off, and the user turns it on.

On the record. This is where the CAI has been most direct about AI. In the OpenAI investigation, it found that training models on ChatGPT conversations by default breached section 9.1, despite the information OpenAI gave users. In the joint TikTok investigation (September 2025), it found that profiling functions were not off by default, in breach of sections 8.1 and 9.1. It rejected TikTok's argument that settings designed for the European GDPR satisfied section 9.1: the Quebec rule is worded differently.

Biometrics carry their own lead time. Law 25 amended the IT framework act so that a biometric database, such as facial or voice templates, must be disclosed to the CAI at least 60 days before it goes into service (section 45). That disclosure is what put the Metro project in front of the CAI.

Warning sign. The product team describes long-term memory or "help improve the model" as an opt-out, and nobody has asked whether it is on by default.

5. Secondary use and training data

What the law says. Personal information may be used only for the purposes for which it was collected, unless the person consents or a specific exception applies (section 12). Sensitive information requires express consent. One exception covers study, research, or statistics using de-identified information.

What it means for AI. Using support transcripts to fine-tune a model, or loading customer files into a general-purpose knowledge assistant, is a new use. That analysis belongs before the training run, not after.

On the record. In the OpenAI investigation, the CAI accepted that training a language model can be a legitimate purpose, but found the collection overbroad. It also held that information published on the web is not "public" under the act just because it can be scraped. In Val-des-Cerfs, student records collected for schooling were reused to train a prediction tool without consent or notice, and the CAI ordered the school service centre to inform parents. Inferences stay in play afterwards: a person's right of access covers all the information you hold on them (section 27). Only portability, the export in a structured format, excludes information "created or inferred" from their data.

Warning sign. The training dataset was assembled by an engineer with production read access and a deadline.

6. Confidentiality incidents

What the law says. When an organization has reason to believe a confidentiality incident has occurred, it must take reasonable measures to reduce the risk of injury and prevent new incidents. If the incident presents a risk of serious injury, it must promptly notify the CAI and the affected individuals (section 3.5). Section 3.6 defines an incident, section 3.7 sets how to assess the risk, and section 3.8 requires a register of every incident, reported or not.

What it means for AI. Generative AI creates new incident paths. A prompt injection that makes an assistant reveal another customer's file is unauthorized communication of personal information. So is a retrieval system that returns documents the user was never authorized to see, or a leaked API key that exposes conversation logs. Your incident response plan needs to recognize them, and your logs need to show what the model actually returned.

On the record. 514 incident notices reached the CAI in a single year. The regulator is used to them. What it will ask about an AI incident is what you can reconstruct: which documents were retrieved, what was returned, and to whom.

Warning sign. The AI system is out of scope for the incident response plan, or its logs record the prompt but not the retrieved documents and the output.

7. Retention and destruction

What the law says. When the purposes are achieved, personal information must be destroyed or anonymized according to the anonymization regulation in force since May 2024 (section 23). The organization's governance policies must frame retention and destruction (section 3.2).

What it means for AI. Personal information persists in more places in an AI stack than in a classic application: prompt logs, vendor abuse monitoring stores, embeddings, caches, evaluation sets, and fine-tuned weights. You cannot destroy what you have not inventoried, and you cannot easily remove one person from a fine-tuned model.

On the record. Destruction is the CAI's standard remedy. It ordered Clearview AI to destroy the images and biometric vectors it held on Quebecers within 90 days, and ordered Transcontinental (September 2024) to shut down an employee facial recognition system and destroy the templates. An order to destroy is cheap to comply with only if you know where every copy lives.

Warning sign. A deletion request is honoured in the CRM while the same customer's emails remain in the vector index.

The courts are moving too

Class actions do not wait for the CAI. In Badji c. Zenleads inc. (2026 QCCS 5), the Superior Court authorized a class action against the data broker behind Apollo.io, including claims for punitive damages under section 93.1. It found that loss of control over personal data and algorithmic profiling were plausible harms (Osler summary). The Court of Appeal granted leave to appeal, with a hearing scheduled for November 2026. In Doan c. Clearview AI inc. (2024 QCCS 3968), the Superior Court authorized a Quebec class action over scraped facial images (Osler review). Neither has been decided on the merits.

Accountability sits at the top

As noted at the top, the person with the highest authority, usually the CEO, holds the role of person in charge of the protection of personal information by default (section 3.1). That role can be delegated in writing, but the accountability does not disappear. The title and contact information must be published on the organization's website.

That is why Law 25 is a board topic, not only a compliance one. Directors should expect to see where AI systems process personal information, which ones have a completed PIA, and what incidents were recorded. Our post on questions a board should ask before an AI coding tool rollout applies the same discipline to a different decision.

How we help

We work alongside your privacy officer and legal counsel. They interpret the law; we make sure the system does what the PIA says it does.

GenAI Security Readiness Assessment. A four-week, fixed-scope review of your AI architecture, data flows, identity model, and providers, including whether your AI vendor terms meet section 18.3. Findings are mapped to Law 25 alongside NIST, ISO, and OWASP references, with an owner and a remediation for each. It produces the technical evidence a section 3.3 PIA and a section 17 assessment need.
Secure architecture and AI-driven SDLC. Threat modeling at design time, so data residency, retention, access control on retrieval, privacy-protective defaults, and decision logging are built in rather than retrofitted. See our services.
Remediation plans. When a finding, an incident, or a notice of non-compliance lands, we turn it into a sequenced, owned, and dated plan your counsel can put in front of the CAI.
Incident response. Playbooks and tabletop exercises for AI-specific incidents such as prompt injection, retrieval leakage, and exposed keys, with a clear path to the "risk of serious injury" assessment and CAI notification.
Board advisory. Briefings for CEOs and directors who carry Law 25 accountability, and a reporting format that shows AI privacy exposure in one page.

Five questions to test yourself

Which of your AI systems make a decision about a person with no meaningful human review?
For a decision a model made eleven months ago, can you produce the inputs it used and the sources it retrieved?
Do your AI vendor contracts require the vendor to tell you about attempted breaches, and let you verify?
Is training on user data, or memory across sessions, off by default in every AI product you offer the public?
If a prompt injection exposed one customer's file to another today, would your incident plan recognize it, and could you tell the CAI what was returned?

If you hesitated on two of them, that is the conversation to have before production, not after a notice. Book a 30-minute call: bring one AI system, and leave with the obligations that apply to it and your top three gaps.

Why Use NIST OSCAL? Compliance as Data, Not Documents

2026-10-14T00:00:00.000Z

TL;DR

What it is. OSCAL, the Open Security Controls Assessment Language, is an open standard from NIST. It expresses security controls, system security plans, assessments, and remediation plans as structured data in XML, JSON, or YAML. It is free, open source, and developed in the open on GitHub. The latest release at time of writing is v1.2.3.

Why use it. Most compliance work still lives in Word documents and spreadsheets that are out of date the day they are signed. OSCAL turns that paperwork into data that tools can validate, compare, and update. Write a control implementation once and reuse it across frameworks. Hand assessors evidence they can process by machine. Move from an annual snapshot toward continuous assurance.

Why now. Regulators are publishing controls as data. US federal cloud authorization is moving toward machine-readable packages, Canada's control catalogues descend from the NIST controls OSCAL was built around, and Germany now publishes its national baseline natively in OSCAL. Meanwhile the number of frameworks you answer to only grows.

The problem OSCAL solves

Ask a security team how many frameworks it reports against. SOC 2, ISO 27001, NIST SP 800-53, CIS, provincial privacy law, customer questionnaires, and now AI-specific frameworks. Each one gets its own document, its own spreadsheet, and its own late-night copy-and-paste session before the audit.

The underlying facts barely change between them. Multi-factor authentication is enforced the same way whether ISO or SOC 2 asks about it. Yet the same answer is rewritten for each framework, in prose no tool can check, by people who would rather be fixing things.

That model has three failures:

Drift. The document says one thing, the system does another, and nobody notices until an assessor or an attacker does.
Duplication. Every new framework multiplies the work instead of reusing what is already known.
Opacity. A 300-page PDF cannot be queried. "Which controls depend on this identity provider?" takes a week to answer.

What OSCAL actually is

NIST describes OSCAL as a set of formats that give a standardized, machine-readable representation of control information, from catalogs to assessment results. The models fall into three layers:

Control layer. Catalogs hold the control definitions, such as NIST SP 800-53. Profiles select and tailor controls into a baseline, such as Moderate or your own internal standard.
Implementation layer. Component definitions describe how a product or service satisfies controls. A system security plan (SSP) describes how a specific system implements its baseline.
Assessment layer. Assessment plans, assessment results, and plans of action and milestones (POA&M) record what was tested, what was found, and what is being fixed.

Each layer references the one above it. An SSP points to a profile, which points to a catalog. An assessment result points to the SSP. That chain of references is the point: change a control, and you can trace every system and finding it touches.

Why organizations should use it

Write once, map many times

Describe how your platform enforces encryption at rest as a component definition. Reference it from every system that uses the platform. When a framework or a customer asks, the answer comes from the same source, not a fresh rewrite.

Documentation that can be checked

OSCAL content validates against published schemas. A missing control, a broken reference, or a parameter left blank is caught by a tool, not by an assessor three weeks before your deadline. That changes compliance from a writing exercise into something closer to software, with versions, reviews, and automated checks.

Vendor and supplier assurance at scale

If your cloud and SaaS providers publish component definitions, you inherit their control descriptions as data instead of re-reading their PDFs. The same applies in reverse: if you sell to regulated buyers, a clean OSCAL package shortens their review of you.

Continuous assurance instead of annual theatre

Once controls, systems, and findings are linked as data, evidence from your pipelines and cloud configuration can feed assessment results directly. Status becomes a query, not a project. This is where OSCAL meets the rest of a modern secure software development lifecycle: compliance becomes another artifact the pipeline produces and checks.

Exceptions become data, too

Every program has controls it cannot meet as written: a legacy system that cannot take multi-factor authentication, a scanner finding that turns out to be a false positive, a risk the business decides to accept. In most organizations these exceptions live in email threads and a spreadsheet tab. They rarely have an expiry date, the approver has often moved on, and nobody looks at them again until an assessor asks.

OSCAL records exceptions inside the risk record, not beside it. A risk in a POA&M moves through statuses such as open, remediating, deviation-requested, deviation-approved, and closed. Its response is typed as avoid, mitigate, transfer, share, or accept. Properties flag a risk as a false positive, accepted, or risk-adjusted, and a risk log records each status change, who logged it, and when. Because the risk references the finding, the control, and the system, an exception never drifts away from what it excuses.

The payoff is the same as everywhere else in OSCAL: questions become queries. Which accepted risks reach their deadline this quarter? Which systems carry exceptions against the same control? Who approved this deviation, and on what evidence? Give every exception a deadline and the pipeline can flag it for review before it quietly becomes permanent.

Regulatory direction of travel

Machine-readable assurance is no longer a US-only idea. Where you sit changes the details, not the direction.

United States. FedRAMP's RFC-0024 proposes that Rev5 cloud providers submit authorization packages in an approved machine-readable format, with OSCAL named as one, starting with new submissions on September 30, 2026. Check FedRAMP's final rules for the scope and dates that apply to you. NIST also publishes its own catalogs, including SP 800-53 and SP 800-171, as OSCAL content.

Canada. The federal control catalogue, ITSG-33, is based on NIST SP 800-53, so it maps naturally onto OSCAL's catalog and profile model. The Canadian Centre for Cyber Security has not published an official OSCAL edition, but AWS has encoded the CCCS Medium and Protected B High Value Assets cloud profiles in OSCAL as open samples. For defence suppliers, the Canadian Program for Cyber Security Certification (CPCSC) rests on a standard technically identical to NIST SP 800-171, which NIST already publishes in OSCAL. Level 1 self-assessments enter select defence contracts from summer 2026, and Level 2 third-party certification from spring 2027.

Quebec. Nothing in Quebec requires OSCAL today. But Quebec organizations already carry structured obligations that live badly in documents: public bodies answer to the government's information security directive and framework, and Law 25 requires, among other things, privacy impact assessments for certain projects and a register of confidentiality incidents. OSCAL lets you model those obligations as your own catalog, alongside ISO 27001 or ITSG-33, and answer each from the same evidence.

Europe. NIS2 and DORA do not prescribe a data format, and the EU has not published official OSCAL catalogs for them. Germany has moved first: the BSI now publishes its security requirements, including Grundschutz++, the successor to IT-Grundschutz, as OSCAL documents on GitHub. The catalog was published in September 2025, with NIS2 implementation as a legal driver. If one of Europe's most influential baselines now ships as data, others will follow.

If none of these apply to you, the signal still matters: regulators and large buyers are tired of reading PDFs, and machine-readable assurance is where procurement is heading.

No lock-in

OSCAL is an open standard with open-source tooling from NIST and the community. Your compliance data stays portable between GRC platforms, consultants, and assessors. That alone is worth something the next time a tool contract renews.

Why adoption stalls

If OSCAL is this useful, why do so few organizations use it? In our experience, for three reasons:

The gap between the standard and the system. The schemas are well documented. Mapping your actual architecture, inherited controls, and shared responsibilities into them is not.
The legacy pile. Years of SSPs, spreadsheets, and policy documents do not convert themselves, and a naive conversion just produces machine-readable versions of the same drift.
Ownership. OSCAL sits between security, compliance, engineering, and procurement. Without a clear owner and a narrow first scope, it becomes a side project that never ships.

None of these are technical blockers. They are scoping, sequencing, and change-management problems.

Where to start

Pick one system and one baseline. Not the whole estate. One system that matters, assessed against the framework you already answer to most often.
Model the shared components first. Identity, logging, encryption, and your cloud platform carry most controls. Describe them once as component definitions.
Build the SSP from references, not prose. Let the system plan inherit from components instead of restating them.
Validate in the pipeline. Run schema and content checks on every change, the same way you test code.
Connect one evidence source. Feed one automated control check into assessment results, prove the loop works, then expand.

How CODACON can help

The standard is free. The hard part is the translation: turning your real systems, inherited controls, and existing paperwork into an OSCAL model your teams will actually maintain, then embedding it in how you build and assess.

That is where we work. We help organizations scope a first OSCAL implementation, model their shared components, convert what is worth keeping from the legacy pile, and wire validation into their delivery pipeline so the model stays true after we leave.

We have already done this end to end for one organization. The engagement is confidential, so we cannot name the client or share their artifacts, but we can walk you through the approach, the pitfalls we hit, and what it would take in your environment. If you are weighing OSCAL, or have started and stalled, let's talk.

What Should a Board Ask Before Approving an AI Coding Tool Rollout?

2026-09-06T00:00:00.000Z

Vendor demos show how fast AI can write code. The proposal in front of the board should show what happens when that code is wrong.

Before approving an AI coding tool rollout, directors need more than a productivity promise. They need evidence of value, clear limits on access, and someone accountable for stopping the rollout if those limits fail.

Here are the questions that turn an impressive demo into a defensible decision.

Where the board's job starts and stops

A board does not approve every developer tool. It governs material exposure and delegated authority. An AI coding tool crosses that line when it can read source code, hold secrets, or change production systems. Management owns implementation. The board owns the conditions for proceeding.

We frame the questions using the NIST AI Risk Management Framework 1.0 (Govern, Map, Measure, Manage), its Generative AI Profile, NIST AI 600-1, for AI-specific risks, and the Secure Software Development Framework, SP 800-218, for software assurance. NIST describes the AI RMF as voluntary. The questions are our application of that guidance, not a NIST checklist, a certification, or a legal requirement.

1. What business outcome are we buying, and how will we prove it?

Why it matters. Code volume is not a business outcome. Faster code can also mean more review, defects, and rework. METR's July 2025 randomized study found that 16 experienced open-source developers took 19% longer on 246 tasks when allowed to use early-2025 AI tools, while believing they had been faster. METR cautions against generalizing to other teams or current tools, but the lesson holds: perceived speed is not measured speed.

Evidence to request. A baseline taken before the pilot and repeated after: delivery lead time, defect rate, review time per change, rework, and total cost. The AI RMF asks organizations to document expected benefits and costs, including non-monetary ones (MAP 3.1 and 3.2).

Warning sign. The business case rests on the vendor's productivity figure or on "acceptance rate," the share of suggestions developers kept.

2. What can the tool read, retain, transmit, and change?

Why it matters. Exposure depends on permissions, not product names. Suggesting code in an editor is one thing. An agent that runs commands, edits repositories, and triggers deployments is another. AI 600-1 (section 2.9) describes indirect prompt injection: instructions hidden in content the AI reads, such as a code comment or a web page, that can cause it to leak data or run malicious code. Autonomy multiplies that risk.

Evidence to request. A data map: what source code, secrets, and customer data the tool can access, what the vendor retains and for how long, whether inputs train models, and which subprocessors are involved. A permission map: what the tool can execute, commit, and deploy, and whether a human approves each step. Suggest, execute, and deploy should each carry their own approval conditions.

Warning sign. "The vendor doesn't train on our data," with no confirmation that the statement applies to the tier and settings actually purchased.

3. What evidence shows our controls work with AI-generated code?

Why it matters. The SSDF describes practices for reducing vulnerabilities in released software, limiting their impact, and preventing recurrence. Our position: AI-generated code must pass the same assurance process as any other code. The risk is that volume overwhelms it. AI 600-1 names automation bias, the tendency to defer to machine output, as a risk in its own right (section 2.7).

Evidence to request. Pilot results from the company's own repositories and workflows, not a vendor sandbox (MEASURE 2.3). Confirmation that human review, security testing, dependency and licence checks, and build pipeline protection apply to AI-generated changes. Reviewer capacity: changes per reviewer per week, before and after.

Warning sign. Approval times fall while change volume rises. That is rubber-stamping, not efficiency.

4. Who is accountable when something goes wrong?

Why it matters. The AI RMF places responsibility for AI risk decisions with executive leadership (GOVERN 2.3). A rollout owned by "engineering" has no one to call when the tool pushes a credential to a public repository.

Evidence to request. A named executive sponsor with documented acceptance of residual risk. A responsibility matrix spanning engineering, security, legal, and procurement. An exception process: who can grant broader access, and who is informed.

Warning sign. Accountability is shared, or sits with the champion who selected the tool.

5. What vendor and contractual risks are we accepting?

Why it matters. The tool is a supply-chain dependency. The AI RMF asks for third-party risk policies, including intellectual property, and contingency plans for third-party failures (GOVERN 6.1 and 6.2). AI 600-1 notes that the legal status of generated content resembling copyrighted work is still debated (section 2.10). Contract terms are the practical answer, and they vary by tier.

Evidence to request. A vendor assessment covering confidentiality, intellectual property protections and their limits, incident notification timelines, the vendor's right to change models or terms, and exit arrangements including data deletion. A continuity plan if the service is withdrawn. SP 800-218A, NIST's profile for building AI models, applies to vendor diligence, not to the company's developers.

Warning sign. Legal reviewed the standard terms, but nobody confirmed which apply to the purchased configuration.

6. What are the pilot's success criteria and stop conditions?

Why it matters. The AI RMF expects mechanisms and assigned responsibility to disengage or deactivate a system behaving inconsistently with its intended use (MANAGE 2.4). A pilot without stop conditions is a rollout with extra steps.

Evidence to request. Phased access starting with low-risk repositories and non-production environments. Acceptance thresholds set in advance. Incident triggers that pause the pilot: a credential exposure, a vulnerable change reaching production, a data-handling breach. A tested revocation: someone has turned off access and timed it.

Warning sign. Success criteria are written after the pilot, or the stop authority's objectives depend on the rollout.

7. What will the board see after approval?

Why it matters. Approval starts oversight. The AI RMF calls for post-deployment monitoring, incident response, and change management (MANAGE 4.1).

Evidence to request. A one-page quarterly dashboard: outcomes against baseline, adoption by team, defect and rework trends, security incidents involving the tool, control exceptions granted, and exposure changes such as new permissions, model versions, or repositories in scope.

Warning sign. The first report covers adoption and satisfaction only.

Hypothetical example: evidence changes the decision

Illustrative scenario, not a client case. A manufacturer's CTO proposes a company-wide rollout of a coding agent, citing a vendor claim of 40% faster delivery. The audit committee asks for question 1 and 3 evidence. A six-week pilot on two internal services shows lead time down 11%, review time per change up 30%, and two vulnerable dependencies caught only after merge. The committee approves suggestion-only mode across engineering, keeps execute permissions with the pilot teams for one more quarter, and requires reviewer capacity and scan coverage in the next report. Same tool, different decision.

Approval checklist

Baseline and post-pilot measurements of outcomes and cost
Data map and permission map, with separate conditions for suggest, execute, and deploy
Evidence that review, security testing, and pipeline controls cover AI-generated code
Named executive owner, responsibility matrix, and exception process
Vendor assessment, contract terms confirmed for the purchased tier, continuity plan
Written success criteria, incident triggers, and a tested revocation
Quarterly dashboard format agreed

Three decisions

Approve a bounded pilot when the value case is credible but unproven, and the pilot stays in low-risk environments with stop conditions.

Approve expansion subject to conditions when pilot evidence meets the thresholds and the remaining gaps are named, owned, and dated.

Defer pending evidence when management cannot yet answer questions 1, 2, or 6. That is not a rejection. It is oversight.

For an independent view on rollout readiness, our GenAI security readiness assessment covers this ground.

Why We Started CODACON

2025-07-24T00:00:00.000Z

AI is reshaping security faster than most organizations can adapt. Detection systems powered by large language models catch threats that rule-based tools miss. AI-augmented development pipelines find vulnerabilities before code ships. And boards that once treated security as an IT line item now face questions about AI risk they are not equipped to answer.

The gap between what AI makes possible and what most security teams actually do is wide. That is the gap CODACON exists to close.

What we mean by AI security

We use the term broadly and deliberately. AI security is not one thing:

AI-assisted defense — using AI to detect, triage, and respond to threats faster than human-only workflows allow.
Securing AI systems — making sure the AI you build and deploy does not become your biggest vulnerability.
AI-driven SDLC — integrating AI into the software development lifecycle to catch security issues early, not in production.
AI risk governance — helping leadership understand and manage the risks that come with adopting AI across the organization.

Most firms specialize in one of these. We work across all four because in practice they are not separable. The team building your AI pipeline is the same team that needs to secure it, and the board asking about AI risk needs answers grounded in what the engineering team actually ships.

How we work

We are a small, senior team. We do not sell packages or run a managed SOC. Every engagement is scoped to the client's actual situation — their stack, their threat model, their team's capability.

We are direct. If something is working, we will not invent a project to justify our presence. If something is broken, we will say so without hedging.

What is next

This blog will cover what we learn as we work: practical techniques for AI-driven security, architecture decisions that make systems harder to break, and honest assessments of which AI security tools actually deliver.

If you are building with AI and thinking about security — or thinking about AI and worried about security — we should talk.