Thousands of Agents: How We Build and Manage an Agent Workforce

An agent investigating a security finding needs freedom to follow the evidence. An agent changing a production configuration needs tightly bounded authority and independent verification. Both need to understand the customer’s environment and stay within that customer’s data and permission boundaries.

Running a workforce of these agents makes those requirements harder to maintain. Investigations can span hours, credentials can expire, and agent definitions change as teams improve them. Throughout those changes, each run must stay within the customer’s data and permission boundaries and record what the agent did and which version it used.

At Cogent, we built our Agent Development Platform (ADP) around this unique combination of demands. Fixing a cloud misconfiguration requires understanding which services rely on the setting being changed. Upgrading a vulnerable dependency in a base image requires verifying the fix in derived images and checking that the applications built on them still work. These requirements shaped ADP as a security-native agent control plane: a shared system for managing agents, restricting their access to each customer’s environment, enforcing policy outside the model, and linking every run’s evidence to the version that performed it.

Different jobs require different levels of autonomy and control

We treat each agent as a member of a workforce, with a defined job, success criteria, instructions, tools, and limits on its authority. For each agent, Cogent selects the harness that runs its reasoning and tool calls, and the execution mode suited to its job, such as an interactive session or background execution. Whether created through code or a user interface, agents remain inspectable and manageable through the same platform.

Each use case brings a different operating requirement:

• Product agent: acts on behalf of a user, responds promptly, and clearly indicates when it has finished or needs input.

• Background agent: handles scheduled or event-driven work and can resume after interruptions.

• Cloud agent: an agent running in an isolated execution environment, with scoped credentials and network policy.

• Internal operations agent: cross-system access to logs, infrastructure, code, and incident data, bounded by strong identity, complete auditability, and approval for high-impact actions.

Each job occupies a different point on the autonomy spectrum, with controls matched to its responsibilities.

Consider a cloud-misconfiguration remediation agent investigating a service exposed by an overly permissive network rule. It gathers evidence, identifies which applications depend on that rule, and proposes a narrower configuration. Because this change requires approval, the agent waits for that decision. The approved change then passes predefined validation checks and is executed using credentials limited to the authorized scope. The result is checked afterward to confirm whether the exposure was resolved, and the trace—the run’s evidence record—captures the evidence, policy decisions, actions, and outcome.

Context Engine gives agents a shared understanding of the enterprise

Context Engine is the system we’re building at Cogent to give agents a working understanding of each customer’s environment. Specialized agents connect fragmented data into evidence-backed knowledge of architecture, ownership, dependencies, and operating constraints. Each agent can build on that accumulated understanding rather than start from scratch.

For the network-rule change, its knowledge graph connects the rule to the affected service, dependent applications, and their owners. Those relationships help the remediation agent identify which connections could break, assess the blast radius, and weigh potential disruption against the exposure being addressed.

For container-image remediation, that graph connects running workloads to build pipelines and image lineage. Tracing those relationships to a shared base layer helps identify where one fix can address multiple affected images—and which image and pipeline owners need to act.

Our VR-1 work showed why reasoning depends on environmental context. A separate post will explore how Context Engine builds that understanding for our agent workforce.

Identity and policy define the trust boundary

A production system must identify the agent, its immutable version, the customer and user context, and its granted capabilities. Permissions and policies determine which systems and subsystems the agent can access, which actions it can take, and when it must request approval. Linking those decisions to the agent and its version makes each run auditable.

The model never governs its own authority.

Simplified runtime-policy excerpt; additional workflow permissions are omitted:

# Agent Identity
agent_key: global:cloud_misconfiguration_remediation_agent
agent_version

# Agent Identity
agent_key: global:cloud_misconfiguration_remediation_agent
agent_version

# Agent Identity
agent_key: global:cloud_misconfiguration_remediation_agent
agent_version

# Sample policy
@id("remediation-agent-read-findings")
@severity("low")
@description("Allow the misconfiguration remediation agent to read findings")
allow (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"read_findings", context)
when { true };

@id("remediation-agent-cloud-config-changes")
@severity("high")
@description("Escalate cloud configuration changes by the agent")
escalate (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"change_cloud_configuration", context)
when { true };

@id("remediation-agent-default-deny")
@severity("medium")
@description("Deny all other misconfiguration remediation agent actions")
deny (agent == Agent::"global:cloud_misconfiguration_remediation_agent"

# Sample policy
@id("remediation-agent-read-findings")
@severity("low")
@description("Allow the misconfiguration remediation agent to read findings")
allow (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"read_findings", context)
when { true };

@id("remediation-agent-cloud-config-changes")
@severity("high")
@description("Escalate cloud configuration changes by the agent")
escalate (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"change_cloud_configuration", context)
when { true };

@id("remediation-agent-default-deny")
@severity("medium")
@description("Deny all other misconfiguration remediation agent actions")
deny (agent == Agent::"global:cloud_misconfiguration_remediation_agent"

# Sample policy
@id("remediation-agent-read-findings")
@severity("low")
@description("Allow the misconfiguration remediation agent to read findings")
allow (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"read_findings", context)
when { true };

@id("remediation-agent-cloud-config-changes")
@severity("high")
@description("Escalate cloud configuration changes by the agent")
escalate (agent == Agent::"global:cloud_misconfiguration_remediation_agent", tool == Tool::"change_cloud_configuration", context)
when { true };

@id("remediation-agent-default-deny")
@severity("medium")
@description("Deny all other misconfiguration remediation agent actions")
deny (agent == Agent::"global:cloud_misconfiguration_remediation_agent"

For this remediation agent, the Agent Gateway and Policy Engine enforce the approval boundary. The proposed action is evaluated against its verified identity, customer scope, and applicable policy to allow, escalate, or deny it, outside the model’s reasoning.

In internal testing, constraining the available actions sometimes helped agents finish in fewer iterations. We will share the test setup and supporting evidence in a future Policy Engine post.

Every change needs an explicit lifecycle

From agent definition to production execution

Product and engineering teams should be able to add an agent with minimal setup by defining its job, prompt blocks, skills, tools, and policies—not by building another service. ADP provides the shared versioning, deployment, and execution infrastructure, so teams can test a candidate and select it for production without redeploying the platform. That separation is central to our design for supporting thousands of agent definitions.

ADP runs the selected agent version with the customer’s context and applicable permissions. The results inform the next round of testing and refinement. Teams can improve agents without building a separate deployment and operations system for every new job.

The cloud-remediation agent introduced earlier can be expressed as a compact specification covering its job, prompt blocks, skills, tools, integrations, and static policies. This illustrative YAML uses simplified names rather than ADP’s internal configuration schema.

name: Cloud Misconfiguration Remediation Agent
key: cloud_misconfiguration_remediation_agent
description: Resolve cloud misconfigurations and verify the result.
harness: adp-sdk # which harness to use
mode: background # execution mode 
skills:
  - exposure-investigation
  - change-impact-analysis
tools:
  - validate_configuration_change
  - verify_remediation
mcps:
  context_engine:
    scopes: [read]
  cloud_environment:
    scopes: [read, write]
agent_conf:
  # Predefined prompt blocks; runtime context is supplied per run.
  system_prompt_spec:
    blocks:
      - id: tenant_context
        source_type: runtime
      - id: guardrails
        source_type: common_resource
        source_name: guardrails
      - id: temporal_context
        source_type: runtime

# Static policies define baseline constraints.
# Separate runtime policy evaluates each action using
# customer permissions, session context, and prior actions.
policy

name: Cloud Misconfiguration Remediation Agent
key: cloud_misconfiguration_remediation_agent
description: Resolve cloud misconfigurations and verify the result.
harness: adp-sdk # which harness to use
mode: background # execution mode 
skills:
  - exposure-investigation
  - change-impact-analysis
tools:
  - validate_configuration_change
  - verify_remediation
mcps:
  context_engine:
    scopes: [read]
  cloud_environment:
    scopes: [read, write]
agent_conf:
  # Predefined prompt blocks; runtime context is supplied per run.
  system_prompt_spec:
    blocks:
      - id: tenant_context
        source_type: runtime
      - id: guardrails
        source_type: common_resource
        source_name: guardrails
      - id: temporal_context
        source_type: runtime

# Static policies define baseline constraints.
# Separate runtime policy evaluates each action using
# customer permissions, session context, and prior actions.
policy

name: Cloud Misconfiguration Remediation Agent
key: cloud_misconfiguration_remediation_agent
description: Resolve cloud misconfigurations and verify the result.
harness: adp-sdk # which harness to use
mode: background # execution mode 
skills:
  - exposure-investigation
  - change-impact-analysis
tools:
  - validate_configuration_change
  - verify_remediation
mcps:
  context_engine:
    scopes: [read]
  cloud_environment:
    scopes: [read, write]
agent_conf:
  # Predefined prompt blocks; runtime context is supplied per run.
  system_prompt_spec:
    blocks:
      - id: tenant_context
        source_type: runtime
      - id: guardrails
        source_type: common_resource
        source_name: guardrails
      - id: temporal_context
        source_type: runtime

# Static policies define baseline constraints.
# Separate runtime policy evaluates each action using
# customer permissions, session context, and prior actions.
policy

The agent definition specifies the prompt blocks in advance; runtime blocks supply customer and time context for each run. Static policies set baseline constraints, while separate runtime policy evaluates actions using customer permissions, session context, and prior actions. Declaring write access does not itself authorize a change.

Versioning separates experimentation from production

Today, ADP can store an agent’s configuration and system prompt as an immutable version. Branches organize experiments, letting teams compare and test candidates without overwriting earlier definitions.

Over time, versioning will capture more of the inputs that shape behavior, including tools, skills, model settings, and policy. A trace identifies the version that ran; capturing more of its dependencies will make that behavior easier to reproduce and compare.

Deployment selects what production runs

An agent release is independent from service deployment. ADP runs the version explicitly selected for production, so teams can change agent behavior without redeploying the runtime service. Each release retains an activation event, deployment history, and a rollback path.

Deployment selects an exact version. Updating an experimental branch does not automatically change production behavior.

Asynchronous execution lets the workforce scale

A burst of new findings can trigger more investigations than the system can handle at once. Asynchronous invocation separates accepting that work from completing it. A centralized queue holds pending agent requests, and workers execute them as capacity becomes available. As queued work grows, the platform adds worker capacity; model, tool, and sandbox limits still constrain how much work can run at once.

Isolated execution needs a control plane

Some jobs need an isolated environment for code execution. The Sandbox Control Plane (agent-sandbox-control) sits between ADP or durable workers and sandbox providers. Before provisioning, it checks tenant-specific quotas, provider capacity, and network rules. It selects a prebuilt container image, injects short-lived credentials, tracks lifecycle state, and reclaims expired or orphaned sandboxes.

Agent health is an end-to-end property

A green agent process doesn’t mean a healthy agent. The remediation agent’s runtime can be healthy while its investigation is blocked. The customer’s integration credentials may have expired, an MCP server may be unavailable, or a misconfigured permission policy may block access the agent needs for its assigned job. Agent Health must identify what is preventing that agent from working for that customer.

• Definition and execution: selected version, runtime, worker, and durable executor.

• Isolated compute: sandbox provisioning, capacity, lifecycle, credentials, and network policy.

• Intelligence and context: model access and the Context Engine.

• Capabilities: Agent Gateway, MCP servers, and individual tools.

• Operating conditions: identity credentials, authorization, quotas, queues, latency, timeouts, errors, and resources.

These tenant- and version-specific signals help teams diagnose failures, recover credentials, restart execution, or roll back.

Every run should improve the next version

Deployment is not the end of the lifecycle. Each run produces evidence that teams can inspect, evaluate, and use to improve the next version.

Traces make behavior inspectable

A trace is the evidence record for a run. It connects the outcome to the agent identity and immutable version, then captures the relevant context, model activity, tool calls, policy decisions, errors, cost, and final result at the appropriate level of detail.

That evidence separates different failure modes: bad context, wrong tool selection, a degraded dependency, a policy denial, or poor reasoning after valid evidence. Two runs can produce similar prose while taking very different paths.

Evaluation determines whether the work was right

For the remediation agent, evaluation asks whether the change addressed the exposure, respected the approved scope, and produced sufficient evidence to verify the result. Evaluation is a funnel and feedback loop:

• Tests before release: the agent equivalent of unit and integration tests for known behavior.

• Checks on live runs: predefined rules and model-based assessments review traces and outcomes.

• Review across runs: separate evaluator agents with the same domain capability review traces and outcomes across runs, compare versions, and narrow uncertain cases before human review. They receive evidence access, not the production agent's execution authority.

• Human review: judgment for ambiguous or high-impact cases.

• Tests that prevent repeat failures: validated failures become durable test cases, stronger guardrails, or both.

Validated failures become tests that the next candidate must pass before it is selected for production.

ADP is the agent control plane

ADP manages the agent lifecycle. The ADP SDK and Cogent’s custom harness execute the selected version. The Context Engine supplies customer context, the Agent Gateway controls tool access, and the Sandbox Control Plane manages isolated execution environments.

Building a production-ready cybersecurity agent workforce takes more than connecting agents to security tools. Security context, tenant isolation, governed action, and verifiable evidence must shape the platform from the beginning. That is the platform we are building at Cogent: ADP, a security-native control plane for an agent workforce.

Future posts will explore the Context Engine, Policy Engine, Cloud Agents, and the automations built from these managed agents.