Build an agent self-improvement routine
AI agents tend to fail in the same places. A bug triage agent works out the right priority but can't write it back to Jira. It refuses a bug because a skill rule doesn't fit how another team works. It repeats searches it doesn't need, wasting tokens on every run. Each failure looks like a one-off, but most trace back to the agent's setup: a tool it's missing, a skill that gives it the wrong guidance, or steps it doesn't need.
For every one of these failures, you pay three times: once for the run that failed, again for the loops and retries that burned tokens, and a third time for fixing it yourself. A better model won't fix this. The evidence of what's wrong is already in the agent's conversations.
In this guide, we will build a self-improvement routine that runs every 24 hours. It reads each agent's conversations from the last day, finds the failures that repeat, traces each one to the tool, skill or prompt that causes it, and suggests a concrete fix. A person reviews each suggestion on a dashboard, then applies or dismisses it in one click.
Common use cases
- Missing tools - an agent works out what to do but can't do it, because its allowed tools don't include the write operation it needs.
- Skills that contradict the setup - a skill rule that doesn't fit how a team works, or that contradicts the agent's prompt. Neither piece is wrong on its own, so fixing one piece at a time never surfaces it.
- Wasted tool calls - repeated searches, calls whose results are never used, and trial-and-error retries that make every run more expensive.
- Setups that never get revisited - an agent's tools and skills are usually guessed on day one and never revisited, so most agents get too much access or miss what they need. Once this routine is in place, you can stop guessing: start every new agent with the minimum of tools, skills and access it needs, and let the routine suggest each addition only when its conversations show it needs it.
How it works
The agent never changes itself. Every suggestion is stored with its evidence (the conversations where the problem happened) and its root cause (a specific tool, skill or prompt), a person decides whether to apply it, and every decision is logged as a workflow run, together with the person who applied or dismissed it.
Prerequisites
- A Port account with admin permissions, to edit blueprints and create workflows.
- At least one AI agent that people use, so its conversations are recorded as AI invocations.
- If your agents load skills: a
skillblueprint with aninstructionsproperty, related to the_ai_agentblueprint through askillsrelation. See Set up a skills registry to create it.
This guide reviews Port AI agents, whose conversations Port already records as AI invocations. If you also run other types of agents, such as Claude managed agents, Cursor cloud agents, AWS Bedrock or Azure AI Foundry agents, ingest their sessions and runs into Port too. Then adjust the data model and the workflows in this guide to include them, so one routine reviews all your agents, wherever they run.
Step 1: Set up the data model
Create the agent improvement blueprint
Every problem the routine finds becomes an improvement: the agent and evidence conversations, the failure pattern, the asset that causes it (tool, system_prompt, or skill), and the suggested fix. Each entity stays pending until a person applies or dismisses it, with an audit trail of when it was found, by which run, and who decided.
-
Go to the Data model page in Port.
-
Click + Blueprint.
-
Click on the
{...} Edit JSONbutton. -
Copy and paste the following JSON configuration:
Agent improvement blueprint (click to expand)
{"identifier": "agent_improvement","title": "Agent Improvement","icon": "Infinity","schema": {"properties": {"status": {"title": "Status","type": "string","enum": ["pending","applied","dismissed"],"enumColors": {"pending": "orange","applied": "green","dismissed": "lightGray"}},"asset_type": {"title": "Asset type","type": "string","enum": ["skill","tool","system_prompt"],"enumColors": {"skill": "purple","tool": "blue","system_prompt": "turquoise"}},"asset_identifier": {"title": "Asset identifier","type": "string"},"failure_pattern": {"title": "Failure pattern","type": "string"},"reasoning": {"title": "Reasoning","type": "string","format": "markdown"},"occurrences": {"title": "Occurrences","type": "number"},"found_at": {"title": "Found at","type": "string","format": "date-time"},"review_run_id": {"title": "Review run ID","type": "string"},"fix_summary": {"title": "Fix summary","type": "string","format": "markdown"},"suggested_fix": {"title": "Suggested fix","type": "object"},"applied_at": {"title": "Applied at","type": "string","format": "date-time"},"review_note": {"title": "Review note","type": "string"}},"required": ["status","asset_type","asset_identifier"]},"relations": {"agent": {"title": "Agent","target": "_ai_agent","required": true,"many": false},"conversations": {"title": "Evidence conversations","target": "_ai_conversation","required": false,"many": true}},"mirrorProperties": {},"calculationProperties": {},"aggregationProperties": {}} -
Click Create.
Add the ready flag to the AI agent blueprint
The routine uses two workflows:
- A dispatcher - runs every 24 hours, finds the agents that ran in the last 24 hours, and marks them ready for self-improvement.
- An agent self-improvement workflow - reviews one agent's conversations from the last 24 hours and suggests fixes.
A ready_for_self_improvement flag on the agent connects them. The dispatcher sets it to true, and the self-improvement workflow runs whenever it turns true, then sets it back to false when it finishes.
-
Go to the Data model page in Port.
-
Expand the AI Agent blueprint, click the
...button, and select Edit blueprint. -
Click on the
{...} Edit JSONbutton. -
Add the following property under
schema.properties:"ready_for_self_improvement": {"title": "Ready for self-improvement","type": "boolean","description": "Set to true by the self-improvement dispatcher when the agent ran in the last 24 hours. The self-improvement workflow runs when it turns true and resets it to false when it finishes."} -
Click Save.
Step 2: Create the self-improvement workflow
This workflow does the reviewing. Given an agent, it reads the agent's turns from the last 24 hours, finds the failures and wasted tool calls that repeat, traces each one to the tool, skill or prompt that causes it, and saves a pending improvement with a suggested fix.
It looks at the prompt, tools and skills together, so it can find problems that come from how they combine, not just from one of them. It only reads things every agent has, so the same workflow works for any agent in your registry.
It triggers when the ready flag turns true, then resets the flag when the run finishes. The webhook nodes call Port's API, so you don't need to add an API token: Port authenticates these requests with your organization's credentials.
To create the workflow:
-
Go to the Workflows page in Port.
-
Click + Workflow.
-
Fill out the Create new workflow form, then click Confirm.
-
Click the
{...}button to open the JSON editor. -
Copy and paste the following workflow JSON:
Agent self-improvement workflow JSON (click to expand)
{"identifier": "agent_self_improvement","title": "Agent self-improvement","icon": "Infinity","category": "Agent self-improvement","description": "When an agent is marked ready for self-improvement, reads its conversations from the last 24 hours, finds repeating failures and wasted tool calls, traces each one to the agent's prompt, tools or skills, and saves an improvement with a suggested fix.","allowAnyoneToViewRuns": true,"nodes": [{"identifier": "trigger","title": "On agent ready for self-improvement","config": {"type": "EVENT_TRIGGER","event": {"type": "ENTITY_UPDATED","blueprintIdentifier": "_ai_agent"},"condition": {"type": "JQ","expressions": [".diff.before.properties.ready_for_self_improvement != true",".diff.after.properties.ready_for_self_improvement == true"],"combinator": "and"}}},{"identifier": "fetch_index","title": "Find the agent's turns from the last 24 hours","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_invocations/entities/search","headers": {"Content-Type": "application/json"},"body": {"query": {"combinator": "and","rules": [{"relation": "agent","operator": "=","value": "{{ .outputs.trigger.diff.after.identifier }}"},{"property": "asked_at","operator": ">=","value": "{{ (now - 86400) | todateiso8601 }}"}]},"include": ["$identifier","asked_at"],"limit": 1000}},"variables": {"latest_ids": "{{ (.result.response.data.entities // []) | sort_by(.properties.asked_at) | reverse | .[0:20] | map(.identifier) }}","total": "{{ (.result.response.data.entities // []) | length }}"},"verbose": true},{"identifier": "fetch_conversations","title": "Group turns into conversations","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_invocations/entities/search","headers": {"Content-Type": "application/json"},"body": {"query": {"combinator": "and","rules": [{"property": "$identifier","operator": "in","value": "{{ .outputs.fetch_index.latest_ids }}"}]},"limit": 20}},"variables": {"conversations": "{{ (.result.response.data.entities // []) | map(. + {cid: (.relations.conversation | if type == \"object\" then .identifier else . end)}) | group_by(.cid) | map(sort_by(.properties.asked_at)) | sort_by(.[-1].properties.asked_at) | reverse | map(. as $turns | ($turns | map(((.properties.execution_logs // \"\") | ltrimstr(\"```json\") | rtrimstr(\"```\")) as $log | (if ($log | length) > 2 then ($log | fromjson | .messages // []) else [] end))) as $msgs | ([$msgs[][] | select(.role == \"assistant\") | (.toolCalls // [])[] | {name: .name, args: (.arguments | tojson)}]) as $calls | {conversation_id: $turns[0].cid, started_at: $turns[0].properties.asked_at, turn_count: ($turns | length), feedback: ($turns | map(select(.properties.feedback_rating != null) | {rating: .properties.feedback_rating, comment: .properties.feedback_comment})), tool_stats: {total_calls: ($calls | length), repeated_calls: (($calls | length) - ($calls | unique | length)), calls_by_tool: ($calls | group_by(.name) | map({key: .[0].name, value: length}) | from_entries) }, turns: ([range(0; $turns | length)] | .[-5:] | map(. as $i | {request: $turns[$i].properties.prompt, steps: ($msgs[$i] | map(if .role == \"assistant\" then ((.toolCalls // []) | map(\"CALL \" + .name + \" \" + (.arguments | tojson | .[0:150]))) elif .role == \"tool\" then [\" -> \" + ((.content | if type == \"string\" then .[0:300] else (tojson | .[0:300]) end) | gsub(\"\\\\s+\"; \" \") | .[0:160])] else [] end) | flatten), reply: (($turns[$i].properties.response // \"\") | .[0:1500])})) }) }}","count": "{{ (.result.response.data.entities // []) | map(.relations.conversation | if type == \"object\" then .identifier else . end) | unique | length }}","turns_fetched": "{{ (.result.response.data.entities // []) | length }}"},"verbose": true},{"identifier": "find_failures","title": "Find repeating failures and wasted tool calls","config": {"type": "AI","tools": [],"systemPrompt": "You are a reliability reviewer for AI agents. You read condensed transcripts of an agent's recent conversations and find problems that REPEAT across conversations. Each conversation has its turns in order (the user's request, the agent's tool calls with short results, and its reply), any user feedback, and tool_stats for the whole conversation.\n\nLook for two kinds of problems:\n\n1. Failures - the agent:\n- could not complete a step it was asked to do (e.g. it had no tool for it),\n- declined or skipped the task the user asked for, especially if the user pushed back in a later turn,\n- made a decision that the evidence in the same conversation contradicts, or flagged its own doubt about the outcome.\n\n2. Unnecessary tool calls - calls that didn't move the task forward:\n- the same call repeated with the same arguments (see tool_stats.repeated_calls),\n- calls whose result was never used (e.g. listing all blueprints, fetching an entity again right after fetching it),\n- trial-and-error retries after a malformed or too-narrow query,\n- searching for tools or data that don't exist.\nFor these, quote the numbers from tool_stats (total calls, repeats, the tools involved) and estimate how many calls per conversation were unnecessary.\n\nRules:\n- Only report a pattern that appears in at least 2 conversations.\n- Use only evidence from the transcripts. Quote short snippets.\n- Don't propose fixes yet; just describe what goes wrong.","userPrompt": "Agent under review: `{{ .outputs.trigger.diff.after.identifier }}`\n\nLatest {{ .outputs.fetch_conversations.count }} conversations (newest first):\n\n{{ .outputs.fetch_conversations.conversations | tojson }}\n\nReturn the repeating problems.","outputSchema": {"type": "object","properties": {"failures": {"type": "array","items": {"type": "object","properties": {"pattern": {"type": "string","description": "Short name of the failure"},"description": {"type": "string","description": "What goes wrong, with quoted evidence"},"occurrences": {"type": "number","description": "Number of conversations with this problem"},"conversation_ids": {"type": "array","items": {"type": "string"}}},"required": ["pattern","description","conversation_ids","occurrences"]}}},"required": ["failures"]}},"variables": {"failures": "{{ .result.response | if type == \"string\" then fromjson else . end | .failures }}"},"verbose": true},{"identifier": "fetch_agent","title": "Fetch agent configuration","config": {"type": "WEBHOOK","method": "GET","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_agent/entities/{{ .outputs.trigger.diff.after.identifier }}"},"variables": {"agent": "{{ .result.response.data.entity | {identifier, title, prompt: .properties.prompt, tools: .properties.tools, skills: (.relations.skills // [])} }}"},"verbose": true},{"identifier": "fetch_skills","title": "Fetch agent skills","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/skill/entities/search","headers": {"Content-Type": "application/json"},"body": {"query": {"combinator": "and","rules": [{"property": "$identifier","operator": "in","value": "{{ .outputs.fetch_agent.agent.skills }}"}]}}},"variables": {"skills": "{{ (.result.response.data.entities // []) | map({identifier, title, instructions: .properties.instructions}) }}"},"verbose": true},{"identifier": "trace_to_asset","title": "Trace problems to agent assets and suggest fixes","config": {"type": "AI","tools": [],"systemPrompt": "You find the root cause of an AI agent's repeating problems in its configuration, and propose the fix. The configuration has three kinds of assets:\n- `system_prompt`: the agent's prompt (asset_identifier = the agent identifier)\n- `tool`: the agent's allowed tools, a list of regex patterns matched against tool names (asset_identifier = the agent identifier). A missing write tool means a pattern that doesn't match it, e.g. `upsert_entity`.\n- `skill`: one of the agent's skills (asset_identifier = the skill identifier)\n\nRules:\n- Return at most ONE improvement per asset. If several problems point to the same asset, merge them into a single improvement.\n- When an asset has both a failure and unnecessary tool calls, lead with the failure in the title and the first part of the reasoning, then cover the tool-call waste.\n- Unnecessary tool calls usually trace to missing guidance: a skill that doesn't say which fields, filters or identifiers to use, or a prompt that doesn't say what the agent already knows.\n- Point to the exact line or setting responsible and quote it in `reasoning`.\n- Only use assets that appear in the configuration you are given.\n\nFor each improvement, also propose the smallest concrete change that fixes it:\n- `new_properties` holds only the property to change, with its full new value:\n - skill: {\"instructions\": \"<the full new skill text>\"}\n - tool: {\"tools\": [<the full new list of regex patterns>]} - add the narrowest pattern that fixes the problem, e.g. \"^upsert_entity$\"\n - system_prompt: {\"prompt\": \"<the full new prompt>\"}\n- Keep everything you don't need to change exactly as it is.\n- `fix_summary` is a short markdown explanation for the human approver: what changes (quote removed and added lines) and why.","userPrompt": "Agent configuration:\n{{ .outputs.fetch_agent.agent | tojson }}\n\nSkills:\n{{ .outputs.fetch_skills.skills | tojson }}\n\nRepeating problems found in the latest conversations:\n{{ .outputs.find_failures.failures | tojson }}\n\nFor each root cause, return an improvement with its suggested fix.","outputSchema": {"type": "object","properties": {"improvements": {"type": "array","items": {"type": "object","properties": {"title": {"type": "string","description": "Imperative, e.g. 'Let the agent update Jira issues'"},"asset_type": {"type": "string","enum": ["skill","tool","system_prompt"]},"asset_identifier": {"type": "string"},"failure_pattern": {"type": "string"},"reasoning": {"type": "string"},"occurrences": {"type": "number"},"conversation_ids": {"type": "array","items": {"type": "string"}},"fix_summary": {"type": "string"},"new_properties": {"type": "object","description": "Only the property to change with its full new value: {instructions} for a skill, {tools} for a tool, {prompt} for a system prompt"}},"required": ["title","asset_type","asset_identifier","failure_pattern","reasoning","conversation_ids","occurrences","fix_summary","new_properties"]}}},"required": ["improvements"]}},"variables": {"improvements": "{{ .result.response | if type == \"string\" then fromjson else . end | .improvements }}"},"verbose": true},{"identifier": "has_improvements","title": "Any improvements?","config": {"type": "CONDITION","outlets": [{"identifier": "found","title": "Improvements found","expression": "((.outputs.trace_to_asset.improvements // []) | length) > 0"}]}},{"identifier": "create_improvements","title": "Save improvements","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/agent_improvement/entities/bulk?upsert=true&merge=true","headers": {"Content-Type": "application/json"},"body": {"entities": "{{ .outputs.trigger.diff.after.identifier as $agent | .workflowRun.identifier as $run | (now | todateiso8601) as $now | .outputs.trace_to_asset.improvements | map({identifier: ($agent + \"-\" + .asset_type + \"-\" + (.asset_identifier | ascii_downcase | gsub(\"[^a-z0-9]+\"; \"-\")) + \"-\" + ($run | sub(\"^wfr_\"; \"\"))), title: (.title // .failure_pattern), properties: {status: \"pending\", asset_type: .asset_type, asset_identifier: .asset_identifier, failure_pattern: .failure_pattern, reasoning: .reasoning, occurrences: .occurrences, found_at: $now, review_run_id: $run, fix_summary: .fix_summary, suggested_fix: {blueprint: (if .asset_type == \"skill\" then \"skill\" else \"_ai_agent\" end), identifier: .asset_identifier, properties: .new_properties } }, relations: {agent: $agent, conversations: .conversation_ids } }) }}"}},"variables": {},"verbose": true},{"identifier": "reset_flag","title": "Reset the ready flag","config": {"type": "WEBHOOK","method": "PATCH","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_agent/entities/{{ .outputs.trigger.diff.after.identifier }}","headers": {"Content-Type": "application/json"},"body": {"properties": {"ready_for_self_improvement": false}}},"variables": {},"verbose": true}],"connections": [{"sourceIdentifier": "trigger","targetIdentifier": "fetch_index"},{"sourceIdentifier": "fetch_index","targetIdentifier": "fetch_conversations"},{"sourceIdentifier": "fetch_conversations","targetIdentifier": "find_failures"},{"sourceIdentifier": "find_failures","targetIdentifier": "fetch_agent"},{"sourceIdentifier": "fetch_agent","targetIdentifier": "fetch_skills"},{"sourceIdentifier": "fetch_skills","targetIdentifier": "trace_to_asset"},{"sourceIdentifier": "trace_to_asset","targetIdentifier": "has_improvements"},{"sourceIdentifier": "has_improvements","targetIdentifier": "create_improvements","sourceOutletIdentifier": "found"},{"sourceIdentifier": "has_improvements","targetIdentifier": "reset_flag","fallback": true},{"sourceIdentifier": "create_improvements","targetIdentifier": "reset_flag"}]} -
Click Apply changes.
fetch_index keeps the 20 newest turns from the last 24 hours, so the AI nodes get a predictable amount of context. To review more turns, change .[0:20] in its latest_ids variable and the limit of fetch_conversations together.
Step 3: Create the dispatcher workflow
The dispatcher runs every 24 hours, decides which agents need a review, and sets their ready_for_self_improvement flag to true. Each of those agents then gets its own self-improvement run, with its own conversations, configuration and suggested fixes.
The dispatcher fetches the AI invocations from the last 24 hours. Each invocation is one turn, and belongs to a conversation through its conversation relation. The dispatcher only marks an agent ready when its invocations belong to at least 2 different conversations, because the review only reports problems that repeat across conversations.
It runs on a daily schedule at 06:00 UTC (with a Run now action for admins), finds those agents, resets their flags, then marks them ready so each gets its own self-improvement run.
The self-improvement workflow only runs when the ready flag changes to true. If a previous run failed before it reset the flag, setting it to true again would not start a new run. That's why reset_flags clears the flag first, so every listed agent is reviewed every day.
To create the workflow:
-
Go to the Workflows page in Port.
-
Click + Workflow.
-
Fill out the Create new workflow form, then click Confirm.
-
Click the
{...}button to open the JSON editor. -
Copy and paste the following workflow JSON:
Agent self-improvement dispatcher workflow JSON (click to expand)
{"identifier": "agent_self_improvement_dispatcher","title": "Agent self-improvement dispatcher","icon": "Infinity","category": "Agent self-improvement","description": "Every 24 hours, finds the AI agents that had at least 2 conversations in the last 24 hours and marks them ready for self-improvement.","allowAnyoneToViewRuns": true,"nodes": [{"identifier": "trigger","title": "Every 24 hours","config": {"type": "SCHEDULE_TRIGGER","cron": "0 6 * * *"}},{"identifier": "manual_trigger","title": "Run now","config": {"type": "SELF_SERVE_TRIGGER","permissions": {"roles": ["Admin"]},"userInputs": {"properties": {},"required": []},"published": true}},{"identifier": "find_active_agents","title": "Find agents that ran in the last 24 hours","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_invocations/entities/search","headers": {"Content-Type": "application/json"},"body": {"query": {"combinator": "and","rules": [{"property": "asked_at","operator": ">=","value": "{{ (now - 86400) | todateiso8601 }}"}]},"include": ["$identifier","agent","conversation"],"limit": 1000}},"variables": {"agents": "{{ (.result.response.data.entities // []) | map({agent: (.relations.agent | if type == \"object\" then .identifier else . end), conversation: (.relations.conversation | if type == \"object\" then .identifier else . end)}) | map(select(.agent != null)) | group_by(.agent) | map(select((map(.conversation) | unique | length) >= 2)) | map(.[0].agent) | .[0:20] }}","invocations": "{{ (.result.response.data.entities // []) | length }}"},"verbose": true},{"identifier": "has_agents","title": "Any agents to review?","config": {"type": "CONDITION","outlets": [{"identifier": "found","title": "Agents found","expression": "(.outputs.find_active_agents.agents | length) > 0"}]}},{"identifier": "reset_flags","title": "Clear the ready flag","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_agent/entities/bulk?upsert=true&merge=true","headers": {"Content-Type": "application/json"},"body": {"entities": "{{ .outputs.find_active_agents.agents | map({identifier: ., properties: {ready_for_self_improvement: false} }) }}"}},"variables": {},"verbose": true},{"identifier": "mark_ready","title": "Mark agents ready for self-improvement","config": {"type": "WEBHOOK","method": "POST","synchronized": true,"url": "https://api.port.io/v1/blueprints/_ai_agent/entities/bulk?upsert=true&merge=true","headers": {"Content-Type": "application/json"},"body": {"entities": "{{ .outputs.find_active_agents.agents | map({identifier: ., properties: {ready_for_self_improvement: true} }) }}"}},"variables": {},"verbose": true}],"connections": [{"sourceIdentifier": "trigger","targetIdentifier": "find_active_agents"},{"sourceIdentifier": "manual_trigger","targetIdentifier": "find_active_agents"},{"sourceIdentifier": "find_active_agents","targetIdentifier": "has_agents"},{"sourceIdentifier": "has_agents","targetIdentifier": "reset_flags","sourceOutletIdentifier": "found"},{"sourceIdentifier": "reset_flags","targetIdentifier": "mark_ready"}]} -
Click Apply changes.
To run it at a different time, change the cron expression of the trigger. See Schedule trigger for the syntax. Keep the schedule at once a day, since both workflows only read the last 24 hours.
Port's bulk entity API accepts up to 20 entities per request, so the dispatcher marks at most 20 agents per run. If more than 20 of your agents run every day, duplicate reset_flags and mark_ready for the next batch, using .agents[20:40] instead of .agents.
Step 4: Add apply and dismiss actions
Every suggested change waits for a person. This workflow adds two actions to every pending improvement. Apply improvement writes the suggested fix to the agent or skill, and Dismiss improvement rejects it. Only admins can run them, and every decision is logged as a workflow run, so you can always see who applied what and when.
The workflow has two branches, one for each action. The apply branch reads the improvement, patches the agent or skill with the suggested fix, and marks it applied. The dismiss branch marks it dismissed. Both save the reviewer's comment as Review note.
To create the workflow:
-
Go to the Workflows page in Port.
-
Click + Workflow.
-
Fill out the Create new workflow form, then click Confirm.
-
Click the
{...}button to open the JSON editor. -
Copy and paste the following workflow JSON:
Apply or dismiss agent improvements workflow JSON (click to expand)
{"identifier": "apply_agent_improvement","title": "Apply or dismiss agent improvements","icon": "Infinity","category": "Agent self-improvement","description": "Two actions on a pending agent improvement: Apply writes the suggested fix to the agent or skill; Dismiss rejects it.","allowAnyoneToViewRuns": true,"nodes": [{"identifier": "apply","config": {"type": "SELF_SERVE_TRIGGER","permissions": {"roles": ["Admin"]},"executeActionButtonText": "Apply","contexts": [{"on": "ENTITY","userInput": "improvement"}],"userInputs": {"properties": {"improvement": {"type": "string","format": "entity","blueprint": "agent_improvement","title": "Improvement","dataset": {"combinator": "and","rules": [{"property": "status","operator": "=","value": "pending"}]}},"comment": {"type": "string","title": "Comment"}},"required": ["improvement"],"order": ["improvement","comment"]}},"variables": {},"verbose": false,"title": "Apply improvement"},{"identifier": "dismiss","config": {"type": "SELF_SERVE_TRIGGER","permissions": {"roles": ["Admin"]},"executeActionButtonText": "Dismiss","variant": "ALERT","contexts": [{"on": "ENTITY","userInput": "improvement"}],"userInputs": {"properties": {"improvement": {"type": "string","format": "entity","blueprint": "agent_improvement","title": "Improvement","dataset": {"combinator": "and","rules": [{"property": "status","operator": "=","value": "pending"}]}},"reason": {"type": "string","title": "Reason"}},"required": ["improvement"],"order": ["improvement","reason"]}},"variables": {},"verbose": false,"title": "Dismiss improvement"},{"identifier": "fetch_improvement","title": "Fetch the improvement","config": {"type": "WEBHOOK","method": "GET","synchronized": true,"url": "https://api.port.io/v1/blueprints/agent_improvement/entities/{{ .outputs.trigger.improvement | if type == \"object\" then .identifier else . end }}"},"variables": {"improvement": "{{ .result.response.data.entity | {identifier, title, status: .properties.status, fix: .properties.suggested_fix} }}"},"verbose": true},{"identifier": "apply_fix","title": "Apply the fix to the agent or skill","config": {"type": "WEBHOOK","method": "PATCH","synchronized": true,"url": "https://api.port.io/v1/blueprints/{{ .outputs.fetch_improvement.improvement.fix.blueprint }}/entities/{{ .outputs.fetch_improvement.improvement.fix.identifier }}","headers": {"Content-Type": "application/json"},"body": {"properties": "{{ .outputs.fetch_improvement.improvement.fix.properties }}"}},"variables": {},"verbose": true},{"identifier": "mark_applied","title": "Mark as applied","config": {"type": "UPSERT_ENTITY","blueprintIdentifier": "agent_improvement","mapping": {"identifier": "{{ .outputs.fetch_improvement.improvement.identifier }}","properties": {"status": "applied","applied_at": "{{ now | todateiso8601 }}","review_note": "{{ .outputs.trigger.comment // \"\" }}"}}},"variables": {},"verbose": false},{"identifier": "mark_dismissed","title": "Mark as dismissed","config": {"type": "UPSERT_ENTITY","blueprintIdentifier": "agent_improvement","mapping": {"identifier": "{{ .outputs.trigger.improvement | if type == \"object\" then .identifier else . end }}","properties": {"status": "dismissed","review_note": "{{ .outputs.trigger.reason // \"\" }}"}}},"variables": {},"verbose": false}],"connections": [{"sourceIdentifier": "apply","targetIdentifier": "fetch_improvement"},{"sourceIdentifier": "fetch_improvement","targetIdentifier": "apply_fix"},{"sourceIdentifier": "apply_fix","targetIdentifier": "mark_applied"},{"sourceIdentifier": "dismiss","targetIdentifier": "mark_dismissed"}]} -
Click Apply changes.
apply_fix patches the agent or skill entity in Port directly. If your skills live in a Git repository, change the apply branch for skill fixes to open a pull request in that repository instead, so the change goes through your normal code review. See Ship new skills through one golden path for an example.
Step 5: Build the dashboard
The dashboard is where cleaning up after agents turns into reviewing fixes. It shows what is waiting for a decision, where the problems come from, and how the findings change over time.
- Navigate to your catalog page in Port.
- Click the + button in the left sidebar and select New dashboard.
- Name the dashboard Agent self-improvement, and select the
Infinityicon. - Click Create.
Now add the following widgets:
Improvements by status (click to expand)
- Click + Widget and select Pie chart.
- Set Title to
Improvements by status. - Set Blueprint to
Agent Improvement. - Set Breakdown by property to
Status. - Click Save.
Pending improvements (click to expand)
-
Click + Widget and select Table.
-
Set Title to
Pending improvements. -
Set Blueprint to
Agent Improvement. -
Under Initial filters, use the following filter:
{"combinator": "and","rules": [{"property": "status","operator": "=","value": "pending"}]} -
Click Save.
-
Click the Open widget button on the widget.
-
Click Manage Properties, and show these columns: Title, Asset type, Asset identifier, Occurrences, Fix summary, Reasoning, Agent and Found at.
-
Sort the table by Found at, newest first.
-
Click Save view.
Improvements suggested over time, by asset type (click to expand)
- Click + Widget and select Line chart.
- Set Title to
Improvements suggested over time, by asset type. - Set X axis:
- Title:
Date. - Time interval:
Day. - Time range:
In the past 30 days.
- Title:
- Set Y axis title:
Improvements. - Click + Line and configure:
- Title:
Improvements. - Chart type:
Count entities. - Blueprint:
Agent Improvement. - Function:
Count. - Measure time by:
Found at. - Breakdown by property:
Asset type.
- Title:
- Click Save.
Improvements by asset type (click to expand)
- Click + Widget and select Pie chart.
- Set Title to
Improvements by asset type. - Set Blueprint to
Agent Improvement. - Set Breakdown by property to
Asset type. - Click Save.
Recent suggested improvements (click to expand)
- Click + Widget and select Table.
- Set Title to
Recent suggested improvements. - Set Blueprint to
Agent Improvement. - Click Save.
- Click the Open widget button on the widget.
- Click Manage Properties, and show these columns: Found at, Title, Status, Asset type, Asset identifier, Occurrences, Fix summary and Applied at.
- Sort the table by Found at, newest first.
- Click Save view.
Let's test it
You don't need to wait for the schedule to see the routine work. Run the dispatcher now, so you test the whole chain:
- Go to the Self-service page, and run Run now from the Agent self-improvement dispatcher workflow.
- Go to the Workflows page, open Agent self-improvement, and follow the runs the dispatcher started, one per agent. Each takes a few minutes.
- When a run finishes, its agent's Ready for self-improvement flag is back to
false, and the Agent self-improvement dashboard shows the new pending improvements, if the run found any.
To review a single agent instead, open its page, edit the entity, set Ready for self-improvement to true, and save. The run only starts when the flag changes to true, so if it's already true, set it to false and save first.
Each run reads the conversations from the last 24 hours. If you apply a fix and run the review again on the same day, it reads the same conversations from before the fix, and may suggest changing the fix you just applied. On the daily schedule the review windows don't overlap, so this mostly happens while testing. To avoid it, skip conversations that started before the agent's latest Applied at, and pass the agent's applied improvements to trace_to_asset.
Then decide on each improvement:
- On the dashboard, click a pending improvement to open its page, and read its Fix summary and Reasoning.
- Click the ⚡ icon at the top right, and choose Apply improvement if you agree, or Dismiss improvement if you don't. If a fix is only partly right, dismiss it and make the right change by hand in the agent or the skill.
- Run the agent again on a request it failed before.
This is the loop: the next day, the dispatcher marks the agent ready again, and the next run reviews the new conversations and suggests what to improve next.
Take it to production
The routine you built reviews every active agent in your registry once a day. Before you rely on it, think through these questions:
- Repeats - how do you stop the routine from suggesting an improvement again when it's already pending, or was dismissed? One option is to pass the agent's open and dismissed improvements to
trace_to_asset, and tell it to skip them. - Stale or conflicting fixes - what if a newer fix to the same skill or agent was applied before someone approves this one?
suggested_fixholds the full new value of the property, so applying an older fix overwrites the newer one. - Shared skills - what if a fix targets a skill that other agents use too? Use the dependency graph to show the approver every agent that loads the skill, and route the approval to the skill's owner.
- Revising a fix - a suggested fix is sometimes only partly right: it points at the right asset but gets part of the change wrong. Consider adding a Revise improvement action, where a person describes in plain words what's wrong with the fix, an AI node rewrites
suggested_fixandfix_summaryto match, and the improvement stayspendinguntil someone applies it. - Improving the improver - the comments and dismiss reasons saved as Review note are valuable feedback. Pass them to the next review run, so it learns which kinds of fixes your team accepts.
Because every conversation and every improvement is data in Port, you can also track each agent's quality and token spend over time, and prove the fixes worked. See ROI measurement for the metrics to start with.
Related pages
- Self-improvement routines - the concepts behind this routine, and other routines you can build on the same pattern.
- AI invocations - the record of every Port AI agent interaction that this routine reviews.
- Schedule trigger - running a workflow on a cron schedule.
- Event trigger - running a workflow when an entity changes.
- AI node - configuring
systemPrompt,toolsandoutputSchema. - Condition node - branching a workflow on an expression.