Monitoring AI agent runs for empty or bad output

A traditional job fails loudly: an exception, a non-zero exit code, a red execution. An AI agent usually fails politely. The model call returns 200 OK, the workflow finishes, and the result is an empty string, half a JSON object, or "I'm sorry, I can't help with that." Everything downstream treats it as a valid result. The summary email goes out blank and the CRM field gets an apology.

This guide covers the ways agent runs go wrong without erroring, and how to check every run's output automatically.

How agent runs fail while "succeeding"

• Empty output. A content filter blocks the response, a tool call returns nothing and the agent has nothing to say, or a parsing step silently produces "" or [].
• Truncated output. The model hits its output token limit mid-answer. Most LLM APIs report this in a stop or finish reason, but few workflows check it, and the result is JSON that ends halfway through an object.
• Wrapped or chatty output. You asked for JSON and got JSON inside a Markdown code fence, or a sentence of preamble before the object. Valid to a human, invalid to JSON.parse.
• Missing fields. The JSON parses, but summary is empty or items is missing, often after a prompt or model change.
• Refusals and error text. "As an AI language model…", "I cannot access that URL", or an upstream error message passed through as the answer.
• Runs that never finish. An agent loops on tool calls until a timeout kills the process, and nothing reports back.
• Runs that never start. The schedule or trigger stopped, the same as any other automation.

The first five are content problems: the run happened and the output is bad. The last two are liveness problems. You need checks for both.

Step 1: create a monitor with output checks

In Flowhush, click New monitor and set Expected every (seconds) and Grace period (seconds) to match how often the agent runs. Then, under Output checks (Pro and Team plans):

How the rules behave, straight from the implementation:

• A required key must exist and be non-empty. null, "", [] and {} all fail. 0 and false pass.
• Dot paths walk objects and arrays: items.0.id means "the id of the first element of items".
• Forbidden phrases are matched as case-insensitive substrings of the whole body. Pick them carefully: error would also match a key called error_count or the text "no errors found".
• The body is read up to 64 KB for validation. Send the result, not a huge transcript, because a truncated body won't parse as JSON.

When a success ping's body fails a rule, Flowhush records the ping as a failure, the monitor goes down, and you get one alert with the reason, e.g. Missing or empty JSON key: summary. The HTTP response is still 200, with the body OK (recorded as failure: …), so your workflow doesn't break because of the check.

Step 2: send the agent's output after every run

n8n (AI Agent node or any LLM node)

Add an HTTP Request node after the node that produces the final answer:

Method: POST
URL: https://flowhush.com/p/YOUR-MONITOR-UUID
Send Body: on
Body Content Type: JSON
Specify Body: Using JSON
JSON (expression): {{ JSON.stringify($json) }}
Settings → Execute Once: on
Settings → On Error: Continue

The n8n AI Agent node returns its answer in a field called output. If the answer is plain text, require the key output. If you use a structured output parser, require the nested fields instead, e.g. output.summary. If the run handles several items, turn on Execute Once so you send one ping per run. Otherwise you'll get one ping per item.

Python

import json, urllib.request

PING_URL = "https://flowhush.com/p/YOUR-MONITOR-UUID"

def report(result=None, error=None):
    """Send the agent's final output (or an error) to Flowhush. Never raises."""
    url = PING_URL + ("/fail" if error else "")
    body = json.dumps({"error": str(error)} if error else result).encode()
    req = urllib.request.Request(url, data=body, method="POST",
                                 headers={"Content-Type": "application/json"})
    try:
        urllib.request.urlopen(req, timeout=10).read()
    except Exception:
        pass  # monitoring must never break the agent

try:
    result = run_agent()        # your code; returns a dict
    report(result)
except Exception as e:
    report(error=e)
    raise

Shell or any other runtime

curl -fsS -m 10 --retry 3 -X POST -H 'Content-Type: application/json' \
     --data-binary @result.json https://flowhush.com/p/YOUR-MONITOR-UUID

Step 3: catch runs that hang

For agents that can loop or wait on slow tools, ping /start when the run begins:

curl -fsS -m 10 https://flowhush.com/p/YOUR-MONITOR-UUID/start

If the completion ping doesn't arrive within the grace period, the monitor goes down with "Run started but never reported completion". Set the grace period above your slowest normal run. Flowhush records the duration of each completed run, which shows you how long runs normally take.

Step 4: check that the alerts work

Send a deliberately bad body:

curl -sS -X POST -H 'Content-Type: application/json' \
     -d '{"summary": ""}' https://flowhush.com/p/YOUR-MONITOR-UUID
# OK (recorded as failure: Missing or empty JSON key: summary)

You should get a down alert on your channel. A good body afterwards sends the recovery alert:

curl -sS -X POST -H 'Content-Type: application/json' \
     -d '{"summary": "3 new tickets triaged", "items": [{"id": 1}]}' \
     https://flowhush.com/p/YOUR-MONITOR-UUID

What this doesn't replace

These are structural checks. They can tell you the answer is missing, malformed, or contains a phrase you've banned. They can't tell you a well-formed answer is wrong. For quality, keep evaluating samples of real outputs, check the finish reason in your own code, and validate business rules (amounts, IDs, dates) before acting on them. And don't send secrets or sensitive personal data in ping bodies. Send the fields you need checked.

Summary

• Liveness: one ping per run, alerting when runs stop arriving or hang after /start.
• Shape: non-empty, valid JSON and required keys on Pro.
• Content: forbidden phrases such as refusals on Team.
• Alerts by email, Slack, Discord or Telegram: one per incident, plus a recovery notice.

Heartbeat monitoring is free for up to 5 agents. Output checks start on Pro at $12/month.