JOÃO NEVES
CEO · CTO · AI BUILDER
PRODUCTION AI AGENTS / REAL-WORLD CASE STUDY
I built AI agents that run a sports newsroom.
They collect the news.
Decide which stories belong together.
Decide what's worth covering.
Find the evidence.
Plan the article.
Write it.
Verify what they wrote against the original sources.
Correct what can safely be corrected.
Reject what can't.
And prepare the result for human review.
That last part is more important than everything before it.
Because making an AI write a convincing article is easy.
Making it know when the evidence isn't good enough to publish one isn't.
Sports happens to be the use case.
The engineering problem isn't sports.
It's making AI perform multi-step work where the inputs change, decisions have consequences, mistakes need to be detectable, and sometimes the correct output is nothing at all.
That applies to almost every industry where people repeatedly collect information, interpret it and make decisions.
Sports Terminal is the case study. Production AI systems are the capability.
What production AI agents actually need.
Most AI agent demos prove that a model can complete a task.
Production AI agents have a harder requirement.
They need:
- orchestration
- durable state
- evidence
- guardrails
- structured outputs
- observability
- cost controls
- concurrency protection
- bounded recovery
- clear limits on what the model is allowed to decide
- human review where consequences matter
I built all of that into an autonomous editorial system.
Sports happens to be the use case.
The engineering applies far beyond sports.
The short version.
Sports Terminal is an autonomous editorial system I designed for sports media.
An individual agent can be configured around:
But an "agent" isn't a chatbot thinking indefinitely.
Each run moves through specialized stages for:
collection → clustering → planning → evidence grounding → writing → verification → correction → SEO → internal linking → quality control → queueing
Those stages communicate through typed contracts and durable state.
Not unconstrained model-to-model conversation.
AI performs work where probabilistic reasoning is useful.
Code owns rules that cannot be negotiable.
The problem wasn't writing.
A sports newsroom repeats the same process constantly:
Find stories.
Determine whether they matter.
Find additional sources.
Work out whether the sources describe the same event.
Choose an angle.
Research it.
Write it.
Check the facts.
Prepare SEO.
Add links.
Choose media.
Review.
Publish.
Then start again.
Sports makes this harder because the information never stops.
The original question was:
How much of this workflow could become software without giving the software permission to invent reality?
Then one article
broke the illusion.
One generated article came from a cluster containing stories that did not belong together.
The article was fluent.
It looked correct.
It wasn't.
It:
- mixed unrelated subjects
- promoted contextual people into primary subjects
- introduced unsupported information
- reused stale information
- selected an irrelevant image
The obvious reaction would have been:
fix the prompt.
I didn't.
Because the prompt wasn't the real problem.
It was a systems problem.
The incident exposed weaknesses across the entire pipeline:
- weak story clustering
- size-biased relevance scoring
- a broken prompt-template contract capable of leaving planning without source data
- insufficient source context
- no durable evidence model
- no reliable distinction between primary and contextual entities
- source-free factual plausibility scoring
- image selection based too heavily on source order
- no safe bounded correction loop
- insufficient version/concurrency protection between verification, editing and approval
So I stopped treating this as a prompting problem.
I redesigned it as an evidence-control problem.
I designed and built the system end to end.
My responsibilities included:
I wasn't implementing a predefined AI specification.
I was translating editorial requirements into enforceable technical contracts.
For example:
Do not publish unsupported factual claims.
became:
- source identities
- evidence-unit IDs
- paragraph IDs
- source hashes
- claim coverage
- quote validation
- content hashes
- versioned state transitions
- concurrency protection
What makes an AI agent production-ready?
Can you answer:
What data did it use?
Why did it make that decision?
Which version of the source did it see?
What was the model allowed to decide?
What was code responsible for?
What happens if a source changes?
What happens if a model returns malformed output?
How much is one run allowed to spend?
Can two runs overlap?
Can a stale verification result authorize newer content?
Can a human reconstruct what happened later?
Can the system safely say:
there isn't enough evidence to continue?
Those questions matter more in production than whether the model can produce impressive text.
AI proposes.
Code decides.
Models are useful for:
- semantic grouping
- interpretation
- editorial reasoning
- claim extraction
- evidence relevance
- writing
They should not independently own:
- source identity
- evidence identity
- budgets
- scope
- state transitions
- version validity
- concurrency
- whether malformed output should be accepted
- publishing authority
Whenever possible
AI proposes a structured decision.
Application code validates whether that decision is allowed to exist.
Underneath the agents is a fairly boring stack.
And that's intentional.
C# and PostgreSQL own durable workflow state and deterministic enforcement.
Hangfire runs scheduled and long-running work.
Models perform bounded semantic work.
Every important decision becomes observable state.
Humans retain final publishing authority.
The architecture in ten seconds
A controlled path from source to publication.
Every stage can pass work forward, correct it within limits, or stop it safely.
One system. Different autonomous newsrooms.
A Sports Terminal agent is a configurable autonomous editorial instance scoped to:
- sport
- competition
- league
- language
- country
- topic
- publishing identity
- cadence
- source policy
- quality threshold
- per-stage model configuration
Examples can include:
NFL / English / United States
Football / Portuguese / Portugal
Formula 1 / English / Global
The workflow remains controlled while scope and editorial identity change.
The entire system, stage by stage.
For anyone who wants the technical version.
View the full production architecture
| Stage | Input | AI Responsibility | Deterministic Safeguards | Durable Output | Failure / Repair |
|---|---|---|---|---|---|
| Scheduling | Agent configuration + schedule | None | Atomic claim, overlap prevention, validated config, retries disabled on main orchestration | Run + configuration snapshot | Fail before collection or LLM spend |
| Collection | Resolved source policy | None | Provider quotas, bounded collection, exact filter resolution | Stored feed items + content-delivery classification | Typed provider/collection outcome |
| Deduplication | Current + recent historical items | None | Normalized title/publisher fingerprint, historical and within-run comparison | Unique candidates | Deterministic exclusion |
| Clustering | Current anchors + bounded supplemental context | Group stories describing the same event/development | Anchor accounting, membership validation, duplicate prevention, event identity checks | Validated clusters | Bounded recovery + deterministic singleton fallback |
| Relevance / Scope | Clusters | Tie-break only when deterministic evidence cannot settle scope | Taxonomy, coherence, recency, trend signals, duplicate-story history | Ranked viable candidates | Reject/out-of-scope/backfill |
| Source Selection | Cluster | None or bounded ranking assistance if already implemented | Single authoritative source-selection component, recency, URL/publisher dedup, source count limits | Exact source set used downstream | Fail when selection cannot satisfy policy |
| Evidence Grounding | Exact selected sources | Select relevant paragraph IDs | Application maps IDs back to exact stored text, coverage validation, hashes, budgets | Evidence units + coverage manifest | One bounded repair or fail closed |
| Editorial Planning | Grounded evidence | Select angle, evidence IDs, primary entities, article type and length | ID validation, source minimums, entity-role rules | Validated editorial plan | One repair or typed refusal |
| Writing | Frozen validated evidence | Generate article | Evidence budgets, unknowns, forbidden assumptions, attribution policy, HTML validation | Canonical article HTML | One format-only retry, otherwise reject |
| Verification | Article + complete retained supporting sources | Extract/classify claims and propose findings | Complete claim-evidence matrix, source hashes, quote validation, deterministic blocking rules | Version-bound verifier result | Block / incomplete / unverified |
| Correction | Persisted blocking findings + exact article version | Propose surgical HTML patches | Server-owned targets, evidence IDs, one-to-one patches, optimistic concurrency | Corrected article + audit record | Reject / no progress / stale / unsafe |
| Quality Gate | Complete final article | Readability, SEO, originality, editorial quality | Full article, prompt preflight, factual gate remains separate | Version-bound quality result | Below threshold |
| Queueing | Verifier + quality results | None | Content hash, revision, run ID, compare-and-swap transition | pending_review | Bounded re-evaluation or safe terminal state |
| Human Approval | Exact queued article version | None | Reservations, author permissions, allowlists, concurrency checks | Final publishing decision | Rejected / stale / unauthorized |
The writer never gets to decide what the evidence was.
Each selected source is normalized.
The source is divided into numbered paragraphs.
The grounding model selects paragraph IDs.
It doesn't reproduce arbitrary text as a new source of truth.
Application code maps the IDs back to stored source text.
Unknown IDs fail.
Duplicate or omitted IDs fail validation.
Each normalized source has a hash.
Coverage is persisted.
If the source changes before authoritative verification, the current source hash no longer matches the grounding hash.
Verification stops before model spend.
The system cannot quietly verify the article against a different source version than the writer saw.
A person mentioned for context must not silently become the article's primary subject.
The evidence model distinguishes:
PRIMARY
and:
CONTEXTUAL
The planner explicitly selects primary entity keys.
Application code validates them.
The writer receives only permitted primary subjects.
Editorial policy became data.
Fluent isn't the same as true.
A general LLM quality score is not factual verification.
The verifier:
- re-fetches complete supporting source text
- validates the source version
- deterministically segments the article
- extracts atomic claims
- evaluates a bounded claim-to-evidence matrix
- requires complete coverage
Missing coverage produces:
INCOMPLETE
not:
PASS
The model may propose a finding.
Code decides whether it can block.
A blocking factual finding requires:
- the sentence actually exists
- the finding type is allowed to block
- the issue is material
- confidence is high enough
- the cited source quote actually exists
A hallucinating verifier cannot automatically block a clean article.
I didn't build an infinite AI-fixes-AI loop.
The corrector does not rewrite the entire article.
It returns structured replacement patches.
Patches must:
- target valid server-owned blocks
- reference valid finding keys
- reference valid evidence IDs
- return the expected HTML element type
Only approved blocks may change.
No-progress patches fail.
Concurrent edits make the correction stale.
Every corrected article is always re-verified.
Correction attempts are capped.
Sometimes the correct result is to stop.
The article that passed is the article that gets queued.
Verification and quality results are bound to a content hash/version.
Before queueing:
- reload article
- compare hash/revision
- reject stale evaluation
Queueing uses compare-and-swap bindings involving:
- content hash
- revision
- run ID
- updated timestamp
Only the exact evaluated version reaches:
pending_review
Human review remains final.
Guardrails shouldn't live only in prompts.
Critical guardrails exist in application code and durable state.
Examples:
Prompts guide model behaviour.
Guardrails determine what the system is actually allowed to do.
Images are evidence too.
Only supporting sources may supply images.
Excluded or contextual-only sources are ineligible.
Eligible images are ranked using:
- evidence support
- primary entity overlap
- source order
- deterministic candidate order
Images are copied to trusted storage.
Final HTML allows only trusted copied URLs.
Important limitation:
The system guarantees the image comes from a supporting source.
It does NOT claim vision-level confirmation that the image visually depicts the primary subject.
Prompts are production dependencies.
Prompts live in the database.
Application code owns:
- schemas
- model settings
- allowed placeholders
- required placeholders
- renderer behavior
Unknown placeholders or unresolved variables are not sent to the model.
Every LLM call is traceable to its exact prompt version, rendered fingerprint, schema, provider/model, renderer version, token usage and estimated cost.
View the complete LLM call record
Reruns:
- pin exact prompt versions
- use frozen writer inputs
- create isolated shadow articles
- do not mutate canonical output automatically
If an agent made a decision, I wanted to know why.
Every stage records its duration, model, prompt fingerprint, token usage, cost, fallbacks, scoped entity IDs and diagnostics.
Distinguish:
- nothing publishable
- editorial rejection
- infrastructure failure
- cancellation
- supersession
- successful queueing
0 articles generated
does not automatically mean:
the system failed
It may mean:
the system correctly found nothing safe to publish
Autonomous doesn't mean unlimited.
Bound:
Failed orchestration must not silently duplicate expensive work.
Costs are aggregated per actual model call.
Production engineering
The controls behind reliable autonomy.
Security
Security
Collected source content is untrusted internet input.
Treat it as data.
Not instructions.
Source text is delimiter-neutralized before prompt inclusion.
Final HTML is sanitized.
External images are copied to trusted storage.
Do not claim perfect prompt-injection security.
Use grounded language.
Testing
I tested the rules, not just whether the demo worked.
Invariant-driven testing across:
A demo proves the system worked once.
Invariants define what it is allowed to do every time.
Rollout & measurement
Measure first. Enforce second.
Shadow evaluation lets the new architecture make decisions without controlling production.
Only after the evidence supports it does the new path become authoritative.
What changed structurally.
- evidence-controlled generation
- explicit stage contracts
- durable provenance
- primary/contextual entities
- source-backed verification
- bounded surgical correction
- stale evaluation protection
- image eligibility controls
- fail-closed evidence behavior
- explicit cost limits
- durable observability
What I measure in production.
Grounding, precision, safe repair, quality, cost, latency, queue yield, and human editorial outcomes.
View the production measurement framework
This wasn't really a sports-content problem.
It was a production AI systems problem.
Production AI architecture
Models operate inside bounded, durable workflows rather than owning the workflow.
Evidence grounding & verification
Source evidence becomes traceable IDs, hashes, claims and version-bound verification.
Hybrid AI + deterministic systems
AI handles ambiguous semantic work. Code owns critical invariants.
Durable workflow orchestration
Long-running multi-stage work is observable, recoverable and stateful.
Concurrency & safe recovery
Content hashes, version checks and compare-and-swap transitions prevent stale decisions from changing newer state.
Observability & cost governance
Runs can be reconstructed, while calls, tokens, retries and concurrency remain explicitly bounded.
View the complete technical capability map
Prompt engineering and structured outputs: Prompts are versioned, validated, fingerprinted and schema-bound.
Evidence grounding and provenance: Editorial evidence becomes paragraph IDs, hashes, evidence units and traceable claims.
Hallucination reduction: Source-backed verification makes unsupported output detectable without pretending uncertainty can be eliminated completely.
Failure-mode analysis: The redesign began from a production incident and followed root causes through the complete pipeline.
Safe repair loops: Correction is bounded, surgical, evidence-aware and re-verified.
Prompt-injection defense: External source content is handled as untrusted data.
Progressive rollout: Shadow behavior is measured before it gains authority.
Test-driven invariant design: Editorial requirements become technical rules that can be independently tested.
The models were only part of the engineering.
What the system deliberately doesn't pretend to solve.
Models remain probabilistic
Grounding and verification reduce risk.
They do not mathematically prove truth.
Source quality still matters
Perfect grounding in a bad source can still produce bad information.
Image relevance is source-level
The current system verifies source relevance, not visual subject identity.
Reliability costs more
Grounding, verification and re-verification add latency and model spend.
This is deliberate.
Humans remain involved
The objective is not to remove humans from every decision.
It is to remove repetitive work while giving humans better-controlled output.
Future improvements
- vision-based image validation
- source-authority modelling
- cross-source contradiction detection
- richer trend signals
- production quality dashboards
- calibrated verifier confidence against human decisions
- additional domains beyond sports
The strongest part isn't that it can write.
Generating fluent sports articles is no longer particularly difficult.
The hard part was everything around the generation.
Knowing which stories belong together.
Knowing which evidence supports which claim.
Knowing which person is the subject and which person is merely context.
Knowing whether a source changed.
Knowing whether a verifier actually found something wrong.
Knowing exactly what a correction is allowed to change.
Knowing that the article being queued is still the article that passed verification.
And knowing when there simply isn't enough evidence to publish anything.
That's the system I built.
Not an AI that is always expected to produce an answer.
A controlled production AI agent system where every important decision is evidence-aware, observable, bounded, reproducible...
and allowed to say no.
Your company probably has a workflow like this.
Maybe it isn't publishing sports news.
Maybe it is:
If people repeatedly:
collect information
interpret it
make decisions
and move work between systems
there may be an opportunity to redesign the process around AI agents.
Not by throwing a chatbot at it.
By engineering the system around it.