Evaluate the prose before optimizing its score

A revision produces a better readability score and a worse paragraph. It shortens the sentences, removes a necessary qualification, and replaces a specific explanation with familiar wording. The number improves because the number measures a limited property. The reader loses the relationship the paragraph was supposed to explain.

An automated prose loop needs a clear account of what its measures can establish. Word count can verify a length requirement. A sentence-length distribution can identify a rhythmic mismatch. Neither can establish whether the argument is sound, whether a scene works, or whether the writing resembles an author’s practice in the ways that matter.

In March, I asked how to measure changes in prose across agentic loops, using authored aesthetic criteria and a project style guide. The request included objective measures and allowed judgment-based scales. The retained exchange establishes that evaluation problem; it does not establish a validated universal metric for literary quality or a successful completed novel-writing run.

Preserve what the revision must not change

Before generating alternatives, record the passage’s factual or narrative obligations. For an explanatory article, these include claims, sources, qualifications, and the intended reader. For fiction, they may include actors, event order, viewpoint, and the endpoint of a scene. A style improvement that changes these obligations should fail before anyone compares its elegance, because a smooth explanation of a stronger claim cannot replace a less polished explanation of the claim the source actually supports.

Separate mechanical requirements from editorial judgments. A required heading can be checked directly. Whether a transition gives the reader enough context needs a reader or an explicitly defined review procedure. Keep the results separate rather than combining them into an unexplained quality score whose weights conceal a serious failure.

Set a stopping condition before starting the loop. A revision can stop when it meets required constraints and an authorized editor accepts it. A run can also stop with unresolved disagreements. Endless rewriting is not evidence of increasing quality, especially when each new candidate makes comparison with the original more difficult.

An illustrative revision test

Consider a hypothetical explanation of a research result. The source says an improvement appeared in a small, restricted evaluation. Candidate A preserves that limit but uses a long, awkward sentence. Candidate B reads smoothly and omits the restriction. Candidate C preserves the restriction while separating the finding from its applicability.

A readability measure might prefer Candidate B. A factual-parity check should reject it. Candidate A and Candidate C can then proceed to editorial comparison, with the reviewer asked which makes the finding and its limits easier to understand. This example supplies a test design; it does not report measured rankings.

Now add a deliberate failure. Produce a version that repeats the same sentence pattern throughout the passage while keeping all facts intact. The factual check should pass it, while the editorial review should identify the monotonous construction. This helps establish that the evaluation procedure can distinguish factual fidelity from prose movement instead of awarding one broad impression to both.

Use anonymous candidates when practical. Show the reviewer the source obligations and the relevant comparison material without labeling one candidate as the improved version. Ask for specific spans that justify the judgment. A general statement that a passage “feels stronger” gives the next revision little usable information.

Keep measurements useful and limited

A metric becomes especially risky when the generator can improve it without improving the work. An agent asked to reduce average sentence length can split a qualification from its claim, while one rewarded for vocabulary variety can replace a precise repeated term with confusing synonyms that make the passage harder to follow. Both responses follow the objective the system was given.

Inspect rejected candidates as well as accepted ones. Record which constraints failed, which judgments remain disputed, and whether revisions repeatedly sacrifice the same requirement. That record can reveal a problem in the generation instructions or the evaluation criteria before another expensive run begins.

For a practical first loop, choose one short passage and one editorial problem. Freeze its content obligations, generate a small set of alternatives, and run the relevant mechanical checks. Have an editor compare the surviving candidates with the original and cite the passages that improved or deteriorated. Save the accepted text with the reasons for acceptance, while keeping any unresolved judgment visible for the next review.

Source notes

Based on the March 2026 prose-evaluation requests and retained evaluation designs. Candidate comparisons are hypothetical and no validated universal literary metric is claimed.