Agree the measure before the budget: lessons from ChatGPT ads

A trade article published on 14 September quoted two paid media leaders.

Shamsul Chowdhury, SVP of paid media at Zeno Group, on ChatGPT ads so far: “While they’ve been a hot new platform for advertisers, the ROI just hasn’t been there for the most part.”

Morgan Brown, director of paid media at Wayfair, on the platform’s newest format: “Yes, Wayfair is participating in OpenAI’s Sponsored Agent pilot to explore how conversational commerce can help customers make more informed, confident decisions.”

Wayfair set guardrails around product accuracy, transparency and handoffs to service.

OpenAI announced that format on 16 September. It describes it this way: after seeing an ad, a ChatGPT user can open a clearly labelled conversation with a business’s own agent. It is a US-only pilot with selected advertisers. The ads themselves already run in India, Europe, the Middle East and North Africa.

I want to flag the limit. This is two practitioners in one trade article, not a study. Neither is showing numbers. One is negative about the results so far, and the other is exploring a new format. Neither tells you what your campaign would return.

That is the part I keep coming back to. A launch this big reads like proof that a channel works. It is proof that a channel exists. Revenue to the platform shows that advertisers are buying. It does not show that buying pays off for them.

The newest format matters for a measurement reason. When the ad opens a conversation instead of a link, the click is no longer the thing you are buying.

For two decades, digital advertising has been measured on a chain that starts with a click, moves to a visit and ends in a conversion. A conversation breaks the chain at the first step. The shopper may ask questions, receive recommendations and decide inside the chat, and a click-based report will describe none of that.

So the usual metrics, click-through rate and cost per click, lose meaning. You need to decide what a good conversation looks like before you can price one.

Agree the measure before you agree the budget. In practice this means writing down, in advance and with the people who will judge the result, what counts as success.

The post ended with a question: if you were testing this, what would you need to see in 30 days to keep spending? Here is how I would answer it, as a proposal rather than a finding.

Before spending:

1. Fix a small budget and a stop date. Decide the amount you can lose without difficulty, and the day you will review.
2. Name the outcome. Choose one primary measure that your own systems can count, such as qualified enquiries, orders or new customers, not a platform metric. Add a few secondary ones if you must, but decide which one decides.
3. Set the threshold in advance. Write the number that would make you continue and the number that would make you stop. If you wait until the results arrive, you will find a reason to read them kindly.
4. Plan the comparison. Without a comparison, you cannot tell ad effect from what would have happened anyway. Hold back some comparable regions or audiences, or alternate periods, and compare outcomes in your own sales records.

During the test:

5. Read the conversations. Sample the actual exchanges. A handful of transcripts will tell you whether the agent answered accurately, stayed within the guardrails and handed over to a person when it should have. Wayfair’s guardrails on accuracy, transparency and handoffs are a sensible checklist.
6. Track the cost of a qualified outcome, not of an interaction. A long conversation that goes nowhere is a cost.

At day 30:

7. Apply the rule you wrote. If the result meets the threshold, extend the spend in steps. If it does not, stop or change one thing and test again.

I would not scale because the channel is large or because competitors are testing it. Both are reasons to look, not reasons to spend. I would also not judge the channel by one agency’s comment, in either direction. A negative quote and a positive launch are both anecdotes.

Every new advertising format arrives with a story about its promise, and most of the early reporting comes from the party selling it. The discipline that protects you does not change: agree the measure, set the threshold, keep a comparison and count the result in your own books.

It is also the same discipline I apply to ad platforms generally. When the party that sells the ad also reports its performance, someone on your side should be able to check the figure independently.

If you were testing this, what would you need to see in 30 days to keep spending?