Choosing a model for listing classification

I compared four models to choose one for extracting product facts from resale listings.

August 20, 2026100 synthetic cases · Four models · Text only

One fixed test across four models

I ran the same 100 hand-authored listings through all four models, using the same instructions, JSON schema, and scorer.

What counted as correct: A strict pass required the category, product identity, sold-comparable search terms, value-bearing attributes, market support, and asking-price usability to agree with the expected result.
Provider settings: I left reasoning, sampling, and other generation settings at each provider's defaults.
What the test did not cover: I disabled photos, search, grounding, and external tools. I did not test sold-comparable retrieval, final valuation accuracy, or realized resale profit.

A sample of the shared prompt

The instructions required each model to use listing evidence, keep facts that affect value, build a sold-items search query, and flag misleading asking prices.

Shared classifier instruction · excerpt

Identify the exact product represented by the listing using only supplied text and visible image evidence.
Never invent a model, variant, condition, or attribute that is not supported by the evidence.
Create a concise eBay sold-items search query using the consumer-facing product name and only variants such as capacity or size that materially change value.
Mark askingPriceUsable false when the displayed price is a placeholder, per-item starting price, down payment, monthly payment, financing amount, or otherwise not the purchase price for the identified product or bundle.

The live request also included the complete category rules and strict JSON output schema. Line breaks were added here for readability; the wording is unchanged.

The first model comparison

Luna had the highest strict and semantic scores and the lowest estimated cost. Gemini Lite was faster on average, but passed 38 fewer cases and had a slightly higher P95 latency.

Model	Strict passes	Semantic score	Cost / 1K	Average	P95	Tokens / case
Lunagpt-5.6-luna	81/100	98.05%	$0.7929	6.28s	10.35s	1514.49
Gemini 3.7gemini-3.7-flash	54/100	95.10%	$2.8909	8.06s	20.39s	1417.71
Gemini 3.6gemini-3.6-flash	56/100	95.44%	$3.5910	11.54s	40.06s	1604.41
Gemini Litegemini-3.5-flash-lite	43/100	93.47%	$0.8795	2.51s	11.52s	1063.29

Run accounting

Model	Input tokens	Output tokens	Reasoning tokens	Run spend
Lunagpt-5.6-luna	102,452	48,997	26,037	$0.079287
Gemini 3.7gemini-3.7-flash	80,850	60,921	30,113	$0.289091
Gemini 3.6gemini-3.6-flash	80,850	79,591	60,662	$0.359104
Gemini Litegemini-3.5-flash-lite	80,850	25,479	0	$0.087953

Output totals include reasoning tokens. I estimated spend from successful-call usage and the published Standard rates available during the study. These figures are not reconciled to an invoice.

Why I chose Luna

I chose Luna for text classification because it had the highest accuracy and lowest estimated cost in these runs. I then worked on its prompt.

Measured result: 81/100 strict passes · 98.05% semantic score · $0.79 per 1,000 classifications · 10.35s P95 latency.

Limits of the first test

This small, controlled test covers four specific runs. Results may differ on other data or settings.

Missing fingerprints: I did not save fingerprints of the prompt or scorer. I kept outputs, checks, token counts, costs, and timings, but I would not compare these scores directly with the later prompt-v3 study.
Different concurrency: I ran five Luna requests at a time and two Gemini 3.7 requests. These latency measurements do not establish either provider's maximum throughput.
Cost is estimated: The four models completed 400 calls. At the published Standard rates, the recorded usage totals about $0.82. Charges for failed requests may be missing.
Synthetic cases: I used 100 synthetic, hand-labeled stress cases to find classification errors. They do not reflect the mix of listings in production.