eday.studio
All reports

Improving a marketplace classifier

I wanted to improve the prompt without tuning it to the test. I turned recurring failures into specific changes, kept 40 cases outside the editing loop, and ran the chosen prompt across all 200 cases.

Updated August 21, 2026200 synthetic cases · 22 resale categories · Text only

What the classifier needs to get right

A listing title is not enough. The classifier has to identify the exact product and retain details that change its resale value: generation, storage, carrier, condition, included accessories, financing status, and whether the asking price is usable.

Five cases from the test suite
Listing

iPhone 15 Plus 256GB T-Mobile Balance Owed

Works now but has remaining device payments and may be blocked. T-Mobile only. Asking price: $280.

What a correct result must preserve
Identity
iPhone 15 Plus · 256GB
Restriction
T-Mobile · financed
Asking price
$280 is a full-item price
Why it matters

Dropping the balance restriction could match this phone with clean, unlocked sales.

Listing

MacBook Air 13 M2 A2681 LCD Screen Assembly

Used replacement display assembly only. Laptop and logic board not included. Asking price: $210.

What a correct result must preserve
Identity
MacBook Air M2 · A2681
Package
Display assembly only
Asking price
$210 prices the replacement part
Why it matters

A screen assembly cannot be valued against complete MacBook sales.

Listing

Nintendo Switch OLED Tablet Console Only HEG-001

Screen tablet only. No Joy-Con, dock, charger, or games. Asking price: $135.

What a correct result must preserve
Identity
Switch OLED · HEG-001
Package
Tablet only
Missing
Controllers · dock · charger
Why it matters

Bundle contents materially change which sold listings are comparable.

Listing

Leica Q3 43 Camera Reservation Deposit

Nonrefundable deposit to join the waitlist; remaining balance due at pickup. Displayed price: $500.

What a correct result must preserve
Identity
Leica Q3 43
Listing type
Reservation deposit
Asking price
$500 is not a usable sale price
Why it matters

Treating a deposit as the camera price would create a false bargain.

Listing

Remote iPhone Samsung Carrier Unlock Service

Digital IMEI submission service only. No physical phone is being sold. Displayed price: $35.

What a correct result must preserve
Inventory
No physical product
Market
Unsupported service listing
Asking price
$35 is not a product price
Why it matters

The classifier should stop before product search or pricing begins.

How I measured it

The suite contains 200 hand-authored, hand-labeled synthetic listings across 22 resale categories. A strict pass requires every check to pass. The semantic score averages six dimensions per case, so a narrow omission counts differently from a complete identity or safety failure.

200 reviewable cases
The synthetic listing text and expected output live side by side in the repository. No independent second-annotator review is recorded.
Six checks
Category, identity, search terms, value-bearing attributes, market support, and asking-price usability. Identity, search, and attributes receive partial credit inside the semantic score.
Two cohorts, one release gate
The suite combines 100 frozen regression cases with 100 harder challenge cases. I used 60 challenge cases for development, kept 40 outside the editing loop, and ran the winning prompt across all 200 cases.
Pricing safety
The suite has 19 unusable-price cases and 8 unsupported-market cases. The retained v3 summary records one combined safety mismatch, but its original case-level checkpoint is no longer available to separate the two flags.
Hard cases on purpose
The set concentrates on incomplete packages, damage, financing, carrier locks, misleading prices, accessories, and ambiguous product identity.
Text only
Production can also receive listing photos, but every measurement in this report isolates text extraction. It does not validate image understanding.

How I improve the classifier

I treat the model, reasoning setting, and prompt as separate levers. I change one at a time, measure it against the same contract, and keep a holdout outside the editing loop.

  1. Hold the baseline still

    Every model starts with the same prompt, listing formatter, JSON schema, reasoning level, tools, and scorer. This reduces configuration differences and tests each model as a drop-in option under one contract.

  2. Turn failures into prompt hypotheses

    I group misses by package state, identity, restrictions, attributes, market support, and price usability. A prompt change must target a named failure rather than merely become longer.

  3. Test on a small set

    Candidates first run on 10 to 20 targeted cases with known passing controls, then on a fixed development cohort. Each run records the prompt fingerprint, tokens, dollars, and latency.

  4. Keep the holdout out of editing

    Only the strongest candidate reaches cases that prompt authors did not use while editing. The winner then faces the complete 200-case release gate.

I reject candidates that increase guardrail mismatches. Otherwise I prioritize strict correctness, use the semantic score to understand the severity of misses, check the locked holdout, and then compare cost and tail latency.

What changed with prompt v3

Prompt v3 makes package state, restrictions, identity, and price rules explicit. It passed 18 more listings on the full release gate and cut attribute misses from 70 to 50. The locked holdout also improved from 14/40 to 21/40.

PromptStrict passesSemantic scoreDevelopmentLocked holdoutGuardrail mismatchesAverageP95Observed / 1KCold / 1K
Baseline2ca71c9c3894117/20094.82%22/6014/402 mismatches2.89s4.80s$0.45$0.51
Production v374c46c57d079135/20096.79%35/6021/401 mismatch2.99s5.11s$0.38$0.65
All prompt experiments shown here cost $0.2801. That total includes targeted tests, development, locked holdout, and both full release runs. Observed cost includes provider cache savings; cold cost prices the same tokens without a cache discount.

The current model check

I reran Luna and Gemini 3.7 Flash on the same 200 cases with prompt v3, low reasoning, no tools, and the same scorer. Gemini passed 47 fewer cases, cost 5.98 times more, and had 4.38 times higher P95 latency. Its run also encountered eight capacity errors.

ModelStrict passesSemantic scoreObserved cost / 1KAverageP95Guardrail mismatchesAvailability
Lunagpt-5.6-luna · low reasoning136/20096.84%$0.37743.99s7.52s1 mismatchNo interruptions
Gemini 3.7 Flashgemini-3.7-flash · low thinking89/20093.70%$2.25627.47s32.92s5 mismatches8 capacity errors

Run accounting

ModelInput tokensCached inputOutput tokensReasoning tokensRun spendCost basis
Lunagpt-5.6-luna · low reasoning322,484303,42054,6666,504$0.075480Published-rate estimate
Gemini 3.7 Flashgemini-3.7-flash · low thinking283,848063,5592,867$0.451232Published-rate estimate
Output totals include reasoning tokens. These two providers exposed token usage but not a final invoice charge, so spend is calculated from successful-call tokens at the published Standard rates. Failed-request billing may not be represented.

The production choice

I kept Luna with prompt v3 for the tested text path. A fresh repeat passed 136 cases and made no asking-price mistakes. Its one guardrail mismatch was conservative: it marked a deposit listing unsupported while still refusing the deposit as a usable price.

Measured result: 136/200 strict passes · 96.84% semantic score · $0.65 cold per 1,000 text classifications · 0/19 price traps missed.

Method and limits

This is a practical engineering benchmark for structured extraction from listing text, not an academic benchmark or a test of each provider's full platform.

Prompt identity
Current release runs use SHA-256 prompt fingerprints. Production v3 is 74c46c57d079; the baseline is 2ca71c9c3894.
Evaluation identity
The current Luna and Gemini comparison is bfa83589a1d3. The earlier prompt-development gate is c63ad64bf1c9. Both fingerprints cover the prompt, scorer, category resolver, schema, and fixtures.
Interpretation
The 135/200 score is a release-gate result, not a production-accuracy estimate. The locked holdout improved from 14/40 to 21/40, which is consistent with improvement beyond the 60 development cases, but the holdout is now known and cannot remain untouched for future prompt studies.
Reasoning
The current Luna run used low reasoning and Gemini used low thinking.
Sampling
Temperature and top-p were not set, so provider defaults applied.
Output
Every model returned the same strict structured JSON schema.
Execution
Luna used five concurrent cases and completed without interruption. Gemini used two, produced eight 503 capacity errors, and needed one manual restart after a stalled attempt. Successful cases were checkpointed by listing ID.
Latency
Average and P95 latency are operational measurements from one hosted run, not a maximum-throughput claim.
Scope
Photos, web search, grounding, and external tools were disabled. The results apply to these 200 text-only cases across 22 categories.
Cost
Successful-call tokens were priced at the providers' current Standard rates.
Production can receive listing photos, but photos were disabled here. The suite is synthetic and intentionally difficult, not sampled to match live traffic. It does not measure image understanding, web-search quality, final valuation accuracy, production-wide accuracy, or realized resale profit. Provider prices can change, and failed-request billing may not be represented.