eday.studioImproving a marketplace classifier
I revised the prompt to address recurring errors and tested it on 200 synthetic listings, including 40 cases kept outside prompt editing.
What the classifier needs to get right
The classifier has to identify the product and retain details that change its resale value: generation, storage, carrier, condition, included accessories, financing status, and whether the asking price is usable.
iPhone 15 Plus 256GB T-Mobile Balance Owed
Works now but has remaining device payments and may be blocked. T-Mobile only. Asking price: $280.
- Identity
- iPhone 15 Plus · 256GB
- Restriction
- T-Mobile · financed
- Asking price
- $280 is a full-item price
Dropping the balance restriction could match this phone with clean, unlocked sales.
MacBook Air 13 M2 A2681 LCD Screen Assembly
Used replacement display assembly only. Laptop and logic board not included. Asking price: $210.
- Identity
- MacBook Air M2 · A2681
- Package
- Display assembly only
- Asking price
- $210 prices the replacement part
A screen assembly cannot be valued against complete MacBook sales.
Nintendo Switch OLED Tablet Console Only HEG-001
Screen tablet only. No Joy-Con, dock, charger, or games. Asking price: $135.
- Identity
- Switch OLED · HEG-001
- Package
- Tablet only
- Missing
- Controllers · dock · charger
Bundle contents materially change which sold listings are comparable.
Leica Q3 43 Camera Reservation Deposit
Nonrefundable deposit to join the waitlist; remaining balance due at pickup. Displayed price: $500.
- Identity
- Leica Q3 43
- Listing type
- Reservation deposit
- Asking price
- $500 is not a usable sale price
Treating a deposit as the camera price would create a false bargain.
Remote iPhone Samsung Carrier Unlock Service
Digital IMEI submission service only. No physical phone is being sold. Displayed price: $35.
- Inventory
- No physical product
- Market
- Unsupported service listing
- Asking price
- $35 is not a product price
The classifier should stop before product search or pricing begins.
How I measured it
The suite contains 200 hand-authored, hand-labeled synthetic listings across 22 resale categories. A strict pass requires every check to pass. The semantic score averages six dimensions per case, so a narrow omission counts differently from a complete identity or safety failure.
- Cases and labels
- The listing text and expected output are stored together in the repository. No independent second-annotator review is recorded.
- Six checks
- Category, identity, search terms, value-bearing attributes, market support, and asking-price usability. Identity, search, and attributes receive partial credit inside the semantic score.
- Development and holdout cases
- The suite combines 100 frozen regression cases with 100 harder challenge cases: 60 for development and 40 held out.
- Pricing safety
- The suite has 19 unusable-price cases and 8 unsupported-market cases. The retained v3 summary records one combined safety mismatch, but its original case-level checkpoint is no longer available to separate the two flags.
- Cases covered
- The set concentrates on incomplete packages, damage, financing, carrier locks, misleading prices, accessories, and ambiguous product identity.
- Text only
- Production can also receive listing photos, but every measurement in this report isolates text extraction. It does not validate image understanding.
How I tested prompt changes
I changed the model, reasoning setting, or prompt one at a time, keeping the test cases and scoring rules fixed.
Keep the setup fixed
For model comparisons, I kept the prompt, listing format, JSON schema, reasoning level, tools, and scorer fixed.
Group the errors
I grouped errors by package contents, identity, restrictions, attributes, market support, and price usability, then wrote instructions to address those failures.
Test on a small set
I tested candidates on 10 to 20 targeted cases alongside known passing controls, then on the fixed development set. Each run recorded the prompt hash, tokens, estimated cost, and latency.
Test the held-out cases
I evaluated the strongest candidate on cases kept outside prompt editing, then tested the chosen prompt on all 200 cases.
I rejected prompts with more market-support or asking-price errors. I then compared strict passes, the severity of remaining errors, holdout results, cost, and P95 latency.
Prompt excerpts
The baseline asked for conservative extraction. V3 added explicit rules for incomplete packages, comparable searches, and amounts that were not full purchase prices.
requiredTerms must be the smallest stable identity terms that every valid comparable should contain.
valuationAttributes must contain only explicitly evidenced, value-changing facts.
For a bundle, identify and search the principal value-bearing product; describe accessories in variant and bundle-contents but do not require an identical accessory bundle in sold comparables unless it materially changes value.This was part of the shared instruction before the v3 addendum.
Whenever the text says only, not included, missing, removed, replacement, part, accessory, box, case, stand, mount, bracket, shell, code, lot, pack, kit, or bundle, explicitly record package.
Search queries must name the actual item being sold, not merely the product it fits.
Set askingPriceUsable false for deposits, installments, per-item or per-unit amounts applied to a larger lot or bundle, financing amounts, and any displayed amount that is not the full purchase price of the identified item or bundle.Additional instructions selected after development and holdout testing.
What changed with prompt v3
Prompt v3 passed 18 more listings across the full test set and cut attribute misses from 70 to 50. The locked holdout improved from 14/40 to 21/40.
| Prompt | Strict passes | Semantic score | Development | Locked holdout | Guardrail mismatches | Average | P95 | Observed / 1K | Cold / 1K |
|---|---|---|---|---|---|---|---|---|---|
| Baseline2ca71c9c3894 | 117/200 | 94.82% | 22/60 | 14/40 | 2 mismatches | 2.89s | 4.80s | $0.45 | $0.51 |
| Production v374c46c57d079 | 135/200 | 96.79% | 35/60 | 21/40 | 1 mismatch | 2.99s | 5.11s | $0.38 | $0.65 |
Luna and Gemini with prompt v3
I reran Luna and Gemini 3.7 Flash on the same 200 cases with prompt v3, low reasoning, no tools, and the same scorer. Gemini passed 47 fewer cases, had 5.98 times the estimated cost, and had 4.38 times higher P95 latency. Its run encountered eight capacity errors; concurrency and restart differences are documented below.
| Model | Strict passes | Semantic score | Observed cost / 1K | Average | P95 | Guardrail mismatches | Availability |
|---|---|---|---|---|---|---|---|
| Lunagpt-5.6-luna · low reasoning | 136/200 | 96.84% | $0.3774 | 3.99s | 7.52s | 1 mismatch | No interruptions |
| Gemini 3.7 Flashgemini-3.7-flash · low thinking | 89/200 | 93.70% | $2.2562 | 7.47s | 32.92s | 5 mismatches | 8 capacity errors |
Run accounting
| Model | Input tokens | Cached input | Output tokens | Reasoning tokens | Run spend | Cost basis |
|---|---|---|---|---|---|---|
| Lunagpt-5.6-luna · low reasoning | 322,484 | 303,420 | 54,666 | 6,504 | $0.075480 | Published-rate estimate |
| Gemini 3.7 Flashgemini-3.7-flash · low thinking | 283,848 | 0 | 63,559 | 2,867 | $0.451232 | Published-rate estimate |
The production choice
I kept Luna with prompt v3 for text classification. A repeat run passed 136 cases with no asking-price errors. The other 64 cases included 51 attribute misses, 22 comparable-search misses, 2 identity misses, 1 category miss, and 1 conservative market-support mismatch. Some cases failed more than one check; any mismatch fails a strict case even when the rest of the output is correct.
Measured result: 136/200 strict passes · 96.84% semantic score · $0.65 cold per 1,000 text classifications · 19/19 price traps handled safely.
Method and limits
These settings describe the current Luna and Gemini runs. The earlier prompt comparison was recorded separately.
- Prompt hashes
- The runs use SHA-256 prompt hashes. Production v3 is 74c46c57d079; the baseline is 2ca71c9c3894.
- Evaluation hashes
- The Luna and Gemini comparison is bfa83589a1d3; the earlier prompt comparison is c63ad64bf1c9. Both hashes cover the prompt, scorer, category resolver, schema, and test cases.
- Interpretation
- The 135/200 result measures this test set, not production accuracy. The holdout improvement is consistent with gains beyond the 60 development cases, but its results are now known. Future prompt studies cannot treat it as an untouched holdout.
- Reasoning
- The current Luna run used low reasoning and Gemini used low thinking.
- Sampling
- Temperature and top-p were not set, so provider defaults applied.
- Output
- Every model returned the same strict structured JSON schema.
- Execution
- Luna used five concurrent cases and completed without interruption. Gemini used two, produced eight 503 capacity errors, and needed one manual restart after a stalled attempt. Successful cases were checkpointed by listing ID.
- Latency
- Average and P95 latency are operational measurements from one hosted run, not a maximum-throughput claim.
- Scope
- Photos, web search, grounding, and external tools were disabled. The results apply to these 200 text-only cases across 22 categories.
- Cost
- Cost estimates use published rates, which may change.