Catalog Normalization: Units, Prices, and Providers in One Model

Supplier catalogs are messy by default. The same material can appear with different names, packaging, units, currencies, tax assumptions, and regional availability. A human can often tell two items are equivalent, but software needs a normalized model.

The mess isn't an edge case — it's the steady state. One distributor lists reinforcing bar by the metric ton, another by the linear meter, a third by the piece at a fixed stock length. Names drift too: "rebar #5," "corrugated steel B500S Ø16," and "varilla 5/8" can all describe steel that an estimator would treat as interchangeable for a footing. If you store those rows as-is and try to price against them directly, every budget inherits the supplier's formatting decisions. Change one product page and the budget breaks.

Omnicost separates raw supplier observations from canonical catalog items. A raw observation preserves what the source published: name, URL, price, currency, unit, timestamp, and provider. A canonical item represents the construction concept teams actually want to estimate against.

Each observation also carries a trust score from 0 to 100. That score reflects how much weight the source should get: a manufacturer's published price list earns more than a one-off scrape of a marketplace listing. Crucially, the observation is immutable history — it records what a given source said at a given moment, not what we wish it said. The canonical item is the interpretation layer that sits on top.

This distinction matters. If a supplier changes a product name, the raw observation can change without breaking every budget. If several suppliers publish equivalent items, Omnicost can aggregate them under one canonical item and compute a more useful market benchmark.

The benchmark itself is a median_price_cents field, recomputed nightly and weighted by source trust. We use a trust-weighted median rather than a plain average on purpose: a median shrugs off a single outlier — a clearance dump or a typo'd price — that would otherwise drag a mean off course, and trust weighting lets a reliable source pull harder than a shaky one. Cents are the storage unit so currency math stays in integers and never loses a fraction to floating point.

A worked example: three rebar rows into one item

Say three sources report the same Ø16 reinforcing bar on the same week:

• Source A — manufacturer list, trust 90 — 0.92 per kg
• Source B — regional distributor, trust 70 — 0.97 per kg
• Source C — marketplace scrape, trust 30 — 1.45 per kg

All three become raw observations: distinct names and URLs, same canonical concept, each with its own currency, unit (kg), timestamp, and trust score. The nightly job collapses them into one canonical item. The trust-weighted median lands near A and B — roughly 0.93 to 0.94 per kg — and source C's high outlier is present in the history but barely moves the benchmark because its weight is low. Nothing was deleted; the three rows are still queryable for audit. The estimate just sees one defensible number.

Units are one of the hardest parts. Construction teams price per m2, m3, kg, hour, day, unit, roll, bag, pallet, and more. The system has to preserve the source unit while helping estimators compare like with like. When conversions are unsafe, Omnicost marks the uncertainty instead of hiding it.

The unsafe-conversion problem is real. Converting that rebar from "per ton" to "per kg" is exact arithmetic. Converting "per linear meter" to "per kg" requires the bar's mass per meter, which depends on diameter — knowable, but only if you trust the source's diameter. Converting a bag of mortar to m³ of placed mortar depends on yield and waste, which the catalog can't assume. So Omnicost keeps the source unit on the observation and flags the conversion as uncertain rather than fabricating a clean number that a quantity surveyor would have to unwind later.

Classification follows the same restraint. When a source supplies an EU cpv_code, Omnicost stores it on the item; when it doesn't, the field stays empty. CPV codes are never AI-assigned, because a plausible-looking-but-wrong procurement code is worse than no code at all — it silently mis-routes the item in tenders that filter on CPV.

Provider metadata is also part of the model. A price from a trusted supplier with recent updates should carry more weight than an old scraped page. Trust, freshness, and region all influence how useful a price is.

Here is live equipment-rental data from the catalog, sourced from US providers — each row is one source's observation, deduped under a single canonical concept and carrying its own real unit, currency, and timestamp:

A CAT 320 L excavator surfacing at $49.59 per hour is exactly that shape: one canonical rental concept, real unit, real timestamp, with each contributing source kept as its own row underneath.

The result is a catalog that behaves like infrastructure. Budgets, search, APIs, agents, and dashboards all consume the same normalized price layer. Because the canonical item is computed from observations rather than hand-edited, the whole catalog can refresh itself nightly without anyone re-keying prices — and when a number looks wrong, you can always trace it back to the exact source row that produced it.