The bug that bit me

Funny story from my days in the Grocery Supply Chain Software Universe™. In the early aughts, the small company @blaster151 and I worked for was building modern .Net replacements for a world of old 70s and 80s-era software many chains were still running on.

It’s a fascinating industry, in part because margins are terrifyingly slim but volume is mind-bogglingly large. “Pricey” products are a tiny slice of most chains’ profits; too few people buy high end champagne at the grocery store to stock more than a bottle or two, for example.

Which brings us to software — I was the junior dev on the project, given the responsibility of writing and testing the myriad of nightly data migration scripts that updated price tags and shelf labels, POS price data, and so on.

Every night it took the database of interlocking sales, promotions, product SKUs, and stock information that our .Net software managed, and spit them out in a format the legacy systems (shelf tag printers, cash registers, etc…) could handle.

80-character 16-line ASCII files.

These were dense files, with dense documentation. “Row 9, Col 51: Single Character, Text. Allowed values ‘T’, ‘Q’, and ‘L.’ Drives BOGO type; see page 91.” and so on. The internally-developed file format had evolved over decades; every character was precious real estate.

They were all independent — the shelf label file had no structural relationship to the POS data file, for example — but by law in many states, if the label on the shelf and the register at the front disagreed, the customer got the lower price. Staying in sync was critical.

So, one afternoon I got a panicked call at home — it was over the holidays, and while Christmas had passed I’d taken time off through the New Year. Or so I thought.

Labels had gone up, prices were loaded, shoppers were starting to stock up for NYE — and bottles of high-end champagne were ringing up for $9 instead of $109.

record scratch

For a business whose margins were measured in fractions of a percent — pennies per sale, at best — taking a $100 bath on every bottle of top-shelf champaign sold on NYE was a nightmare. It had to be fixed FAST, and teams were standing by to reload data.

Once good data was ready.

The good news is that the problem was embarrassing and obvious. My nightly data exports — the ones that translated our big-ass matrix of promos and pricing into the dumb flat ASCII files — were given 4 characters to store each SKU’s price…

…And my export was naively tossing out any data that didn’t fit its four-character mask. 10999 became 0999 became CHEAP BOOZE ON NYE WOO

Two mitigating factors: first, most stores only stocked a few bottles. Remember that bit about low margin, high volume products being the backbone of the grocery industry? Yeah, top-shelf champaign ain’t that. Second, it was literally the only product over $100 the chain carried.

Back in the mists of time, when the file format was created, the idea of selling a grocery store product for more than a hundred bucks was bizarre and unthinkable. Their POS system had never had to ring up a product that expensive, and there was literally no way to tell it to.

In theory, we could expand the field — use five characters to store the register price even though only one product needed it — but remember that every character in the export format was precious real estate, jam-packed with data to handle the myriad of sale and BOGO scenarios.

We’d used the last of our emergency reserve characters months ago, to squeeze in a new kind of cross-sell promotional discount, and there was just nothing left to give.

So, with no options left, I proposed a simple solution: MIN(price, 99.99).

The team did some digging and discovered, much to their shock, that it’s what the old mainframe software had been doing all along. Never documented, of course — because who’d ever actually use a price higher than $100?

We made the change, pushed the code, re-ran the exports, and teams of runners loaded up every POS cash register with the “correct” champagne price. They still rang up lower than intended — but $99 for a $109 bottle is a lot less painful than $9.

By their estimate, they’d probably been losing thousands of dollars a year in unintentional $4, $5, and $6 discounts as inflation nudged that one product — expensive NYE champagne — just over the $100 line, but the registers steadfastly insisted on a $99.99 sale price.

The patch gave them time to look up and down their whole software stack for other “$100 bugs” (and for us to improve our QA) — but it was certainly one of the more stressful holiday bugs I’d wrangled.