Architecting
systems at scale.

Sales teams uploaded license inventory spreadsheets, all of them in the same few days at month end. Every row was one software license; all anyone wanted was a count per product.

The app was cookie-cutter: web front end, .NET middle tier, SQL back end, with the business logic exhaustively encoded as stored procedures. Uploads went through the middle tier, held whole in memory, then pushed to blob storage — so the progress bar finished and users still waited. Then every row of every workbook was bulk-inserted into one shared table and summarized in SQL.

I identified SQL as the bottleneck. The PM’s fix was to scale the database SKU up for month end and back down after — which in practice means copying the database, with the replication lag and cutover risk that brings.

Plan B was to leave it on the expensive SKU permanently, on the reasoning that it was Microsoft’s cloud and Microsoft’s project, so the cost wasn’t real. I had to explain that every project carries its own cloud budget. Neither plan touched the actual problem.

Upload straight to blob storage and take the middle tier out of the data path entirely. A new blob raises an event onto the service bus. A pool of background workers picks up the event and streams the workbook.

Because only the summary mattered, the workers tally each license as the row goes by and write a single summary record at the end. The raw rows never enter SQL, and the shared table disappears — the data was per-upload all along.

The first implementation leaned on elaborate LINQ queries and took about 24 hours to chew through ~150 MB. A second developer got it to 12 hours.

Both shared the same root flaw: load everything into memory, then do clever parsing on top. The work scaled with total input size, so the only way out was a bigger machine.

Pick one file as the lead. Read a fixed window of it — on the order of 10,000 rows — into a dictionary keyed on the join column. Then stream every other file start to finish once, completing any record that matches the window.

Advance to the next window in the lead file and repeat. When the lead file is exhausted, every combination has been seen.

52 seconds for the same 150 MB — marginally above what it cost to read and write that much data on the drives of the day. The compute bottleneck was gone; what was left was the disk.

Memory is now bounded by the window, not the input. The same code handles multi-gigabyte files that would have taken the earlier versions off a cliff.