assort
A large-scale data classification pipeline built on LLMs
Overview
A batch pipeline that automatically classified hundreds of thousands of Japanese text records, specifically food menu names, according to a prescribed classification standard using the Gemini API. I built it on commission for a research institution at a university. Thorough parallelization and cost management carried the processing of the real dataset to completion, and in a separate engagement I applied the same architecture to ICD-10 classification in the medical field.
The brief
In this engagement, I classified hundreds of thousands of food menu names, stored in a CSV of several tens of megabytes that had been provided to the research institution, according to a proprietary standard defined for the research. It is fair to read the output as an index quantifying roughly how much of each menu item was carbohydrates or vegetables.
At a scale of hundreds of thousands of records, manual classification is not realistic. Rule-based processing, on the other hand, cannot handle spelling variants and near-synonyms, such as “karaage” written several different ways, or its close relative “tatsuta-age”. Interpreting meaning takes an LLM, but naively calling the API hundreds of thousands of times inflates both cost and runtime, and a mid-run failure means starting the whole job over. assort was my design of the entire processing platform needed to carry classification at this scale through to the end reliably. It had to cover not only the classification mechanism but the operational foundation supporting it.
The platform
The design concentrated on three things: controlling parallel execution, optimizing cost, and recovering from failures.
For parallelism, I matched the number of concurrent requests to the API’s rate limits and secured enough throughput to process hundreds of thousands of records within a practical timeframe. On cost, I was systematic about driving down the price of each call, from model selection to prompt structure. Prompts were fixed in a form that let caching work effectively, and outputs were structured against a schema, which suppressed classification drift and redundant retries. The cost of each unit of work was trackable at all times, and I kept the running total within the estimate throughout the operation.
In a long batch job, something always goes wrong partway through. So I made the whole pipeline interruptible and resumable: if a job died in the middle of the night, processing picked up from that exact point the next morning. That is the configuration that carried the real dataset, hundreds of thousands of records, to completion.
Afterward
Swap the classification standard and the input, and this pipeline transfers to other domains as is. In a separate engagement I did exactly that, applying it to ICD-10, the diagnostic classification used in medicine. Between a demo that calls an LLM once and a research data platform that keeps calling it hundreds of thousands of times, there is a wider gap than appearances suggest. Having built the operational design that closes that gap entirely on my own was the biggest gain from this project.