Hellmann, for its Thermo Fisher Scientific (TFS) inbound freight account. TODO: Confirm the exact client wording and whether Hellmann and TFS can be named publicly
Hellmann - AI-Powered Inbound Shipment Booking Automation For Thermo Fisher Scientific
Bitontree delivered an AI-assisted inbound shipment booking platform for Hellmann's TFS account. It takes supplier invoices from a shared mailbox to a delivered Vendor Booking XML in Hellmann's Smart Order system, with an operator review step in between.
Hellmann - AI-Powered Inbound Shipment Booking Automation For Thermo Fisher Scientific
About The Client
Hellmann is a global freight forwarding and logistics provider. One of the customer accounts its operations teams serve is Thermo Fisher Scientific (TFS), which ships laboratory and life-science products from suppliers around the world into its regional operations.
Every TFS inbound shipment arrives with supplier paperwork, mostly commercial invoices. Hellmann's team must turn each one into a vendor booking in Hellmann's Smart Order system, and until now that meant reading each invoice and keying the booking by hand.
Hellmann needed a way to do that reading and keying faster and more consistently. Bitontree built a platform that watches the shared mailbox where supplier documents arrive, reads them in many formats and layouts, extracts the shipment data, checks it against TFS master data, and hands a clean booking to Smart Order, with Hellmann's operators reviewing the result.
| Project Detail | Information |
|---|---|
| Client | Hellmann, for its Thermo Fisher Scientific (TFS) inbound freight account. TODO: Confirm the exact client wording and whether Hellmann and TFS can be named publicly |
| Industry | Freight Forwarding, Life Sciences Logistics, Freight Operations Automation |
| Location | TODO: Confirm client location |
| Duration | Started June 2025; active development through October 2026. TODO: Confirm engagement dates |
| Project Scope | Document-processing and mail-automation service, backend API, and operations dashboard for the TFS inbound booking workflow |
| Services Used | AI Automation Development, Document Intelligence (OCR and LLM extraction), Custom Software Development, Workflow Automation, Logistics Operations Platform Development, Cloud Engineering |
| Engagement | End-to-end product engineering across email intake, AI extraction, backend API, and operator dashboard |
| Live Link | https://tfs.logiai.info/ |
| Client Contact / Testimonial | TODO: Confirm if a testimonial or client quote can be included |
The Objective
Hellmann was booking TFS inbound shipments from supplier invoices that arrived by email. Each invoice had to be read, matched to the right shipper and consignee, enriched with packaging and weight data, and keyed into a booking in Smart Order. The goal was to automate that chain for Hellmann's operations team without losing control over data quality.
The platform needed to:
- Watch a shared mailbox and decide which emails contain shipment documents
- Pick out the commercial invoice pages from PDFs, ZIP files, and scanned images
- Extract shipper, consignee, order numbers, item codes, quantities, prices, and packaging details
- Support many supplier invoice layouts, and let new suppliers be added without code changes
- Match extracted parties to TFS company codes and the right origin and destination ports for each load type
- Enrich invoice lines with CBM, weight, dimensions, pack type, and incoterm from a PO master database
- Process courier shipments by reading DHL and FedEx airway bills for per-package data
- Generate a Vendor Booking XML document and deliver it to Hellmann's Smart Order system
- Give Hellmann's operators a dashboard to review, correct, reprocess, and send each request
- Alert operators by email, routed to the right regional contact
The Challenge
Supplier invoices are not standardised. Each issuer has its own letterhead, table layout, field names, and page structure. A single email can carry a ZIP of mixed documents, with packing lists, declarations, and permits sitting next to the invoice that matters.
The key challenges included:
- Invoices span many issuers, and a single extraction prompt does not work across all of them
- Supplier files can be large and complex, with PDFs of 200 pages or more in which only a few pages are the commercial invoice, so every page has to be processed reliably without slowing the pipeline or losing the pages that matter
- Multi-page invoices continue an order across pages, so items can appear before the page that names their order
- Item codes, order numbers, and units of measure are sometimes missing or ambiguous on the invoice
- Shipper and consignee names and addresses vary from the master list and must be matched reliably
- Load types (courier, air, and several sea modes) change which ports and fields apply
- Courier shipments need package-level data that lives on a separate airway bill, not on the invoice
- A booking must not be sent with unverified data, and operators need to see where the AI was unsure
- A reprocess after a fix must keep a clear history of attempts for the same email
- Inbound mail contains bounces, auto-replies, and non-shipment messages that must not create requests
- Master data (PO items, company codes) changes over time and arrives as spreadsheets
- LLM, OCR, and mail APIs fail intermittently, and the pipeline has to recover without losing emails
Hellmann needed a pipeline that could do the repetitive reading and keying for the TFS account, while leaving a person in the loop wherever the data was uncertain.
Our Approach
Bitontree built the platform for Hellmann as three connected parts: a Python document-processing service that owns mailbox intake and AI extraction, a FastAPI backend that owns enrichment, XML, reports, and supplier management, and a React operations dashboard for Hellmann's team.
Requests are stored in Azure Cosmos DB, and the dashboard and the backend read and update the same request records. The processing service writes requests and results, the backend enriches and delivers them, and operators review everything in the dashboard.
Our approach focused on:
- Using AI to triage and read documents, and keeping deterministic rules for matching, enrichment, and XML output
- Breaking document processing into a LangGraph workflow of small, testable steps
- Using more than one model and showing agreement as a confidence level
- Keeping supplier extraction rules in the database, so onboarding a new supplier does not need a deployment
- Making every step tolerant of retries, partial failure, and re-runs
- Giving operators one place to see the source document beside the extracted data
Core Features Developed
AI Mailbox Triage
The processing service polls the shared mailbox every few minutes and handles several emails in parallel. Before any document work starts, a language model reads the sender, subject, body, and attachment flag and decides one of three actions: process shipment documents, refresh the PO master from an attached spreadsheet, or ignore the message.
This keeps bounces, auto-replies, and chatter from becoming freight requests. The classifier treats the email body as untrusted text and does not rely on sender block lists. If triage fails, or the email matches a dashboard reprocess request, the system defaults to processing it so nothing is dropped.
Both Gmail and Microsoft Graph mailboxes are supported behind one provider interface.
Commercial Invoice Page Detection
Attachments can be PDFs, ZIP archives with nested files, or scanned images. The service converts each page to an image and uses Azure Computer Vision OCR to label pages as commercial invoice, packing list, or other.
Only invoice pages continue to extraction, which also cuts LLM costs because the models read a handful of pages instead of an entire file that may run to hundreds. The filter is built to avoid data loss: if a page cannot be classified it is kept rather than dropped, and when there are several candidate documents but none is an invoice, extraction stops instead of guessing.
Multi-Issuer Invoice Extraction
We built dedicated extraction processors for the invoice layouts TFS sees most, covering more than a dozen issuers, plus a generic extractor for everything else.
Processors use few-shot annotated example invoices and, where useful, a second model to cross-check the result. Multi-page invoices are stitched together: items that appear before their order number is known are held and attached once a later page supplies it.
Database-Driven Supplier Onboarding
New suppliers do not need new code. A supplier's extraction prompt, its few-shot guidance, and its annotated example images are stored in the database and loaded at run time.
In the dashboard, an operator creates a supplier, writes instructions, and uploads annotated images. A Gemini-based step generates the extraction prompt, and editing the instructions or images regenerates it. This makes supplier coverage something operations can grow without waiting for an engineering release.
Shipper And Consignee Matching To TFS Master Data
Extracted parties are matched to the TFS company list with fuzzy matching that scores the name and the address together. Exact matches win ties, the delivery-to address is preferred over the invoice-to address, and a minimum score avoids weak matches.
The match returns the TFS company code, country, and the origin and destination port codes appropriate to the load type. Operators can override the supplier code, and the override persists across reloads. When names or address flags change in the dashboard, the backend recalculates the codes automatically.
PO Master Enrichment
Extracted invoice lines are enriched against a PO master database by PO number, item code, and quantity, with a SKU fallback. A matched line gains CBM, weight, dimensions, pack type, non-stackable flag, incoterm, vendor, and other booking fields.
When one PO and item combination matches several master rows, the system hands out each row's unique drop value in turn. Every enriched field carries a confidence flag, and unmatched lines are marked low confidence for review. A supplier-code conflict across items raises a clear mapping error instead of producing a wrong booking.
Missing Item-Code Recovery
When an invoice line has no item code, the platform tries to recover it in stages: a re-parse with a second model, a description match against the PO master, then a fallback on exact quantity and unit price. Each recovered code is labelled with the confidence of the method that found it.
Multi-Model Confidence Scoring
Key fields are extracted by more than one model and compared. All models agreeing gives high confidence, two of three gives medium, and disagreement gives low.
Operators see this confidence on the request, so review time goes to the fields that need it.
Courier Airway Bill Processing
For courier loads, the service finds DHL and FedEx airway bill pages, identifies the carrier, and extracts per-package details such as dimensions, weight, customs value, currency, and the US ITN number where an export filing applies.
Airway bills can arrive as a separate PDF from the invoice, so a second pass reads every original attachment and merges the packages into the request. The airway bill is also used to recover a missing PO number.
Vendor Booking XML And Smart Order Delivery
The backend builds a Vendor Booking XML document: customer, shipper and consignee nodes, transport mode, ports, load type, cargo ready date, incoterm, and one entry per order line with packaging, dimensions, weight, CBM, price, and currency. Courier requests include a courier packaging section.
Hellmann's operators can generate the XML for review or send it to Hellmann's Smart Order system over FTP from one button. Each send is logged on the request with the file name, size, and time, and the regional contact is notified by email. Currency codes are validated as three-letter ISO codes, and placeholder values such as "unknown" are kept out of the file.
Operations Dashboard
The React dashboard gives operators a working view of every request: a recent-activity dashboard, a searchable and filterable request list, and a detailed request view.
The request view shows the original documents in a built-in PDF viewer, including TIFF rendering and ZIP extraction, beside the extracted data in tabs. Operators can edit values, change Smart Order address codes, download Excel, or send the XML.
A PO item search lets users look up master data by PO number, item code, or quantity.
Reprocessing With A Clear History
If a request needs another pass, an operator triggers reprocess from the dashboard. The backend records the intent and marks the email unread, and the mail service picks it up and creates a new request that is linked to its parent.
The dashboard shows the chain of attempts, a queued or processing status, and a short cooldown that prevents duplicate reprocess clicks.
Reports And Notifications
The backend produces Excel reports for single requests and for all requests, with processing times and Smart Order send times in UTC and Singapore/Philippines time. Reports can be emailed to the operations team.
The processing service sends a summary email for each completed request, with attachments, routed to the right regional contact from a country-to-contact list. A separate alerting module raises outage alerts only after repeated failures, with throttling so a single incident does not flood inboxes.
Analytics
An analytics page shows request volume, pending and completed counts, completion rate, daily activity, and the suppliers with the most documents over a chosen time range.
Operational Resilience
Every language model, OCR, and mail call is wrapped with retries, exponential backoff with jitter, and timeouts. If an attachment cannot be fetched, the email stays unread and is picked up on the next poll. Mailbox claim labels prevent two workers from taking the same email.
Attachments are archived to Azure Blob Storage, and the PO master can be refreshed by emailing a spreadsheet, using header aliases and preserving leading zeros in supplier codes.
Technology Stack
| Layer | Technology |
|---|---|
| Frontend | React 19, React Router, Tailwind CSS, Axios, React PDF Viewer, PDF.js, UTIF (TIFF rendering), JSZip, SheetJS (xlsx), date-fns, Lucide and Heroicons, React Toastify |
| Backend API | Python, FastAPI, Pydantic, pandas, openpyxl, Jinja2, fuzzywuzzy |
| Document Processing Service | Python, asyncio, LangGraph, PyMuPDF, rapidfuzz, Pillow |
| AI / LLM | Azure OpenAI (GPT-5 family), Google Gemini 2.5 Pro, Anthropic Claude, multi-model confidence scoring, few-shot annotated prompts |
| OCR | Azure Computer Vision (Read) |
| Database | Azure Cosmos DB (freight requests, PO master, users, supplier configuration) |
| Storage | Azure Blob Storage |
| Gmail API, Microsoft Graph | |
| Delivery | Vendor Booking XML (XSD-defined), FTP delivery to Smart Order |
| Authentication | Token-based login with protected routes |
| Deployment | Azure Static Web Apps for the dashboard with Azure Pipelines CI, Azure-hosted Python services |
| Testing | pytest for address import, currency, Excel, PO enrichment, supplier mapping, and XML reference rules |
Product Architecture
| System | Technology | Role |
|---|---|---|
| Document Processing Service | Python, LangGraph, Azure OpenAI, Gemini, Azure Vision | Mailbox polling, AI triage, invoice page filtering, classification, extraction, matching, confidence scoring, courier packaging, notifications |
| Backend API | Python, FastAPI, Cosmos DB | Request APIs, PO enrichment, supplier and address mapping, XML generation, FTP delivery, Excel reports, supplier portal, analytics |
| Operations Dashboard | React, Tailwind CSS | Request review, document viewer, data editing, reprocess, send to Smart Order, supplier setup, PO search, analytics |
The Result
Bitontree delivered an AI-assisted inbound shipment booking platform for Hellmann's TFS account. It takes supplier invoices from a shared mailbox to a delivered Vendor Booking XML in Hellmann's Smart Order system, with an operator review step in between.
The system covers email triage, invoice detection, multi-supplier extraction, master-data matching, PO enrichment, courier airway bill handling, XML delivery to Smart Order, reprocessing, and reporting.
| Outcome | Result |
|---|---|
| Mailbox intake | Shared mailbox monitored continuously, with AI triage separating shipment emails from noise |
| Document handling | PDFs, ZIPs, and images reduced to the commercial invoice pages that matter |
| Supplier coverage | Dedicated processors for more than a dozen issuer layouts plus a database-driven extractor for new suppliers |
| Master-data matching | Shipper, consignee, and ports resolved against TFS company codes by load type |
| PO enrichment | Invoice lines enriched with CBM, weight, dimensions, pack type, and incoterm from the PO master |
| Courier loads | DHL and FedEx airway bills read for package-level data |
| Data quality | Field-level confidence from multi-model agreement shown to operators |
| Booking delivery | Vendor Booking XML generated and sent to Smart Order from the dashboard |
| Recovery | Reprocessing creates a linked new request and keeps the attempt chain visible |
| Reporting | Excel reports and analytics with UTC and local processing and send times |
| Platform foundation | Three-part architecture across processing, backend, and dashboard |
Business Impact
The platform replaces manual reading and re-keying of supplier invoices with a reviewed, AI-assisted workflow for Hellmann's operations team.
The final solution helped Hellmann:
- Reduce manual effort in reading supplier invoices and creating bookings, with around 80-85% of requests on average handled end to end without operator intervention, and a best week of around 95%
- Handle a wide range of supplier invoice layouts without a separate manual process for each
- Add new suppliers through configuration instead of a software release
- Catch uncertain data early through confidence levels and operator review
- Keep booking data consistent by enriching lines from one PO master source
- Deliver bookings to Smart Order from a single action
- Serve a demanding life-sciences customer account with faster, more consistent booking
- Keep a traceable history of every attempt, send, and correction on a request
- Give operations visibility into volume, completion, and supplier activity
- Build a scalable foundation for further inbound freight automation
Key Engineering Problems We Solved
Reading Many Invoice Layouts Reliably
No single prompt handles every supplier. We combined per-issuer processors, few-shot annotated examples, a generic database-driven extractor, and a second-model check, so coverage could grow by adding suppliers rather than rewriting the pipeline.
Stitching Multi-Page Orders
Invoices continue across pages, and a page can list items before the page that names their order. The processors carry the last known order forward and hold unassigned items until a later page supplies the order number.
Not Losing Data When Classification Is Uncertain
The document filter is built to fail safe. A page that cannot be classified is kept, a lone standalone document is processed in full, and extraction stops when several candidates exist but none is an invoice.
Showing Where The AI Is Unsure
Comparing several models on the same fields turned disagreement into a visible confidence level, so reviewers could focus on the lines most likely to be wrong.
Matching Parties To Master Data
Names and addresses on invoices rarely match the master list exactly. We scored name and address together, prioritised the delivery address, and kept an operator override that survives reloads and recalculations.
Recovering Missing Item Codes And PO Numbers
Missing item codes are recovered in stages with a confidence label for each method. A missing PO number can be taken from the airway bill. Neither is hidden: the recovery method is visible to the reviewer.
Handling Courier Documents That Arrive Separately
Courier package data lives on the airway bill, which may be a separate attachment that the invoice filter removes. A second pass over every original attachment finds it and merges it into the request.
Keeping Bad Data Out Of The Booking
Currency codes are validated, placeholder values are stripped from the XML, supplier-code conflicts raise explicit errors, and the XML reference is bounded to the target system's length limit.
Making Reprocessing Safe
Each reprocess creates a new request linked to its parent, with the attempt number, status, and cooldown shown in the dashboard. This avoids overwriting history and avoids duplicate runs.
Making Mail Automation Production-Ready
The pipeline combines claim labels, retry and backoff, unread-on-failure behaviour, and threshold-gated alerts, so temporary failures in a model, OCR, or mail provider do not lose an email.
Development Timeline
| Phase | Deliverables |
|---|---|
| Phase 1 | Mailbox polling, document conversion, OCR page filtering, first invoice extraction, Cosmos DB request storage |
| Phase 2 | Backend API, dashboard foundation, request list and detail views, PDF viewer |
| Phase 3 | Multi-issuer processors, shipper and consignee matching, PO master enrichment, confidence scoring |
| Phase 4 | Vendor Booking XML, FTP delivery to Smart Order, Excel reports, regional notification routing |
| Phase 5 | Courier airway bill processing, reprocess flow with attempt history, Gmail and Microsoft Graph support |
| Phase 6 | Database-driven supplier onboarding, address database refresh, analytics, hardening, and ongoing enhancements |
Why This Project Stands Out
Hellmann needed more than invoice OCR. Inbound freight booking depends on matching the right parties, ports, and packaging data for each load type, and on delivering a booking in the exact format Smart Order accepts.
Bitontree built the platform around that real forwarder workflow. AI reads and triages the documents, deterministic rules match, enrich, and format the booking, and Hellmann's operators keep control of the final send.
It also shows how a freight forwarder can use AI to serve a major customer account better: faster booking for the customer, and less repetitive keying for the forwarder's own team.