Designing the review layer for an ML model.

Designing the review layer for an ML model.

An ML-assisted workflow for schedule tagging, designed so the people fixing the model’s mistakes could see how sure it was on every row. Their corrections became the training data for the next version of the model.

An ML-assisted workflow for schedule tagging, designed so the people fixing the model’s mistakes could see how sure it was on every row. Their corrections became the training data for the next version of the model.

Close-up of the review table: five schedule activities, each with its suggested WBS code and a confidence badge at 62, 55, 45, 47 and 68 percent.

ROLE

Staff Product Designer

PROJECT SPAN

2025–2026

TEAM

1 product manager, 2 ML engineers, 1 UI engineer

SCOPE

The human review layer for schedule tagging inside Hexagon Construction Analysis (formerly known as Avvir), Hexagon’s platform for checking construction progress against the BIM: correction workflow, review interface, training-data loop. Hexagon’s AI Hub built the initial model, and our ML engineers trained, developed and implemented it from there.

STATUS

The beta shipped and ran on customer projects; the in-platform review table was prototype-tested with the Customer Service team (the internal team that set up projects and tagged customer schedules) and shipped.

IMPACT

Replaced hand-tagging thousands of schedule line items with a human-in-the-loop review designed to reduce the work over time.

Tagging sat inside project setup and was too irregular to automate with rules.

Tagging sat inside project setup and was too irregular to automate with rules.

WBS codes are how construction materials and tasks get identified, and they connect the schedule, the BIM (the 3D blueprint) and our reporting. Every general contractor formats their schedule differently, so someone had to read each row and decide what it was.

I was on a team spun off to reduce the org’s operating costs, so the PM and I went through the Customer Service timesheets. Tagging and schedule updates took about 20 hours per project in ideal conditions, across up to four people. Our planning docs set the goal at 25 percent less time within six months, and the timesheets that would show it were still coming in.

TWO REASONS THIS NEEDED MACHINE LEARNING

On one project, only 1 in 5 activities appeared in every schedule version.

On one project, only 1 in 5 activities appeared in every schedule version.

The rest were added, renamed, removed, or split between versions. Tags couldn’t carry forward. Each revision started over from scratch.

590

590

unique activities across the baseline and six revisions

118

118

(20%) present in all seven

121

121

(21%) appeared in only one

RENAMED · Floor Trusses - 3rd Floor

Baseline schedule, February 21 2022: row 123, Floor Trusses - 3rd Floor, outlined in red between the 2nd and 4th floor rows.

Baseline / Feb 21, 2022

Revision 3 of the same schedule, September 13 2022: the same activity renamed Floor Trusses/ sheathing - 3rd Floor, now row 55.

Revision 3 / Sep 13, 2022

There was no shared format to write a rule against.

There was no shared format to write a rule against.

Naming changed from customer to customer, project to project, even revision to revision.

Customer A schedule in Excel: activity names such as Sprinkler Head Ceiling Drops - FSA - 2nd Floor - CL 1-5/C-A, built from area codes, grid lines and dashes.

CUSTOMER A · Names read Sprinkler Head Ceiling Drops - FSA - 2nd Floor - CL 1‑5/C‑A · area codes, grid lines, dashes

Customer B schedule in Excel: activity names such as Pour Concrete - Pour 1 - Floor 1, built from pour and floor with no grid.

CUSTOMER B · Names read Pour Concrete - Pour 1 - Floor 1 · pour and floor, no grid

QAing algorithm guesses at scale.

QAing algorithm guesses at scale.

Hexagon Construction Analysis (formerly known as Avvir) was powerful for customers, but every project took a lot of QA, setup and other manual work to give them results they could rely on. The Customer Service team grew out of the need to deliver those verified results, and as the customer list kept growing, the team’s size and capacity didn’t. We had to take some of that process off the team, sometimes by giving a step friendlier UX and making it self-serve for customers, and a lot of the time with AI, ML and computer vision models. Schedule tagging was one of them. Its data was at such a large scale that reviewing each and every row wasn’t feasible, and approving every suggestion one at a time wasn’t going to work either.

The same three questions kept coming up: what did it decide, how do I know it’s right, and how do I fix it if it’s wrong.

“Certainty % would be really helpful so we could just focus on the things in need.”

QA tooling session notes, Jan 2023

“I am looking for things the algorithm didn’t catch.”

QA Discovery, Oct 2022

“If I am wondering how the algorithm decided something, someone likely didn’t do a good job with QAQC.”

QA Discovery, Oct 2022

Quoted from my live session notes at the time; spelling and punctuation regularized, wording unchanged.

I pulled four requirements out of all of it. Show what the system claimed, show how sure it was on each option, make it checkable by eye, and make the override one click.

The model shipped before the interface.

The model shipped before the interface.

Leadership wanted the model in real use quickly, so it needed a path that captured corrections. The Customer Service team also needed the old way left alone. Our two ML engineers were also our strong back-end engineers, and only one engineer was comfortable with the front end, so the full in-platform version was a bit more than we could build quickly. A partial version would’ve still left them doing some of the work in Excel. I had doubts about how stripped down the beta was and agreed to it anyway, since the users were our own team and the engineering lift was small. I kept designing the full table in parallel so it’d be ready when the budget was.

A beta panel was added to the existing spreadsheet workflow: upload a spreadsheet file, download suggested codes, correct them, upload the corrected file, then upload the full schedule as before. The single upload became three uploads and a download. Corrected files were stored to use as training data for the model.

It didn’t take enough judgment off them to justify the logistics it added. They still reviewed every code, and now they had to also work out which upload field to use. The algorithm was right some of the time at first.

The process, before and during the beta.

The process, before and during the beta.

Every schedule update resets the entire process. Most projects update multiple times.

BEFORE

Every schedule update resets the entire process. Most projects update multiple times.

The manual process: the customer sends a schedule, the Customer Service team codes it in Excel, uploads it, and the next schedule update resets the whole thing.

The model was in the loop. Getting the work in was next.

PHASE 1 BETA

The model was in the loop. Getting the work in was next.

Every correction still went out to Excel and back in through a re-upload.

The beta in production. Three new buttons above the two that were already there: upload an unprocessed TSV, download the suggestions, correct them in Excel, re-upload.

WHAT ACTUALLY SHIPPED

The beta in production. Three new buttons above the two that were already there: upload an unprocessed TSV, download the suggestions, correct them in Excel, re‑upload.

What engineering built against.

What engineering built against.

Our lead engineer and I wrote the acceptance criteria the beta was built against. Seven Gherkin scenarios from June 2025.

ACCEPTANCE CRITERIA

Our lead engineer and I wrote the acceptance criteria the beta was built against. Seven Given/When/Then scenarios from June 2025.

Engineering gave me the reasons an upload could fail so I could write copy.

THE ERROR CASES

Engineering gave me the reasons an upload could fail so I could write copy.

Some design iteration for an attempt to simplify the upload fields (not used).

NOT USED

I tried a simpler set of upload fields here. The red notes are my questions at the time: could the existing upload buttons take the file, and were we training the model live or just collecting files for a v2?

Three things kept people from using it.

Three things kept people from using it.

Adoption among the Customer Service team was low. On the surface, the beta added steps to a process that already frustrated them. Volunteering corrections to a model was a hard sell when many on the team were wary of giving data to something that could make them less needed. The payoff stayed invisible: people corrected output, submitted data, and nothing on screen showed what it was building toward. What usage it got came from a few team members who agreed to trial it on some of their projects.

What leadership actually saw.

What leadership actually saw.

What I used to walk Customer Service leadership through the changes. Their team’s review still ran through a spreadsheet.

BEFORE AND AFTER

What I used to walk Customer Service leadership through the changes. Their team’s review still ran through a spreadsheet.

I kept these notes in the Figma file, beside the beta panel’s upload steps.

WORKING NOTES

I kept these notes in the Figma file, beside the beta panel’s upload steps.

Tags survive the next schedule update.

Tags survive the next schedule update.

The schedule changed, and the Customer Service team tagged the whole thing again from scratch as fast as they could. In the version I designed, you upload the new schedule and the previous tags persist onto it, so a row that moved keeps the code someone already corrected, and the work narrows to what’s new or heavily changed. The engineering team worked out how to make it actually happen, which was the harder problem.

Edit to reject, do nothing to accept.

Edit to reject, do nothing to accept.

One wrong code had layers of damage under it. Analysis built on analysis, so a single bad match cascaded wrong data into surfaces all over the product. The hard part was finding a review cadence that fit both the model we had at rollout and the one we expected to have later. The open question was what review should cost.

Most tools that show a person a model’s guess want an accept on every item. Grammarly asks per suggestion. One accounting tool pre-fills the category, shows a confidence badge, and still needs an approval on every row, and its users trade tips on how to approve a whole group at once. Invoice-processing tools handle the same problem differently: a confidence threshold decides whether a person looks at all, and the person only fixes what’s wrong. An approve step needs a reject step beside it, and an approve click on a thousand rows is just moving the old work to a new flow. The table pre-fills every line with the model’s best guess, and the only action is correction, as needed. A correct guess needs nothing from the user at all, and making that the default is the decision I think saves the most time against this work. Approving and rejecting are implied by whether a row gets corrected: one less decision on every line. Engineering shipped the table in my last week, so whether it moved adoption was never measured. The prototype test is in the next section.

A single save persists the schedule, sends every correction to the ML server as labeled training data, and says so on screen. From there the ML team anonymizes it, retrains in a controlled setting, and ships model updates.

Every correction becomes training data.

Every correction becomes training data.

My diagram of the training loop: corrections are stored to train the ML algorithm and applied to the project.

PHASE 2

My diagram of the training loop.

The review table pre-filled with model suggestions, each carrying its confidence badge, and the save confirmation naming what was sent back for training.

IN THE PRODUCT

The review table pre-filled with model suggestions, each carrying its confidence badge, and the save confirmation naming what was sent back for training.

A threshold decides what’s worth a person’s time.

A threshold decides what’s worth a person’s time.

Above a threshold, suggestions aren’t surfaced for review, so they’re accepted by default. Below it, rows are flagged and a filter collapses the table to only what needs human eyes. The threshold is adjustable, because what’s safe to skip differs from one reviewer to the next and moves as the model improves. Reviewing everything was on the table, and so were fixed thresholds, but neither would work with the changing need. The slider let each reviewer set their own comfort level as the model improved.

See how sure it is, fix it in one click.

See how sure it is, fix it in one click.

Transparency: I surfaced the match score the algorithm produced for its own guess, meaning how close a match it found across the factors it weighs. The spread between guesses was wide enough that a label alone would have hidden it.

Inspectability: Every option in the dropdown carries its own match percentage, so a reviewer can see how close the call was.

Visual verification: The code, its plain-English description, and the badge all sit on the row, so a reviewer judges the call without opening anything.

One-click override: A correction is worth more than one row. A customer might split plumbing into line items by wing or floor when all of it maps to one trade, so fixing one code offers to fix the rest.

This interaction was a little too detailed to validate with a Figma prototype, especially the autocomplete search field, so I built it out in React and TypeScript with Claude Code and populated it with sample data. I tested it with three Customer Service team members in January 2026, each working a schedule as if it were a live project. What I wanted to know was whether the badge and the dropdown made sense to someone who’d never seen a confidence score. All three got through the simulated tagging exercise. One of them, thinking out loud: “it tells me which ones need my review so I don’t have to check every row.” Search covers any code the model hadn’t guessed. Type a few characters and the full code library is there under the ranked list.

I built the predictive dropdown to the component standards of Pangaea, Hexagon’s design system, and handed it to the platform team.

From the whole schedule to the rows that need review.

From the whole schedule to the rows that need review.

The threshold popover at 70 percent: 990 activities auto-accepted, 36 flagged for review.

SETTING THE THRESHOLD

The threshold popover at 70 percent on sample data: 990 activities auto‑accepted, 36 flagged for review.

The Needs Review filter collapsing 1,026 activities to 36.

FILTERED TO NEEDS REVIEW

The Needs Review filter collapsing 1,026 sample activities to 36.

Three confidence states, told apart by shape.

Three confidence states, told apart by shape.

Every suggestion carries a shape-coded badge, so the state reads without relying on color.

CHECK · 94% · at or above the default threshold of 70, accepted by default

CHECK

94% · at or above the default threshold of 70, accepted by default

DASH · 62% · 50 to 69, below the threshold, routed to a person

DASH

62% · 50 to 69, below the threshold, routed to a person

TRIANGLE · 45% · below 50, routed to a person

TRIANGLE

45% · below 50, routed to a person

Each shows how a row reads in the review table, at its own score.

Pick a code, then apply it to similar rows.

Pick a code, then apply it to similar rows.

Two moments that cannot share a frame: the propagate prompt fires only after a code is picked, and picking closes the dropdown.

The dropdown, with AI suggestions ranked by confidence.

RANKED ALTERNATIVES

The dropdown, with AI suggestions ranked by confidence.

After the pick, the system offers to propagate the code.

APPLY TO SIMILAR

After the pick, the system offers to propagate the code.

What I’d do differently.

What I’d do differently.

There was a gap between what we could build on that timeline as a product squad and what I knew the users needed, so we counted on the users being our own team and willing to try it. Starting over, I think I’d demo the model running in some kind of low-res UI, just to show the Customer Service team what they were contributing to. I’d also want to look at the uploading with engineering again, because the pipeline already knew which files the model had processed and I wonder if one upload field could have worked off that.

What I’d do next.

What I’d do next.

This work tagged schedule rows to WBS codes. The BIM is tagged with the same codes, which is what ties it to the schedule. That is another of the major setup processes: an architect updates the file and someone retags the elements by hand. The team had started scoping the same matching model for BIM elements, and I was designing the review loop for it.

Two versions of a building model overlaid in green and red, beside a Match Threshold slider and paired element lists from each version.

MODEL TO MODEL

In this unfinished comp, elements above the match threshold keep their tags in the new file and the rest go to a reviewer.

Existing tags would carry over to the architect’s new file, the way they carry over to a new schedule. That’s where I’d pick back up. I’d also show which keywords each guess matched on, an idea we talked about as a future release and never got around to scoping.

Matt Klein

© 2026

Matt Klein

© 2026