Augmented Maintenance
Augmented Maintenance

AI assists.
Judgement stays human.

An offline-first mobile app helping generalist construction field engineers diagnose equipment failure faster.

RoleDesign LeadEnd-to-end product design
Team2 designers (including me)1 manager
PlatformAndroidRugged phones, offline-first
TimelineAug–Oct 20263 months
ImpactSource · Business Analysis team
92%First-time fixes
−45%Task time
−32%Downtime
−15.6%Workforce costs
Clients
The problem

Expertise doesn't scale to every site.

Costly downtimeFew specialistsScattered knowledge
The solution

An assistant the engineer can check, correct and teach.

CheckCorrectTeach
The product

How it works

Human-in-the-loop

The engineer can overrule the AI. The AI learns from it.

The AI is right on paper and blind on site. Engineers steer it with what they see, the recommendation regenerates, and that feedback is saved as a signal for sites with similar conditions.

Learns from the field

The AI learns which signals to check.

After the first “humid day” report, the AI checks the humidity itself. High humidity doesn’t always mean corrosion, so it adds a tagged option beside the normal one. Every engineer gets the lesson.

The trust moment

Changed steps are marked. The reason is one tap away.

A dot marks every changed step. The Change log tab says why, only when asked.

The full story

Deep dive

The problem, the thinking and the design, in detail.

A machine stops.
A crew waits.

The crew lead has a phone, a fault code and a decision to make.

  1. A broken-down excavator with an engine-fault warning, on a construction site.

    01An excavator stops mid-shift.

  2. Three crew members wait beside the stopped excavator, thinking of the specialist they need.

    02The crew stands around, waiting.

  3. The site’s specialists are busy welding and fixing pipes on other jobs.

    03The site’s few specialists are busy on other jobs.

The problem

Expertise doesn't scale to every site

Field engineers lose time to three things.

Downtime is expensive

Every minute spent diagnosing delays the project.

Specialists are few and expensive

Sites have specialists, but not enough for every small fault. Keeping that many on hand costs too much.

Knowledge is scattered

Answers sit across manuals, service records and past repairs.

The idea

Generalists, backed by AI

Let generalist engineers handle the jobs they can, with help from an AI assistant, and call in a specialist only when the job truly needs one.

Less downtime, lower costs, and specialists kept for the complex, high-risk work.

The insight

The AI can't see the site

Everyone assumed the AI should give the answer. In our brainstorming calls, I pointed out the catch: the AI doesn't know what the engineer knows.

The example
The AI

A repair that's technically correct. It has no way of knowing about the heavy rain the night before.

The engineer on site

The ground is now muddy and slippery, so that repair becomes risky to carry out.

The rain example wasn't from a dataset. It came from asking myself what I'd notice if I were standing there.

Why the engineer can. A human can sense that something is off, even without pinpointing what. Past experiences, even unrelated ones, help people join dots.

The pushback

“Just feed all the context into the AI.”

My answer

Sensors aren't the way

To know if a site is stable after a rainy night, you'd need sensors in the ground. Not viable, not cheap.

Weather data isn't enough

It can say it rained, but not whether this patch of ground will hold.

The engineer is faster

They can look at it and judge in seconds, and they're the one standing on it. Expensive tech wasn't needed when the answer could be simple.

What it changed

“make the AI smarter”

“let the AI and the engineer decide together, in a way the engineer can trust.”

AI shouldn't replace expertise. It should amplify it.

The guiding question

What can a human on-site see and judge that AI can't?

Every decision on this project started here.

The users

Built for the crew lead

The crew lead makes the call, and the whole crew acts on it. They don't want to read manuals. They want the right answer, and they're the ones accountable if it's wrong.

Primary user

Generalist crew lead

Age
25–35
Training
Degree/diploma + vocational
Device
Rugged Android phone
Works
Outdoors, in shifts and crews
Breakdowns per shift
1–3 on a normal shift  ·  4–6 on a busy one

Don't make me read the manual. Tell me what's wrong, and show me why you think so.

— Generalist crew lead
Research

New domain, no assumptions

I knew nothing about construction going in. So I couldn't lean on assumptions or biases. Every decision came from the research.

Two findings shaped everything

Finding 01

Downtime is expensive

Downtime costs the construction industry a lot.

Finding 02

The model fits on the phone

A small language model can run entirely on the phone, with no internet.

Offline-first became possible
For engineers at remote sites, with no signal.
Constraints

Designed for the edge of the network

No signalThe AI runs on the device, with the manuals for every machine on site preloaded.
Harsh conditionsRugged phones, outdoors, in every kind of weather.
High stakesOne decision, and a whole crew acts on it.
White-labelOne product, rebranded and customised for each client.
No client accessWe designed from the business requirements and the research team's insights.
Key decisions

The engineer overrules. The AI learns.

Human-in-the-loop by design: a human can check and change every AI output.

The engineer can overrule the AI

Why: the AI is right on paper and blind on site.

A Context pill in the prompt bar lets engineers steer the AI with what they see, and it regenerates. One extra step buys a safe decision.

It learns from every override

Why: the field knows things the manual doesn't.

Tell the app it rained last night, and it factors that into the day's other repairs. When engineers correct the AI, that feedback loop improves recommendations at sites with similar conditions.

Fewer novel breakdowns

Why: a fault one crew solved shouldn't be new to the next.

Every override and every piece of saved context makes the next recommendation better. A fault that was new at one site becomes a known fault at the next, so engineers need specialists less often.

Iteration

Three versions, and the test that changed v3

v1

What changed

AI recommendations with sources

No way for the engineer to correct the AI
v2

What changed

Added context, in a separate box

5 of 9 testers got lost
v3

What changed

Context pill in the prompt bar

Designed to fix it

5 of 9testers got lost in v2. The context box felt small and restrictive.

The final experience

One breakdown, start to finish

  1. 1

    The wrong answer

    The engineer gets a recommendation that won’t work on this site.

  2. 2

    Add context

    They tap Context and add what they see. The AI regenerates, with options that fit.

    ↺ Still not right? Add more context.
  3. 3

    Review the plan

    They review the whole plan and tap a step to check its source.

  4. 4

    Push back on a step

    They push back on one step. Only what must change does, and it’s marked.

    ↺ Back to the plan.
  5. 5

    Start the procedure

    They start the procedure. Mid-repair context goes on the current step.

  6. 6

    Next time

    Next time, the AI already checks for it.

The flow in detail

Overrule, and it learns

One repair: a hydraulic hose leak at the boom of excavator EX-14, next to a trench.

Engineers don’t volunteer context. It feels so natural that they forget it’s there, until the AI gets it wrong. So every wrong answer draws out what they know. That’s why overruling is a loop, not a form.

“The engineer wants the right solution that they themselves may not know yet. They know that the current solution is not right, but giving it certain points of context may yield a better solution.”
“Good design is invisible. It feels so natural that it is hardly ever noticed. It is not loud.”
  1. The wrong answer

    The AI recommends a 2-hour circuit replacement. The ground is too wet for the lift truck. The engineer can’t name the right fix, but they know this one won’t work. They tag the card and add context.

    “Knowing that the recommendation is wrong or not feasible is a call the generalist can almost always take confidently. They know their construction site.”
    The engineer taps the Circuit Replacement card and types: the ground is too wet and unstable for the lift truck, and they need a fix without the full 2-hour job.
    Overrule composer, tagged to the recommendation
  2. A new answer from their context

    Two options come back: the standard fix, and a 55-minute temporary fix tagged “Unstable ground”. No Regenerate button. Every retry carries new context.

    “Minimising downtime is our goal mainly.”
    Two recommendations: the standard Circuit Replacement, and a Hose Replacement tagged Unstable ground, 55 minutes.
    Two options, one tagged by condition
  3. Review before starting

    The engineer sees the whole plan first. Tap a step for its source: the exact manual page, or the past case it came from.

    “Temp fixes may be off-manual. That is why we need the AI+human combo. Humans are trained to know what is safe and what is unsafe.”
    Hose Replacement details: three steps with manual and case sources, and a Start procedure button.
    Recommendation details, step cards with sources
  4. Push back on one step

    The overrule icon scopes context to a single step. Only what must change does, earlier steps included.

    Context entered on a single step, Shore trench edge: shoring plates may not be stable enough for the placement.
    Step-level overrule
  5. The trust moment

    A dot marks every changed step. A Change log tab says why, and only exists on changed steps.

    “Explanation for change on demand. Not shown upfront.”
    Change log on Step 1, Recalibration: two context entries marked with a dot, and the original AI recommendation.
    Change log on a changed step
  6. Procedure mode

    Full screen, one step at a time, no back swipe, on a company-managed, regulated device. New context goes in through the current step’s overrule control.

    “The procedure should be in focus, with no distractions.”
    Procedure mode, Step 1 Recalibration, with the context composer open on the current step.
    Add context mid-procedure
  7. Stop

    Stop opens a popup: tap and hold to stop, pause, or escalate. The hold prevents accidents. Escalate is a ghost button because it’s the last resort.

    Stop popup: tap and hold to stop, pause, or escalate to specialist.
    Stop popup
  8. Next wet day

    After the first “wet ground” report, the AI checks weather data itself. Rain doesn’t always mean unstable ground, so it shows a tagged option beside the normal one. Every engineer gets the lesson.

    “What stops a wrong context reaching everyone is everyone’s judgement. Human judgement.”
    A new breakdown on a rainy morning: a note says heavy rain at this site last night, ground may be unstable, and an option was added from a past field report. Two cards follow: the standard Circuit Replacement and a Hose Replacement tagged Unstable ground.
    Next wet day, tagged recommendation

Recreated screens. All step copy is placeholder.

Trade-offs

Decisions I’d defend

No Regenerate button

A retry without new context teaches nothing. Trade-off: a few seconds of typing.

Reason on demand

Keeps the step in focus. Trade-off: one tap.

Step cards hide detail

Steps hold long instructions. Trade-off: one tap in, one step in focus.

Lessons apply company-wide

The field knows what no site does alone. Trade-off: a wrong lesson could travel. Engineer judgement and the manuals catch it.

Left out on purpose: escalating to a specialist, and the stop summary.

Impact

What changed in the field

Source · Business Analysis team

First-time fix rate
92%
Fixed first timeSecond visit

Most faults are now fixed in one visit, with no trip back to site.

Maintenance task time
−45%
Time saved

The same repair in a little over half the time.

Equipment downtime
−32%
Back on the job

Machines are back at work sooner, so crews and projects wait less.

Workforce costs
−15.6%
Saved

From pairing generalists with AI instead of staffing every job with specialists.

Potential savings

A year of workforce costs, before and now

$17.85Mtotal potential savings a year$15.6M workforce + $2.25M operational
Before100% specialists, no AI
$100M
Now70% generalists with AI, 30% specialists
−$15.6M
$84.4M
Try it with your numbers
Workforce savings are 15.6% of the cost before, when generalists with AI assistants take on the routine work. Operational savings are scaled from the business analysis team's estimate of $2.25M a year per 100 machines.

Specialist interventions are falling month on month, as field feedback and saved context improve the recommendations.

Reflection

What I took away

Design for what the AI can't see.

The hardest part was finding the edge cases. I started by learning what a construction site actually involves, then asked what only a human there could know.

AI shouldn't replace expertise. It should amplify it.

Every decision on this project came back to that.

Start by asking why the human matters.

That one question gave me every edge case: what can a person on site see and judge that AI can't?