Abtin Aryanpour

Image Stitcher

Merges out-of-order photos from several drones, each tagged with a GPS (Global Positioning System) position, into one top-down map that sits in the right place on the Earth.

10 of 10simulated missions built one connected map, with photos lined up to 4.0–8.3 cm on average. Tested in simulation only.

Status
Merged Merged into the team’s main branch on November 18, 2026, after four rounds of review. The last round shortened the code and tightened refinement and the GPS checks.
Role
Built for the autonomy software team at the Waterloo Aerial Robotics Group, from October 2026. I wrote the whole pull request: the stitcher, the simulator that makes test missions with known true positions, the 130 tests and the evaluation script.

Why it exists

Drones fly back and forth over a field taking photos, and each one reaches the ground laptop whenever its drone sends it. The photos arrive shuffled, from several drones, with GPS that’s metres off.

GPS finds which photos should overlap, image matching lines them up, and all the GPS readings together place the finished map on the Earth.

The finished map after all 91 photos
One simulated mission, saved after 20, 40, 60, 80 and all 91 photos, each drawn in the same GPS frame. The stepped outline is the edge of the stitched photos, and only the painted ground markers are kept inside it.

From shuffled photos to one map

One simulated mission: where each of the 91 photos really landed, shown in the order they reached the ground laptop.
  1. Check. Each photo is checked for blur, exposure and tilt. The ones that pass are undistorted for the lens.

  2. Find neighbours. GPS says which photos should overlap, even when they arrive out of order from different drones.

  3. Turn north-up. Each photo comes with the drone’s yaw, so it’s turned north-up before matching. If yaw is close to a quarter turn it just gets that quarter turn, which loses no detail.

  4. Match. XFeat, a learned feature detector, finds the same ground points in both photos. RANSAC (random sample consensus) keeps only the matches that agree with each other and throws out the rest.

  5. Refine. A least-squares solver nudges every photo at once until the total mismatch between overlaps is as small as it can be. It works like bundle adjustment, the standard way mapping software corrects many photos together.

  6. Place. All the GPS readings together put the finished map in the right place on the Earth. A cluster is only fitted to GPS once both GPS and the images show its photos are spread out.

The data and the method

Where the photos came from
There are no real drone photos yet. I wrote a simulator that makes every test mission. It draws a random 120 by 90 m field of blobs, rectangles, lines and noise, and cuts photos out of it as a camera 25 m up would see them, lens distortion included, using the calibration of a 640 × 480 Arducam camera. Three drones fly back and forth in rows. The simulator adds the errors a real flight has: GPS 1 or 3 m off, altitude and heading slightly off, compass error, and photos arriving shuffled. Because it made the photos, it knows where each one truly was.
What was tuned and what was scored
Nothing in the stitcher is trained on these photos, so there’s no training set. Settings such as the blur limit were picked on simulated photos. The results come from 5 simulated fields, each flown at 1 m and 3 m of GPS error. They come from the same simulator, and two of them are also used in the blur check’s tests, so there’s no set held back and scored only once at the end.
What kind of method it is
It isn’t a model trained for this project. XFeat is a neural network its authors already trained, used as-is with their published weights, only to find matching points. Everything after it is fixed geometry: RANSAC throws out matches that don’t agree, and a least-squares solver lines all the photos up. The photo checks are hand-set limits on blur, exposure and tilt.
What it outputs
A map image, with a file giving its GPS corners and how many metres each pixel covers, plus a list of rejected photos and why. To score a simulated mission, each placed photo’s corners are compared with where they truly are. That gives the alignment and map-position rows below. A mission counts as one connected map when every placed photo ends up in a single group.

One connected map in 10 of 10 simulated missions

The simulator knows where every photo really was, so each run can be scored against the truth instead of by eye. Alignment is how far photos sit from their true positions relative to each other, averaged over each mission. Map position is how far the whole map sits from where it should be on the Earth.

All simulated: 10 missions of 91 photos from 3 drones, arriving shuffled, flown east-west, with 5° of simulated compass error.
One connected map10 of 10 missions, at 1 m and 3 m of GPS error
Photo-to-photo alignment4.0–8.3 cm on average, about 1 to 2 pixels
Least-squares refinementMisalignment of 13.6–69.8 cm without it, 4.0–8.3 cm with it
Map position against the truth0.162–0.307 m at 1 m of GPS error, 0.472–0.935 m at 3 m. Closer than the GPS error itself
Quality checks0 of 910 evaluation photos wrongly rejected. Blur check wrong on 0 of 2,730 photos under 9 conditions (5 sharp, 4 slightly blurred; none combine noise and blur)
Speed0.13–0.19 s per photo on a laptop CPU, down from 0.42–0.75 s before yaw
Bad inputA broken message or failed map save is logged and skipped. A photo with a GPS glitch, or a position outside valid latitude and longitude, is rejected instead of stretching the map. The first 3 photos are checked against each other, so one wrong first GPS fix can’t become the map’s origin and get every later photo rejected
Tests130 stitcher tests, plus a ground-truth mission simulator

The connected-map, alignment, map-position and speed rows rerun with one command from the pull request.

What was hard

Rows wouldn’t connect

The drones turn 180 degrees at the end of each row, and XFeat isn’t rotation invariant: it can’t match a photo against one that’s upside down. In one simulated pair of photos across a row turn, only 10 of 612 candidate matches survived RANSAC. With the photo turned 180°, 317 of 714 did. The first fix extracted features at four quarter turns. When review confirmed every photo would come with yaw, that became one north-up extraction per photo, which is where the speedup came from.

Before the fix: of 612 candidate matches between these two photos, only the 10 joined by lines survived RANSAC.After the fix: the same photos, with the left one turned 180° before matching. 317 of 714 candidate matches survive. Simulated photos, ground markers only.
Seconds per photo, simulated missions, on a laptop CPU.

A compass can’t be trusted to place photos

Yaw, the compass heading, has two jobs. It lines photos up for matching, and it orients clusters too small to fit to GPS. Final positions still come from the matches and GPS.

A reviewer caught a worse problem. GPS noise alone could pass the old spread check and blow up the map’s scale. Now the images have to show the photos are apart too.

A later version of a safety check depended on yaw, and a 180° yaw offset split one simulated map. It now compares distances, which needs no rotation. With constant yaw offsets of 0, 90 and 180°, 12 of 12 simulated runs formed one map.

Photos almost in one spot, GPS scattered by noise. Fitted to those dots, the map would stretch out to the dashed outline.
Photos really spread out, GPS close by. The fit leaves the map where it is.
A diagram, not data. Squares are photos as the images place them, dots are their GPS readings, and the dashed outline is the map after fitting it to those dots.

Refinement made the map worse

The first version made errors worse, not better, because a huge penalty weight on the pinned photo (the one held still so the map can’t drift) made the solver unstable. Taking that photo out of the values the solver adjusts fixed it.

After the fix: average misalignment of photos relative to each other, across simulated missions.

In the fourth round, the reviewer asked me to keep matches that conflict with a photo’s other matches out of refinement. I measured first. Cutting every matched point that sat a few pixels off made alignment worse, 4.9–14.6 cm on average against 4.0–8.3 cm, because before refinement removes drift, correct matches can sit up to about 30 pixels off. So only clear conflicts are dropped: wrong matches over repeating ground, hundreds of pixels off. Both figures are from the simulated missions.

Good matches split the map

The stitcher builds clusters, groups of photos that line up with each other, and merges two when a new photo bridges them. Small clusters with noisy GPS rejected good matches. Now a cluster’s GPS check only counts once the cluster spans four times the GPS error, and a simulated repeating-ground test checks that wrong matches stay blocked.

The blur check rejected good photos

Sharp and blurred scores overlapped, so no limit could work. The new score compares a photo with itself at half resolution, in each direction. My own review before pushing showed sensor noise could fool it, so a 3×3 median filter runs first. Even mild noise can still let a motion-blurred photo through in simulation, and the limits are tuned on synthetic photos.

Sharp: sharpness 0.53Sharp0.53 kept
Slightly out of focus: sharpness 0.23Slightly out of focus0.23 rejected
Moved 5 px while shooting: sharpness 0.20Moved 5 px while shooting0.20 rejected
The same patch of one simulated photo, enlarged 8×, with its sharpness score. The limit is 0.32. The two blurs are the smallest the check has to catch.

Where it still falls short

Every result is from simulation. There are no real flight photos yet: the drone’s camera software doesn’t log roll, pitch or yaw yet, so its photos are rejected for now, and the lens calibration still has to be checked against the camera actually flown.

Neither the yaw convention nor the camera mounting has been confirmed against real drone output. The accuracy figures come from simulated east-west flights. It assumes flat ground and a camera pointing straight down, the blur limits are tuned on synthetic photos, and it isn’t connected to live drone comms.