Image Stitcher
Merges out-of-order photos from several drones, each tagged with a GPS (Global Positioning System) position, into one top-down map that sits in the right place on the Earth.
10 of 10simulated missions built one connected map, with photos lined up to 4.0–8.3 cm on average. Tested in simulation only.
- Status
- Merged Merged into the team’s main branch on November 18, 2026, after four rounds of review. The last round shortened the code and tightened refinement and the GPS checks.
- Role
- Built for the autonomy software team at the Waterloo Aerial Robotics Group, from October 2026. I wrote the whole pull request: the stitcher, the simulator that makes test missions with known true positions, the 130 tests and the evaluation script.
- Code
- Pull request #215
Why it exists
Drones fly back and forth over a field taking photos, and each one reaches the ground laptop whenever its drone sends it. The photos arrive shuffled, from several drones, with GPS that’s metres off.
GPS finds which photos should overlap, image matching lines them up, and all the GPS readings together place the finished map on the Earth.

From shuffled photos to one map
Check. Each photo is checked for blur, exposure and tilt. The ones that pass are undistorted for the lens.
Find neighbours. GPS says which photos should overlap, even when they arrive out of order from different drones.
Turn north-up. Each photo comes with the drone’s yaw, so it’s turned north-up before matching. If yaw is close to a quarter turn it just gets that quarter turn, which loses no detail.
Match. XFeat, a learned feature detector, finds the same ground points in both photos. RANSAC (random sample consensus) keeps only the matches that agree with each other and throws out the rest.
Refine. A least-squares solver nudges every photo at once until the total mismatch between overlaps is as small as it can be. It works like bundle adjustment, the standard way mapping software corrects many photos together.
Place. All the GPS readings together put the finished map in the right place on the Earth. A cluster is only fitted to GPS once both GPS and the images show its photos are spread out.
The data and the method
- Where the photos came from
- There are no real drone photos yet. I wrote a simulator that makes every test mission. It draws a random 120 by 90 m field of blobs, rectangles, lines and noise, and cuts photos out of it as a camera 25 m up would see them, lens distortion included, using the calibration of a 640 × 480 Arducam camera. Three drones fly back and forth in rows. The simulator adds the errors a real flight has: GPS 1 or 3 m off, altitude and heading slightly off, compass error, and photos arriving shuffled. Because it made the photos, it knows where each one truly was.
- What was tuned and what was scored
- Nothing in the stitcher is trained on these photos, so there’s no training set. Settings such as the blur limit were picked on simulated photos. The results come from 5 simulated fields, each flown at 1 m and 3 m of GPS error. They come from the same simulator, and two of them are also used in the blur check’s tests, so there’s no set held back and scored only once at the end.
- What kind of method it is
- It isn’t a model trained for this project. XFeat is a neural network its authors already trained, used as-is with their published weights, only to find matching points. Everything after it is fixed geometry: RANSAC throws out matches that don’t agree, and a least-squares solver lines all the photos up. The photo checks are hand-set limits on blur, exposure and tilt.
- What it outputs
- A map image, with a file giving its GPS corners and how many metres each pixel covers, plus a list of rejected photos and why. To score a simulated mission, each placed photo’s corners are compared with where they truly are. That gives the alignment and map-position rows below. A mission counts as one connected map when every placed photo ends up in a single group.
One connected map in 10 of 10 simulated missions
The simulator knows where every photo really was, so each run can be scored against the truth instead of by eye. Alignment is how far photos sit from their true positions relative to each other, averaged over each mission. Map position is how far the whole map sits from where it should be on the Earth.
| One connected map | 10 of 10 missions, at 1 m and 3 m of GPS error |
|---|---|
| Photo-to-photo alignment | 4.0–8.3 cm on average, about 1 to 2 pixels |
| Least-squares refinement | Misalignment of 13.6–69.8 cm without it, 4.0–8.3 cm with it |
| Map position against the truth | 0.162–0.307 m at 1 m of GPS error, 0.472–0.935 m at 3 m. Closer than the GPS error itself |
| Quality checks | 0 of 910 evaluation photos wrongly rejected. Blur check wrong on 0 of 2,730 photos under 9 conditions (5 sharp, 4 slightly blurred; none combine noise and blur) |
| Speed | 0.13–0.19 s per photo on a laptop CPU, down from 0.42–0.75 s before yaw |
| Bad input | A broken message or failed map save is logged and skipped. A photo with a GPS glitch, or a position outside valid latitude and longitude, is rejected instead of stretching the map. The first 3 photos are checked against each other, so one wrong first GPS fix can’t become the map’s origin and get every later photo rejected |
| Tests | 130 stitcher tests, plus a ground-truth mission simulator |
The connected-map, alignment, map-position and speed rows rerun with one command from the pull request.
What was hard
Rows wouldn’t connect
The drones turn 180 degrees at the end of each row, and XFeat isn’t rotation invariant: it can’t match a photo against one that’s upside down. In one simulated pair of photos across a row turn, only 10 of 612 candidate matches survived RANSAC. With the photo turned 180°, 317 of 714 did. The first fix extracted features at four quarter turns. When review confirmed every photo would come with yaw, that became one north-up extraction per photo, which is where the speedup came from.
A compass can’t be trusted to place photos
Yaw, the compass heading, has two jobs. It lines photos up for matching, and it orients clusters too small to fit to GPS. Final positions still come from the matches and GPS.
A reviewer caught a worse problem. GPS noise alone could pass the old spread check and blow up the map’s scale. Now the images have to show the photos are apart too.
A later version of a safety check depended on yaw, and a 180° yaw offset split one simulated map. It now compares distances, which needs no rotation. With constant yaw offsets of 0, 90 and 180°, 12 of 12 simulated runs formed one map.
Refinement made the map worse
The first version made errors worse, not better, because a huge penalty weight on the pinned photo (the one held still so the map can’t drift) made the solver unstable. Taking that photo out of the values the solver adjusts fixed it.
In the fourth round, the reviewer asked me to keep matches that conflict with a photo’s other matches out of refinement. I measured first. Cutting every matched point that sat a few pixels off made alignment worse, 4.9–14.6 cm on average against 4.0–8.3 cm, because before refinement removes drift, correct matches can sit up to about 30 pixels off. So only clear conflicts are dropped: wrong matches over repeating ground, hundreds of pixels off. Both figures are from the simulated missions.
Good matches split the map
The stitcher builds clusters, groups of photos that line up with each other, and merges two when a new photo bridges them. Small clusters with noisy GPS rejected good matches. Now a cluster’s GPS check only counts once the cluster spans four times the GPS error, and a simulated repeating-ground test checks that wrong matches stay blocked.
The blur check rejected good photos
Sharp and blurred scores overlapped, so no limit could work. The new score compares a photo with itself at half resolution, in each direction. My own review before pushing showed sensor noise could fool it, so a 3×3 median filter runs first. Even mild noise can still let a motion-blurred photo through in simulation, and the limits are tuned on synthetic photos.



Where it still falls short
Every result is from simulation. There are no real flight photos yet: the drone’s camera software doesn’t log roll, pitch or yaw yet, so its photos are rejected for now, and the lens calibration still has to be checked against the camera actually flown.
Neither the yaw convention nor the camera mounting has been confirmed against real drone output. The accuracy figures come from simulated east-west flights. It assumes flat ground and a camera pointing straight down, the blur limits are tuned on synthetic photos, and it isn’t connected to live drone comms.



