Test results: 6 hours of footage, 98 searches

We tested whether ClipRecall AI can find the right clip from a few typed words, on security camera footage it had never seen. This is what happened, with every kind of miss shown.

The footage

We used public security camera footage that we are allowed to show: the MEVA dataset, made by Kitware Inc. and IARPA. The people in this footage are actors who agreed to be filmed. Kitware and IARPA are not ClipRecall customers and did not take part. Their footage is public, and is named here so you can check it.

Clips72, each five minutes long
Hours of video6
Cameras12: 7 outdoors and 5 indoors
Frames read108,077, five every second
Marks on people and vehicles627,404
Sightings1,776
Pictures kept6,563
LightDaytime only
ModelsOne finder and one matcher, the same two the live service runs. They are named, with their versions, on the credits page

Finding people and vehicles

The footage comes with boxes that people drew around those taking part in an activity. We counted how many of those ClipRecall AI found. The answer depends mostly on how large the person is in the picture, which is 1,080 pixels high.

People found, by how tall they are in the picture

Under 50 pixels tall
103 of 286 (31 to 42 in 100)
50 to 99
339 of 389 (83 to 90 in 100)
100 to 199
470 of 547 (83 to 89 in 100)
200 or more
410 of 439 (91 to 95 in 100)
All people
1,322 of 1,661 (78 to 81 in 100)

Vehicles found

All vehicles
609 of 635 (94 to 97 in 100)
ClipRecall AI The range the true figure probably lies in

Of the people and vehicles it found, 1,295 of 1,931 were kept as one sighting. The rest were split into two or more.

Searching

All 98 searches were written before any search was run. A search counts when a right clip is among the first five results. Beside ClipRecall AI we ran the plain method it has to beat: count the colours in the picture, with no AI at all.

Clothes and colours

40 searches, against about 155 sightings on every camera. The first result was right 22 times for ClipRecall AI, and 18 times for the colour count.

ClipRecall AI
35 of 40 (74 to 95 in 100)
Plain colour count
30 of 40 (60 to 86 in 100)

Vehicles

20 searches, against about 238 sightings on every camera. The first result was right 18 times for ClipRecall AI, and 7 times for the colour count.

ClipRecall AI
20 of 20 (84 to 100 in 100)
Plain colour count
16 of 20 (58 to 92 in 100)

Actions

38 searches taken from the footage’s own activity labels, each against the sightings of one camera.

Shows in one picture
14 of 22 (43 to 80 in 100)
Needs movement
4 of 16 (10 to 49 in 100)
All action searches
18 of 38 (32 to 63 in 100)
ClipRecall AI A plain count of colours, with no AI The range the true figure probably lies in What chance alone would score

What it got wrong, and one it got right

A green frame is a right result. A red frame is a wrong one.

person opening the boot of a vehicleThe first right clip came 32nd, out of 109 sightings on one camera
A wrong result for person opening the boot of a vehicle1, wrong
A wrong result for person opening the boot of a vehicle2, wrong
A wrong result for person opening the boot of a vehicle3, wrong
A wrong result for person opening the boot of a vehicle4, wrong
A wrong result for person opening the boot of a vehicle5, wrong
A right result for person opening the boot of a vehicle32nd, right

The first results are cars, and people at the door of a coach. The labelled moments on this camera are at the luggage hold of a coach, and the first of them came 32nd.

someone carrying something heavyThe first right clip came 48th, out of 124 sightings on one camera
A wrong result for someone carrying something heavy1, wrong
A wrong result for someone carrying something heavy2, wrong
A wrong result for someone carrying something heavy3, wrong
A wrong result for someone carrying something heavy4, wrong
A wrong result for someone carrying something heavy5, wrong
A right result for someone carrying something heavy48th, right

"Heavy" cannot be seen in a picture. The first results are people walking with empty hands. The right person carries a large bag.

vehicle letting a passenger outThe first right clip came 10th, out of 109 sightings on one camera
A wrong result for vehicle letting a passenger out1, wrong
A wrong result for vehicle letting a passenger out2, wrong
A wrong result for vehicle letting a passenger out3, wrong
A wrong result for vehicle letting a passenger out4, wrong
A wrong result for vehicle letting a passenger out5, wrong
A right result for vehicle letting a passenger out10th, right

A drop off does not show in a still picture. The first results are people at the door of a coach, but not the one person the footage's labels mark as dropped off, who came tenth.

person in a red top carrying a backpackThe first right clip came 21st, out of 160 sightings on every camera
A wrong result for person in a red top carrying a backpack1, wrong
A wrong result for person in a red top carrying a backpack2, wrong
A wrong result for person in a red top carrying a backpack3, wrong
A wrong result for person in a red top carrying a backpack4, wrong
A wrong result for person in a red top carrying a backpack5, wrong

Three of the first five wear bright pink, which ClipRecall AI treats as red. The other two wear red, but their marks call the thing they carry a bag, not a backpack.

person in a blue top carrying a boxThe first right clip came 68th, out of 159 sightings on every camera
A wrong result for person in a blue top carrying a box1, wrong
A wrong result for person in a blue top carrying a box2, wrong
A wrong result for person in a blue top carrying a box3, wrong
A wrong result for person in a blue top carrying a box4, wrong
A wrong result for person in a blue top carrying a box5, wrong

ClipRecall AI found blue tops, but with bags, not boxes. It reads a colour better than the thing carried.

grey carThe first right clip came 2nd, out of 239 sightings on every camera
A wrong result for grey car1, wrong
A right result for grey car2, right
A wrong result for grey car3, wrong
A right result for grey car4, right
A wrong result for grey car5, wrong

Right in the first five, but not first.

person talking on a phoneThe first result is right, out of 223 sightings on one camera
A right result for person talking on a phone1, right
A right result for person talking on a phone2, right
A wrong result for person talking on a phone3, wrong
A wrong result for person talking on a phone4, wrong
A right result for person talking on a phone5, right

Three of the first five are right. The two marked wrong also show a hand raised to the head.

Every action search

Each search ran against the sightings of one camera. “Came” is the place of the first right clip in the results.

We corrected the wording of 15 of these searches after the first run. They said “car”, but the labels that came with the footage say “vehicle”, and on some cameras that vehicle is a coach. A search for a car was being scored against a coach. They now say “vehicle”. With the first wording, 20 of 38 action searches found a right clip in the first five. With the corrected wording, 18 of 38 do. For searches that need movement the figure went from 6 of 16 to 4 of 16.

SearchKindSightings searchedRight ones among themFirst right clip cameChance alone
person talking on a phoneShows in one picture22311123 in 100
someone on a phone callShows in one picture1415217 in 100
person looking at their phoneShows in one picture22331253 in 100
someone typing on a phoneShows in one picture1418626 in 100
person carrying a heavy objectShows in one picture2066714 in 100
someone carrying something heavyShows in one picture1241484 in 100
person riding a bicycleShows in one picture223214 in 100
someone on a bikeShows in one picture1415117 in 100
person reading a paperShows in one picture1262118 in 100
someone reading a documentShows in one picture2232434 in 100
person opening a doorShows in one picture376161 in 100
someone at a doorShows in one picture2230Not found0 in 100
person opening a vehicle doorShows in one picture2068318 in 100
someone at a vehicle doorShows in one picture230316 in 100
person opening the boot of a vehicleShows in one picture10943217 in 100
someone at the open boot of a vehicleShows in one picture2069120 in 100
person loading things into a vehicleShows in one picture10934313 in 100
someone taking things out of a vehicleShows in one picture2165211 in 100
two people talkingShows in one picture22387192 in 100
people standing and talkingShows in one picture6326194 in 100
two people huggingShows in one picture2237315 in 100
people embracingShows in one picture2237315 in 100
someone entering a vehicleNeeds movement20651512 in 100
someone leaving a vehicleNeeds movement21651011 in 100
person coming in through a doorNeeds movement6325393 in 100
person going out through a doorNeeds movement2230Not found0 in 100
person sitting downNeeds movement2360Not found0 in 100
person standing up from a seatNeeds movement2301832 in 100
person picking something upNeeds movement2303126 in 100
person putting something downNeeds movement2301282 in 100
person handing something to another personNeeds movement109239 in 100
vehicle making a turnNeeds movement22318235 in 100
vehicle backing upNeeds movement20656312 in 100
vehicle coming to a stopNeeds movement22381417 in 100
vehicle starting to moveNeeds movement1260Not found0 in 100
vehicle turning aroundNeeds movement2232114 in 100
vehicle letting a passenger outNeeds movement10931013 in 100
vehicle collecting a passengerNeeds movement141217 in 100

Three sizes of matcher

We ran the same searches with three sizes of the matcher. The numbers are searches with a right clip in the first five.

SizeClothes, of 40Vehicles, of 20Actions, of 38
Small322022
Middle352018
Large322019

No size was clearly better. The smallest did as well as the largest.

Can it say “nothing there”?

The matcher gives each result a score. We looked at 230 cases where the footage held no right answer at all. With a cut off low enough to keep every right first result, 46 of those 230 cases (20 in 100) still showed a wrong picture as if it were a match. So the score helps, but it cannot be trusted alone.

Who marked the right answers

For the action searches, the right answers are the boxes and activity labels that came with the footage. For clothes and vehicles, 223 pictures were marked for colour, kind and what was carried: 12 by a person and 211 by two AI helpers working apart, with their disagreements settled by an AI assistant. Where the person and the AI both named an answer, they agreed on 26 of 30. Marks made by AI are weaker evidence than a person’s, and we say so here so you can weigh the clothes and vehicle figures.

What this test covers

Daytime footage from one place in the United States, in full high definition. It holds no night video, no footage from Nigerian sites, and no keke or okada. The figures on this page are for this footage.

Video: "Multiview Extended Video with Activities" (MEVA) dataset by Kitware Inc. and IARPA, licensed under a Creative Commons Attribution 4.0 International License. Clips are shortened and resized.