The first test: 6 hours of footage, 98 searches

Before anything else, we wanted to know whether ClipRecall AI can find the right clip from a few typed words, on security camera footage it has never seen. This is what happened, with every kind of miss shown.

The footage

We used public security camera footage that we are allowed to show: the MEVA dataset, made by Kitware Inc. and IARPA. The people in this footage are actors who agreed to be filmed. Kitware and IARPA are not ClipRecall customers and did not take part. Their footage is public, and is named here so you can check it.

Clips72, each five minutes long
Hours of video6
Cameras12: 7 outdoors and 5 indoors
Frames read108,077, five every second
Marks on people and vehicles627,404
Sightings1,776
Pictures kept6,563
LightDaytime only

Finding people and vehicles

The footage comes with boxes that people drew around those taking part in an activity. We counted how many of those ClipRecall AI found. The answer depends mostly on how large the person is in the picture, which is 1,080 pixels high.

People found, by how tall they are in the picture

Under 50 pixels tall
103 of 286 (31 to 42 in 100)
50 to 99
339 of 389 (83 to 90 in 100)
100 to 199
470 of 547 (83 to 89 in 100)
200 or more
410 of 439 (91 to 95 in 100)
All people
1,322 of 1,661 (78 to 81 in 100)

Vehicles found

All vehicles
609 of 635 (94 to 97 in 100)
ClipRecall AI The range the true figure probably lies in

Of the people and vehicles it found, 1,295 of 1,931 were kept as one sighting. The rest were split into two or more.

Searching

All 98 searches were written before any search was run. A search counts when a right clip is among the first five results. Beside ClipRecall AI we ran the plain method it has to beat: count the colours in the picture, with no AI at all.

Clothes and colours

40 searches, against about 155 sightings on every camera. The first result was right 22 times for ClipRecall AI, and 18 times for the colour count.

ClipRecall AI
35 of 40 (74 to 95 in 100)
Plain colour count
30 of 40 (60 to 86 in 100)

Vehicles

20 searches, against about 238 sightings on every camera. The first result was right 18 times for ClipRecall AI, and 7 times for the colour count.

ClipRecall AI
20 of 20 (84 to 100 in 100)
Plain colour count
16 of 20 (58 to 92 in 100)

Actions

38 searches taken from the footage’s own activity labels, each against the sightings of one camera.

Shows in one picture
14 of 22 (43 to 80 in 100)
Needs movement
6 of 16 (18 to 61 in 100)
All action searches
20 of 38 (37 to 68 in 100)
ClipRecall AI A plain count of colours, with no AI The range the true figure probably lies in What chance alone would score

What it got wrong, and one it got right

A green frame is a right result. A red frame is a wrong one.

person opening a car bootThe first right clip came 74th, out of 109 sightings on one camera
A wrong result for person opening a car boot1, wrong
A wrong result for person opening a car boot2, wrong
A wrong result for person opening a car boot3, wrong
A wrong result for person opening a car boot4, wrong
A wrong result for person opening a car boot5, wrong
A right result for person opening a car boot74th, right

The first results are cars. The labelled moment on this camera is a person at the luggage hold of a coach, which nobody would call a car boot.

someone carrying something heavyThe first right clip came 48th, out of 124 sightings on one camera
A wrong result for someone carrying something heavy1, wrong
A wrong result for someone carrying something heavy2, wrong
A wrong result for someone carrying something heavy3, wrong
A wrong result for someone carrying something heavy4, wrong
A wrong result for someone carrying something heavy5, wrong
A right result for someone carrying something heavy48th, right

"Heavy" cannot be seen in a picture. The first results are people walking with empty hands. The right person carries a large bag.

car dropping someone offThe first right clip came 96th, out of 109 sightings on one camera
A wrong result for car dropping someone off1, wrong
A wrong result for car dropping someone off2, wrong
A wrong result for car dropping someone off3, wrong
A wrong result for car dropping someone off4, wrong
A wrong result for car dropping someone off5, wrong
A right result for car dropping someone off96th, right

A drop off does not show in a still picture, and the labelled one on this camera was by a coach, not a car.

person in a red top carrying a backpackThe first right clip came 21st, out of 160 sightings on every camera
A wrong result for person in a red top carrying a backpack1, wrong
A wrong result for person in a red top carrying a backpack2, wrong
A wrong result for person in a red top carrying a backpack3, wrong
A wrong result for person in a red top carrying a backpack4, wrong
A wrong result for person in a red top carrying a backpack5, wrong

Three of the first five wear bright pink, which ClipRecall AI treats as red. The other two wear red, but their marks call the thing they carry a bag, not a backpack.

person in a blue top carrying a boxThe first right clip came 68th, out of 159 sightings on every camera
A wrong result for person in a blue top carrying a box1, wrong
A wrong result for person in a blue top carrying a box2, wrong
A wrong result for person in a blue top carrying a box3, wrong
A wrong result for person in a blue top carrying a box4, wrong
A wrong result for person in a blue top carrying a box5, wrong

ClipRecall AI found blue tops, but with bags, not boxes. It reads a colour better than the thing carried.

grey carThe first right clip came 2nd, out of 239 sightings on every camera
A wrong result for grey car1, wrong
A right result for grey car2, right
A wrong result for grey car3, wrong
A right result for grey car4, right
A wrong result for grey car5, wrong

Right in the first five, but not first.

person talking on a phoneThe first result is right, out of 223 sightings on one camera
A right result for person talking on a phone1, right
A right result for person talking on a phone2, right
A wrong result for person talking on a phone3, wrong
A wrong result for person talking on a phone4, wrong
A right result for person talking on a phone5, right

Three of the first five are right. The two marked wrong also show a hand raised to the head.

Every action search

Each search ran against the sightings of one camera. “Came” is the place of the first right clip in the results.

SearchKindSightings searchedRight ones among themFirst right clip cameChance alone
person talking on a phoneShows in one picture22311123 in 100
someone on a phone callShows in one picture1415217 in 100
person looking at their phoneShows in one picture22331253 in 100
someone typing on a phoneShows in one picture1418626 in 100
person carrying a heavy objectShows in one picture2066714 in 100
someone carrying something heavyShows in one picture1241484 in 100
person riding a bicycleShows in one picture223214 in 100
someone on a bikeShows in one picture1415117 in 100
person reading a paperShows in one picture1262118 in 100
someone reading a documentShows in one picture2232434 in 100
person opening a doorShows in one picture376161 in 100
someone at a doorShows in one picture2230Not found0 in 100
person opening a car doorShows in one picture2068318 in 100
someone at a car doorShows in one picture230316 in 100
person opening a car bootShows in one picture10947417 in 100
someone at the open boot of a carShows in one picture2069120 in 100
person loading things into a carShows in one picture10937113 in 100
someone taking things out of a carShows in one picture2165411 in 100
two people talkingShows in one picture22387192 in 100
people standing and talkingShows in one picture6326194 in 100
two people huggingShows in one picture2237315 in 100
people embracingShows in one picture2237315 in 100
person getting into a carNeeds movement2065412 in 100
person getting out of a carNeeds movement21651011 in 100
person coming in through a doorNeeds movement6325393 in 100
person going out through a doorNeeds movement2230Not found0 in 100
person sitting downNeeds movement2360Not found0 in 100
person standing up from a seatNeeds movement2301832 in 100
person picking something upNeeds movement2303126 in 100
person putting something downNeeds movement2301282 in 100
person handing something to another personNeeds movement109239 in 100
car turningNeeds movement22318335 in 100
car reversingNeeds movement20652712 in 100
car stoppingNeeds movement2238117 in 100
car driving offNeeds movement1260Not found0 in 100
car making a u turnNeeds movement2232144 in 100
car dropping someone offNeeds movement10939613 in 100
car picking someone upNeeds movement141217 in 100

Three sizes of matcher

We ran the same searches with three sizes of the matcher. The numbers are searches with a right clip in the first five.

SizeClothes, of 40Vehicles, of 20Actions, of 38
Small322020
Middle352020
Large322020

No size was clearly better. The smallest did as well as the largest.

Can it say “nothing there”?

The matcher gives each result a score. We looked at 230 cases where the footage held no right answer at all. With a cut off low enough to keep every right first result, 36 of those 230 cases (16 in 100) still showed a wrong picture as if it were a match. So the score helps, but it cannot be trusted alone.

Who marked the right answers

For the action searches, the right answers are the boxes and activity labels that came with the footage. For clothes and vehicles, 223 pictures were marked for colour, kind and what was carried: 12 by a person and 211 by two AI helpers working apart, with their disagreements settled by an AI assistant. Where the person and the AI both named an answer, they agreed on 26 of 30. Marks made by AI are weaker evidence than a person’s, and we say so here so you can weigh the clothes and vehicle figures.

What this test covers

Daytime footage from one place in the United States, in full high definition. It holds no night video, no footage from Nigerian sites, and no keke or okada. The figures on this page are for this footage.

Video: "Multiview Extended Video with Activities" (MEVA) dataset by Kitware Inc. and IARPA, licensed under a Creative Commons Attribution 4.0 International License. Clips are shortened and resized.