In the first post in this series, we introduced the idea behind the Facecam, a Facebook-enabled security camera running on a Raspberry Pi device.
Over the next three posts we’re going to look in more detail at the system’s components. In this post, we’re going to start by looking at the component – the only one that doesn’t actually run on the Raspberry Pi – that downloads the information from Facebook to build the facial recognition database to be used by the device.
As mentioned last time, I’ve coded this as a .NET application – actually by extending the WinForms sample provided with the Facebook C# SDK – in order to dedicate the Raspberry Pi to performing the security cam function. I’m not actually going to share the code for this application, as it’s really a work in progress. The code itself isn’t very complicated – the Facebook piece is mostly executing FQL, which allows you to use a SQL-like syntax to query the Facebook Graph API – and I’ll be describing the techniques it uses should people want to go ahead and roll their own.
Before looking at the behaviour of the application itself, it’s important to understand more about this “database” that it’s going to help build.
The facial recognition code I’ve adopted is based on work by Shervin Emami and uses the Eigenface algorithm to compare detected faces against a database. To understand more about Eigenface, this article provides a thorough explanation.
Shervin’s OnlineFaceRec sample makes use of OpenCV for this process. I’ve borrowed code from this sample for use on the Raspberry Pi, but today’s application simply builds a file and folder structure that can be used by the OnlineFaceRec application to train the database (and essentially build the facedata.xml file we’ll transfer across to the Raspberry Pi).
There are two main parts of this process: downloading the friend data from Facebook and then parsing that data on disk in order to create a training script to be passed to OnlineFaceRec.
These two functions are reflected by the basic UI of the .NET application:
Clicking on the first button lets us log in to Facebook and accept the permissions requested by the .NET application. Which reminds me: to create an application that makes use of the Facebook API, you’ll need to register it with Facebook and include the AppId in your code. As you can see below, I called my app “FriendExtractor”:
Once we’re logged in, we see the main dialog which will help us drive the download process:
Clicking on the “Populate Friends” button pulls down the list of the current user’s friends from Facebook and populates the list on the left:
You can see the list has checkboxes associated with each item. Originally the implementation didn’t allow selection of friends to include, which ended in a huge amount of data being downloaded (it turns out I have friends who seem to do nothing other than upload their pictures to Facebook and have 1000+ tagged images up there). Running OnlineFaceRec to train the database with the results ended up in a 400MB facedata.xml file being created. Given the fact the R-Pi has 256MB of RAM< I figured that was a non-starter. :-)
I therefore enabled selection of friends that are actually likely to visit our home – or that I may end up demoing this to at AU – which greatly reduces the amount of probably redundant data that gets downloaded and processed. The list of selected friends gets saved in a simple text file and reloaded automatically.
I also added the ability to start downloading from a particular friend, to allow incremental additions/updates. The first time it’s run, it’s certainly worth starting from the beginning, of course, but you may want to restart from a friend further down the list a later point.
When you select a friend on the left, the “Start Download from Selected” button should come alive. Pushing it does what’s written on the box:
Once completed, you’ll have a folder full of photos of you and your friends. The folder structure contains one level of indirection to use the user ID – as friends can very well have the same name (there’s a teenager named Kean Walmsley who lives in Canada and friended me on Facebook some time ago, for instance) – and inside that folder you’ll find a collection of images:
Looking closely, above, you’ll see there are both .jpg and .pgm images. The .jpg files are those downloaded directly from Facebook – nothing surprising there. The .pgm files are created by the .NET application using following algorithm:
- For each tagged photo of a user
- Use OpenCV face detection to find the faces in the image
- For this I used Emgu CV – a .NET wrapper for OpenCV – along with OpenCV’s face detection feature that looks for haar-like features
- Compare the tag location with the faces returned. For the best match…
- Take a greyscale copy of the cropped area
- Resize it to 50 x 50
- Equalize the histogram to get consistent brightness and contrast
- Save that file to the .pgm format
- Use OpenCV face detection to find the faces in the image
Regarding the quality of the photos downloaded from Facebook: frankly, it varies. Some photos are poorly tagged, and some are tagged well enough but the face cannot easily be extracted (which sometimes means someone else’s gets picked up in its place). And then some photos simply aren’t well suited to face detection/extraction, so the results end up being just plain strange.
I’ve found that some kind of manual scrubbing process between the download and the script creation/database training helps the quality of the data a great deal. I’ve used a tool called IrfanView to do this: you can search a set of sub-folders for *.pgm files and transfer the results across into “Thumbnail View”, and can then proceed to delete the ugly pics from there. Of course you could do the same thing in Explorer if you had a shell extension that can preview .pgm files, but I wasn’t able to find one. And IrfanView seems to work pretty well.
Here’s a view of some shots of Scott Sheppard and Shaan Hurley, for instance:
In time I expect a bespoke tool – one that remembers the pics you’ve deleted before, to save you having to repeat that task each time – is the way to go. But that’s for another time.
Once this is done, you should have a fairly clean set of normalised (i.e. of the same size and with similar brightness/contrast) files that can be used to generate a training script for the OnlineFaceRec tool.
The second button on our main dialog runs some code that copies the .pgm files to a separate folder and creates a training script listing them all:
private void btnCreateScript_Click(object sender, EventArgs e)
{
const string root = "Z:\\fb_photos\\";
const string destRoot = "Z:\\rp_photos\\";
SortedList<long, long> checkedFriends =
InfoDialog.LoadCheckedFriends();
// Get the list of PGM files in our source directory
DirectoryInfo di = new DirectoryInfo(root);
var files = di.GetFiles("*.pgm", SearchOption.AllDirectories);
StringBuilder sb = new StringBuilder(),
cur = new StringBuilder();
// We'll maintain an index for the persons and an
// old user ID, so we can tell when we have changed
// to a new person's data
int person = 0;
string ouid = "";
// Maintain a counter of the number of photos for
// the current person (we only write out 2+ photos,
// as otherwise they don't train the database),
// as well as a list of their paths
int photosForCurrent = 0;
List<string> filesForCurrent = new List<string>();
foreach (FileInfo fi in files)
{
var path = fi.DirectoryName;
var info = path.Split(new[] { '\\' });
if (info.Length == 4)
{
string name = info[2], uid = info[3];
if (ouid != uid)
{
// We have a new user
ouid = uid;
// If the last user had more than 2 photos...
if (photosForCurrent > 2)
{
// Let's add them to the training script and
// copy the files to the destination
person++;
sb.Append(cur.ToString());
foreach (string file in filesForCurrent)
{
string destFile =
destRoot +
Utils.RemoveAccents(file.Substring(root.Length));
Directory.CreateDirectory(
Path.GetDirectoryName(destFile)
);
File.Copy(file, destFile, true);
}
}
// Reset the string, photos and image file info
// for the current user
cur.Clear();
photosForCurrent = 0;
filesForCurrent.Clear();
}
long id = Int64.Parse(uid);
if (checkedFriends.ContainsKey(id))
{
// Add the information on the current photo to
// the current user's string
cur.Append(
String.Format(
"{0} {1} {2}\r\n",
person, Utils.RemoveAccents(name),
"./" +
Utils.RemoveAccents(fi.FullName).
Substring(root.Length).Replace('\\', '/')
)
);
// Do the same for the photos to copy across to
// the destination
photosForCurrent++;
filesForCurrent.Add(fi.FullName);
}
}
}
// Don't forget to tidy up and copy the last user's
// training string and files across
sb.Append(cur.ToString());
foreach (string file in filesForCurrent)
{
string destFile =
destRoot + file.Substring(root.Length);
Directory.CreateDirectory(
Path.GetDirectoryName(destFile)
);
File.Copy(file, destFile, true);
}
// Finally we write out the string to the training file
File.WriteAllText(
destRoot + "training.txt", sb.ToString()
);
}
In case you’re wondering, I’ve kept the two main pieces of the process separate primarily so I can combine my wife’s friends with my own before training the database. :-)
Now it’s a simple matter of running OnlineFaceRec with the training script:
The output of this tool is pretty interesting: aside from the very important facedata.xml file (the database we’ll use on the Raspberry Pi), we also two files that are created mainly to demonstrate the principle at work.
Firstly we have the average face – which is the absolute average of all the pictures of your friends, and tends to be a bland, slightly androgynous and some might say idealised image of a human face:
And seondly we have an image of the Eigenfaces themselves, which map the differences from this average image:
Neither of these images needs to be transferred across to the Raspberry Pi – their data is captured in the XML database (which stands at around 28 MB for the selected subset of my friends).
In the next post, we’ll look at the implementation of a component that makes use of this XML data to perform facial recognition on images captured by our motion detector.
