Home Tech & ScienceHow Facial Recognition Works on Mobile Devices: Cameras, Math, and the Vault Inside Your Phone

How Facial Recognition Works on Mobile Devices: Cameras, Math, and the Vault Inside Your Phone

by Leo
0 comments
How Facial Recognition Works on Mobile Devices: Cameras, Math, and the Vault Inside Your Phone

You lift the phone, glance at the screen, and the padlock disappears. Start to finish, a modern handset spends somewhere around a third of a second on that exchange, and only a fraction of it goes on the actual comparison. Most of the work is optics, pattern matching and a quiet bit of cryptography happening in a chip you will never see.

Knowing the sequence explains a lot of everyday oddities. Why face unlock works in a dark hallway but sulks in bright sunlight. Why a £900 flagship will authorise a bank transfer with your face while a £120 Android will only wake the screen. The pipeline always runs in the same order: detect, capture depth, encode, compare, decide.

Finding Your Face Comes Before Knowing It

The moment the camera wakes, usually triggered by lift-to-wake or a tap, the phone starts scanning frames at roughly 30 per second. A small, fast neural network hunts for face-shaped regions and returns bounding boxes. This stage is cheap on purpose. It has to run constantly without draining the battery, so it often leans on a dedicated low-power image processor rather than the main CPU.

Once a face is located, a second model pins down landmarks: the corners of the eyes, the tip of the nose, the line of the jaw. Google’s MediaPipe mesh, for example, places 468 points on a single face. Those points let the phone crop, rotate and normalise the image so that the recognition step always receives a straight-on, consistently framed portrait, even if you were slouched sideways in bed.

banner

Detection and recognition are separate jobs

Detection says “there is a face here.” Recognition says “this one belongs to the person who owns this phone.” Plenty of apps use the first without ever touching the second: the camera’s autofocus uses detection to lock onto faces, and photo libraries use it to group snapshots of the same friend. Only the lock screen and payment flows go further.

The Hardware: Three Ways to Capture a Face in Depth

Flat photographs are easy to spoof, so the more serious implementations add a third dimension. There are three common approaches, and the one in your phone largely determines how much you should trust it.

  • Structured light. An infrared projector throws roughly 30,000 invisible dots across your face. A matching IR camera photographs the pattern and measures how it warps over the curve of your nose and the hollows of your eye sockets. This is the method behind Face ID.
  • Time of flight. An emitter pulses infrared light and a sensor times how long each pulse takes to bounce back. Depth resolution is coarser than structured light, but it works at a distance and copes with motion well.
  • Stereo pairs. Two cameras a few millimetres apart compare their views the way your eyes do, calculating depth from the disparity between images.
  • Plain 2D colour. A single RGB frame and nothing else. Cheap, fast, and still the most common setup on budget Android hardware.

Turning a Face Into a List of Numbers

No phone stores a picture of your face for comparison, at least not in any well-designed system. Instead, a convolutional neural network chews through the normalised image and outputs a vector, typically between 128 and 512 numbers. This is called an embedding, and it is the closest thing to a mathematical fingerprint your face has.

The trick that makes embeddings useful is that the same face photographed under different lighting still lands in roughly the same spot in that high-dimensional space. Change your haircut, grow a beard, put on glasses, and the vector shifts slightly but stays close. The comparison step measures the angle between two vectors, or the straight-line distance, and checks whether it falls inside a threshold.

The threshold decides how paranoid your phone is

Set the threshold loosely and the phone unlocks quickly but occasionally accepts a stranger. Tighten it and you get security at the cost of failed attempts. Engineers describe the trade-off with two numbers: the false acceptance rate (a wrong person gets in) and the false rejection rate (you get locked out of your own device). Apple publishes a 1 in 1,000,000 false match rate for Face ID, against 1 in 50,000 for the older Touch ID fingerprint sensor.

All of that inference has to happen in milliseconds on a phone, which is why the neural processing hardware covered in this look at AI in mobile phones matters so much. A model that takes two seconds to run is useless for a lock screen.

Where the Comparison Actually Happens

The reference template lives in a dedicated secure element: Apple’s Secure Enclave, Google’s Titan chip, or an ARM TrustZone-backed StrongBox on many Android devices. That chip has its own memory and its own tiny operating system, isolated from the main processor. The camera frames get passed in, the match result comes out as a simple yes or no, and the raw data is discarded.

This design is deliberate. Malware running on the main operating system can read your photos, your messages and your location, but it cannot pull the biometric template out of the enclave. Failed attempts are counted in hardware too, which is why five unsuccessful face scans force you to type the passcode.

Proving You’re a Person, Not a Printout

Liveness detection is the part that separates a real biometric system from a gimmick. Depth alone stops a flat photo, but not a moulded silicone mask. Good implementations stack several checks together: does the face have the subtle texture and infrared reflectivity of skin, does the depth map show a nose sticking out in front of the cheeks, are the eyes open and tracking, is there any parallax as the phone moves slightly.

Attention detection adds another layer. Face ID by default refuses to unlock unless you are looking at the device with your eyes open, which blocks the obvious attack of holding someone’s phone up to their face while they sleep. It is not flawless. In 2019, researchers defeated the liveness checks on several Android phones by sticking small pieces of white tape over the lenses of a pair of glasses, exploiting how the algorithms detected an eye-like shape. Vendors patched the hole, but the episode is a useful reminder that anti-spoofing is an arms race, not a solved problem.

Not All Face Unlock Is Built the Same

Android splits biometric hardware into classes. Class 3, sometimes labelled Strong, requires a spoof-resistant sensor and can be used to authorise payments, unlock password vaults and approve high-value transactions. Class 2, Weak, is fine for waking the screen and little else. Plenty of mid-range phones ship with a 2D camera that only meets the weaker bar, which is why your banking app may insist on a fingerprint or PIN even though the phone technically has face unlock.

If you’re wiring biometrics into your own product, the class check is the first thing to get right, and the API groundwork in this guide to building apps covers the general shape of that work. Getting it wrong in a financial context is more expensive than most teams expect, as the security and compliance planning laid out in this fintech software development guide makes clear.

It’s Not Just for Unlocking

Beyond the lock screen, the same models power photo libraries that cluster shots of the same person, AR filters that track your expression in real time, and accessibility features that let you scroll a page by raising your eyebrows. Video editors lean on face tracking to keep a subject centred during automatic reframing, a feature that shows up in most of the tools covered in this honest review of video editing software.

The uncomfortable part is that the same detection models appear in shops, airports and public squares, watching faces that never agreed to be watched. The commercial cameras doing that work run the same rough pipeline described here, just at a different scale, and the consequences are explored in this piece on how digital surveillance turns everyday devices into evidence.

What’s Stored, and What Leaves the Phone

On a properly built device, a mathematical template stays in the secure chip and nothing is uploaded. The legal position is different from the technical one, though. Under Europe’s GDPR, biometric data used to uniquely identify a person is a special category of information requiring explicit consent. Illinois’ Biometric Information Privacy Act has generated real money, including a $650 million settlement with Facebook in 2020.

Watch for apps that handle faces differently. Remote identity checks for bank onboarding often capture a short video and send it to a server for a liveness decision. That is a cloud biometric system wearing a mobile interface, and the data handling rules are whatever the company wrote in its policy. Any app asking for your face deserves a look at that policy before you tap allow.

How to Tell What Your Own Phone Is Doing

Point the front camera at the sensor area in a dark room and you may see a faint reddish glow from an IR projector, which tells you there is depth sensing involved. Dig into security settings and look for options labelled Require Attention, Require Open Eyes, or Alternate Appearance; their presence usually signals a stronger implementation. Then run a test: hold up a printed photo of yourself and see whether it unlocks. If it does, treat face unlock as a convenience and let the PIN do the real work.

The next round of hardware is already arriving, with under-display infrared sensors that remove the notch and passive liveness detection that asks nothing of you at all. Until that reaches the cheap end of the market, the honest summary of any given phone is written in its biometric class, not its marketing copy.

You may also like

Leave a Comment