Case study / Document verification

Citizenship Verification System

A web application that checks a Nepali citizenship or ID card against what a person typed and against their photo, instead of a clerk doing it by hand.

Code

The problem

In Nepal, people and businesses can wait more than a week for a document to be verified by hand, which holds up things like opening a bank account.

What it had to do

Read the Nepali and English text on the card
Compare it with what the person entered
Compare the face on the card with their photo

What was built

A Django web application. A person fills in a form and uploads their citizenship card, ID card and a passport-sized photo, and three checks together decide whether the identity is verified.

How it works, step by step

The person fills in the form and uploads the citizenship card, the ID card and a photo.
The faces on the documents are compared with the photo.
A ResNet-50 classifier checks that each upload really is a citizenship card or an ID card.
A fine-tuned TrOCR model reads the Nepali text from the citizenship card and the English text from the ID card.
The text that was read is compared with what the person typed.
The three results together decide whether the identity is verified.

My part

The web interface in HTML, CSS, JavaScript and Bootstrap: the landing page and the verification page with its forms and upload step.
On the form, what a person types in English is translated into Nepali for them to confirm, since the citizenship card itself is in Nepali.

The rest of the team

Teammates trained the text and classification models and built the Django backend that connects them to the interface.

The outcome

The project report gives a character error rate of 4.28% for the Nepali text model on 1,000 test images, and 99% accuracy for the document classifier across its four classes. These are the team’s reported results.

What you can check

Project report: The full report submitted to the university, with the method and the results.
Open: Project report, Citizenship Verification System (opens in a new tab)
Code: The repository lives under a teammate’s GitHub account. My commits there are the front-end ones.
Open: Code, Citizenship Verification System (opens in a new tab)