SRS - Sign2Speech
Introduction
Purpose of the requirements document
This Software Requirements Specification (SRS) identifies the requirements for project "Sign2Speech". This is an open source projet and we shall present what we did for this project in case to catch interest of new potential contributors. This document is a guideline about the functionalities offered and the problems that the system solve.
Scope of the product
Sign2Speech could be used at reception desks or during video conferences to allow signing people to speak with people who don't know the French Sign Language. The main point of this project is to use Intel's Real Sense camera to recognize gestures from the French Sign Language to offer a new means of communication. The program will be able to transcribe gestures, done by a signing person, into written words, printed on the screen of the person who doesn't know the FSL. This communication will be made via a chat application working on WebRTC.
Glossary
- FSL: French Sign Language
- JSON: Javascript Object Notation . We use this format to store our dictionary.
References
- https://software.intel.com/sites/landingpage/realsense/camera-sdk/v1.1/documentation/html/index.html?doc_manuals_sdk_algorithms.html
- http://rapidjson.org/index.html
- https://github.com/dhbaird/easywsclient
Overview of the remainder of the document
In the remainder of the document, the general description of the software will be exposed. The requirements (functional and non-functional) will be specified in another part. The document will end with the product evolution.
General Description
Product perspective
The main aim of our project is to improve the communication between a signing and a non-signing person. To do so, we developed a software that is capable of recognizing gestures, finding their meaning in a linked dictionary and sending the translation to a correspondent via a websocket channel.
Product functions
Our appplication is made of 2 major parts:
- The recognition and translating application
In this function, the program will try to recognize the gestures executed in front of the camera. It is linked to a dictionary (JSON file) that contains all the words that the program will be able to recognize. If the gesture is recognized by the program, its meaning will be printed on the screen. If not, nothing will display.
- The WebRTC chat application
It is used to enable the communication between 2 persons, using the video and a text chat. It also allows the real-time transmission of the subtitles.
User characteristics
There is two types of users for the application:
- The first user (User1) knows the FSL,
- The second user (User2) doesn't understand the FSL
They both want to communicate together. User1 will stand in the front of the camera and will sign while User2 will watch the monitor to see the translation of the gestures. He will be able to reply using the messaging chat.
Operating environment
The camera's SDK only works on Windows. A good Internet connection is also required for the chat application, which only works with Mozilla Firefox due to restriction on Chrome (WebRTC needs a HTTPS connection) and other browser doesn't provide a full support of WebRTC. The user must not use a hotspot or a connection wich is under a complex NAT. The connection process for WebRTC will struggle to pass through the NAT and will send back an error message.
General constraints
Of course, the user needs to have Intel's Real Sense camera. We have reported different factors that have a negative consequence on the hand tracking process:
- The user must not wear bracelets or rings,
- The user should, as much as possible, use the camera under natural light, rather than artificial light sources,
- The user must wear a monochrome top that contrasts with the color of the skin.
These elements can reduce the errors and make the tracking better, but it still won’t be perfect because of the imprecisions of the camera themselves.
Specific requirements, covering functional, non-functional and interface requirements
Requirement X.Y.Z (in Structured Natural Language)
Gesture recognition
Description:
Inputs: Hand and finger data returned by the camera stream
Source: Intel's Real Sense camera
Outputs: The encoding corresponding to the gesture (fingers+trajectorie)
Destination: Computer's memory
Action:
Non functional requirements: Real-time tracking (< 1 second) and margin of error (error scoring result < 8)
Pre-condition: Optimal conditions of use (good light, monochrom top that constrats with the color of the skin, no rings, no bracelets, ...)
Post-condition: The gesture must have been well recognized by the camera in order to be well encoded
Side-effects: If the gesture has not been well recognized and encoded, it won't be recognized/well translated afterwards
Gesture translation
Description:
Inputs: The gesture encoding returned by the gesture recognization step
Source: Computer's memory
Outputs: The current node moves inside the dictionary and sends the corresponding word if there is a match
Destination: WebSocket channel
Action:
Non functional requirements: Real-time search in the dictionary (< 1 second)
Pre-condition: The encoding must have a translation in the dictionary
Post-condition: The gesture must be well translated
Side-effects:
Learning mode
Description:
Inputs: Hand and finger data returned by the camera stream and the meaning of the gesture
Source: Intel's Real Sense camera
Outputs: New dictionary (JSON file) containing the new gestures and their meaning
Destination: Computer's memory
Action:
Non functional requirements: Real-time tracking (< 1 second)
Pre-condition: Optimal conditions of use (good light, monochrom top that constrats with the color of the skin, no rings, no bracelets, ...)
Post-condition: The new entry in the dictionary must correspond to the gesture that the user intended to do
Side-effects: If the gesture was not well recognized, the encoding in the dictionary will be wrong
Product Evolution
- “Real-time” windows that could show a representation of the hand that the camera is currently analyzing. It could allow the user to know if the camera is able to correctly recognize his hand. It could be done with QT Creator. Our application is not at this time really “friendly-user”.
- “2 hands” symbols that are currently not implemented in our application
- Improvements of trajectories recognition
- Language Model
- A better camera