Proj-2013-2014-BrasRobot-Handicap-1

Description and Goals

The long-term goal of this project is to develop a control software manipulator arm for support of persons with disabilities.

Team

• Supervisor: Olivier Richard

• El Hadji Malick FALL, Adji Ndèye Ndaté SAMBE

• Department : RICM 4, Polytech Grenoble

Documents

The Software Requirements Specification (SRS) can be found here

Steps

The projet was attributed on Monday 13th and started the following day January 14th, 2014

Week 1 (January 13th - Janurary 19th)

• Project discovery
• Meeting with the different members of the project for assigning roles. Each pair chooses the part of the project that it wants to process.We choose to process the part on the detection of markers.
• Research on marker detection

Week 2 (January 20th - Janurary 26th)

• Test configuration of the robot
• Learning of C++
• Discovery of OpenCV

In details, we looked at the record and the code of the former group and studied the methods they proposed for the markers detection including hough transform and harritraining. Then we make some researchs to know how OpenCv works. One of the question is what will be the criteria of the markers : color, shape of both.

Week 3 (January 27th - February 2nd)

• Discovery of ArUco

ArUco is a minimal library for Augmented Reality applications based on OpenCv. One of its features of ArUco is Detect markers with a single line of C++ code. This will be very useful for our approach. As it is described in the source forge website, each marker has an unique code indicated by the black and white colors in it. The library detect borders, and analyzes into the rectangular regions which of them are likely to be markers. Then, a decoding is performed and if the code is valid, it is considered that the rectangle is a marker. So our next step will be to choose what kind of marker we will use and define the various steps of the recovery of the marker by the robot. We also start to learn Python langage which will be used to develop ArUco solution. Installing PyDev requires much more time as we though. After successfull completion, we are now developing our own solutions in Python inspired by ArUco's.

Week 4 (February 3rd- February 9th)

As we said, our own solutions are on going process. We expect that in this week they will be completed and fortunately we managed to make our first tests with ArUco. We have experienced a solution with black and white markers with ArUco, and a second with a face detection with OpenCV. The next step is now to be able to test it on video to first measure the efficiency and then take back the coordinates of marker's position that will be interesting for the system group.

facemarker

Arucomarker

Week 5 (February 10th- February 16th)

After a long trying, we have finally succeed to detect several markers contained in a jpeg picture and avi video. We are also able to get back the space coordinates of the marker. We could write it in a file that we will be used by the system team. Our next step is now to detect markers in a live video.

detectedmarkers

MarkersOnVideo

But even if Aruco solution is functional now, we can consider encounter some difficulties. For that OpenCV is still an optional solution. We have already made ​​contact with the Canonball group about it. Knowing that we can retrieve the coordinates x, y, z of the marker, we expect, following our discussion with the group system, to transcribe these data into a text file they will use. The advantage of such a solution is that if the solution proposed by the system group is not practical, we can always use this file to prove the proper functioning of our solution and use it to find another alternative. In the meantime we do our first live detection tests. The first results are rather conclusive but now we will have to determine several scenarios that may happen.

MarkersLive

Week 6 (February 17th- February 23th)

For this week, after making our first tests of real-time detection, we must renew them but this time with the webcam at our disposal. In Aruco, more generally in OpenCV, we have a type VideoCapture from the class of the same name that manages the video stream. It provides an open function whose signature is as follows:

0 is the default webcam of the computer and 1 the first outside webcam connected to the USB port. After making these changes, we are now able to use the webcam provided. Moreover, it should be noted that this phase of the project is a milestone. Before going further, we wanted to make an assessment of future solutions before integration. We first started to make a series of measures to evaluate the performance of detection.

We find that the closer the distance, the greater the chances of detecting a marker. The optimal distance is estimated at 14 cm or less , and that under favorable conditions ( well calibrated light, visible marker) . We also conducted tests on the number of markers needed . We generated a matrix of markers and we tested detection :

Detecting a single marker might fail for different reasons such as poor lightning conditions, fast camera movement, occlusions, etc. To overcome that problem, ArUco allows the use of boards. An Augmented Reality board is a marker composed by several markers arranged in a grid. Boards present two main advantages. First, since there have more than one markers, it is less likely to lose them all at the same time. Second, the more markers are detected, the more points are available for computing the camera extrinsics. As a consecuence, the accuracy obtained increases.

We can see that the ideal would be to have a minimum size of 4x5 to ensure optimal detection . ie we are sure that at least one marker will be detected. Based on these parameters, we can describe a first draft screenplay . The robot detects the marker. By applying the Pythagorean theorem , we can determine the coordinates of the central point. Once these coordinates obtained, the robot performs a translation to this point, as in the diagram below:

Week 7 (February 24th- March 2nd)

During this week we have improved the scenario drawn up last week. After identifying the actions that the robot will perform, we will now look at another problem: how the robot will know the coordinates of the point to be reached?

We considere two possible solutions:

- Write the coordinates on a file and parse it

- Transmit the coordinates through a socket

Both solutions are feasible but the second solution seems more interactive. The first was carried out, while the second is still in progress.

Week 8 (March 3rd- March 9th)

Educational truce

Week 9 (March 10th- March 16th)

The results of the detection of marker are edited in a text file. To send the recovered data, we decided to convert this file into a XML file in python. That may facilitate reading and data retrieval. Here is an excerpt from a example of a XML file with the coordinates of the detected marker:

<?xml version="1.0" ?>
<coordonnees>
: <p1 x="236.551" y="230.127"/>
: <p2 x="239.44" y="176.346"/>
: <p3 x="293.506" y="178.652"/>
: <p4 x="290.598" y="232.49"/>
</coordonnees>

As a marker is a square :

• p1 represents the coordinates of the bottom left.
• p2 represents the coordinates of the top left.
• p3 represents the coordinates of the top right.
• p4 represents the coordinates of the bottom right.

We also documented about the implementation of sockets with the client/server model in python for sending data. We tested several libraries proposed for the implementation of this solution.We realized that this solution was then feasible. So we began the implementation of this solution.

Week 10 (March 17th- March 23th)

We continued the implementation of sockets. We noticed that the method we used seemed complicated. Therefore, we contacted our distributed applications teacher M. Philippe MORAT trying to see if there exists a way to do otherwise and more simpler. Following a discussion with him and thanks to TP on the system of agents in java, we decided to use the model with agents because this model cut off from any language and is one of model of low-level simulations based on a channel.

Week 11 (March 24th- March 30th)

After much consideration, we decided to realize the previously proposed solution.

A multi-agent system (MAS) is a system composed of a set of agents located in a certain environment and interacting according to some relations. An agent is featured by its autonomous. It may be coded in the most ​​known communication languages. The AAA model(Agent Anytime Anywhere) in Java is one of the best known. This solution has many advantages:

• not having to worry about the addresses and listening ports
• a easiness to create behaviors without worrying threads
• a easiness to send strings in the ACL (Agent Communication Language) and retrieve these strings. Serialized data, XML,... can also be sent
• the agent deployment and referencing the Directory Facilitator (directory of agents) is automatic. When an agent disappears, it is removed from the Directory Facilitator.

However, to maintain compliance with a requirement of the specifications that was to use the Python language, we conducted a series of studies which took us to the discovery of SPADE.

SPADE is a platform infrastructure development written in Python and based on the paradigm of multi-agent system. SPADE is based on the XMPP/Jabber protocol. Thus, a Python agent can very well communicate with C + + or Java agent for example. The agent may take any decisions alone or in groups. Each agent has an assigned task (a behavior) which is actually a kind of thread. One could imagine, for example an agent having behavior to send commands to other agents, as well as behavior to retrieve the results.

We therefore conducted various tests of this platform to determine how to adapt the functionalities offered by SPADE in our context of use.

Week 12 (March 31th- April 6th)

In this new stage, we began the implementation of a client/server model based on agents using the SPADE platform in python.

Appended

How to install Aruco

1- http://miloq.blogspot.com.es/2012/12/install-opencv-ubuntu-linux.html

2- http://miloq.blogspot.fr/2012/12/install-aruco-ubuntu-linux.html