air - User contributions [en]

File:Transparents IaaS.pdf

2016-03-17T14:36:25Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Transparents IaaS.pdf"

File:Transparents IaaS.pdf

2016-03-17T11:51:26Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Transparents IaaS.pdf"

File:Transparents IaaS.pdf

2016-03-17T11:41:50Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Transparents IaaS.pdf"

File:Transparents IaaS.pdf

2016-03-17T09:41:27Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Transparents IaaS.pdf"

File:Transparents IaaS.pdf

2016-03-17T09:06:20Z

Alan.Damotte:

Projets 2015-2016

2016-03-17T07:40:53Z

Alan.Damotte: /* Projets */

<<[[Projets 2014-2015]] | [[Projets]] | [[Projets 2016-2017]]>>
=RICM=
==RICM3==

==RICM4==
===Projet Semestre S8===

Enseignants responsables : Olivier Richard, Didier Donsez

* '''Evaluation à mi-parcours le lundi 7 mars''': Format: 10min (5min de présentation 3 slides au plus, 5min de discussion). Cette évaluation sera prise en compte dans la note finale.

'''Consignes générales:'''

* '''Vous devez être pro-actifs !!!''': Si des points sont pas ou mals spécifiés, vous le faîtes et vous justifiez vos choix. Pour les problèmes techniques éventuels vous pouvez: vous creusez la question, vous contactez l'auteur du code si il y a lieux, vous faites un rapport de bug ('''Attention:''' ca se prépare !), vous soumettez un patch, vous contactez l'enseignant ou la personne suivant le projet.

* '''Vous devez maintenir une fiche de suivi de projet''': elle doit être mise à jour chaque semaine, elle rassemble les élements essentiels du projet, elle
indique les évolutions du projet et présente sa feuille de route. '''Note:''' le nom de la fiche doit être composé du nom du projet et suffixé par ricm4_2015_2016.

* '''Vous devez utiliser un logiciel de gestion de version''' pour vos développements comme [http://en.wikipedia.org/wiki/Git_%28software%29 git ] et nous vous conseillons d'utiliser le site [https://github.com github] pour l'hébergement de votre dépôt public.

* Les document public (exemple sur github) doivent être rédigés en anglais (README, documentation, commentaires de code, nom de variables et de fonctions). Une bonnification sera accordée si le rapport et les transparents sont en anglais (la soutenance sera en francais).

{|class="wikitable alternance"
|+ Affectation des projets RICM4 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [[Dashboard pour gestionnaire de tâches et de ressources]]
| CROUZET, MATHIEU
| Richard
| [[Projets-2015-2016-DashBoard| '''Fiche''']]
| [https://github.com/MatthieuCrouzet/Projet4A '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_groupe1.pdf|Rapport Consultant]] - [[Media:Paterns.pdf|Patterns]] - [[Media:PresentationDashboard.pdf|Presentation]]
|-

!scope="row"| 2
| [[Speeding Simplified Script Language]]
| POPEK, BERTRAND-DALECHAMPS, WEI
| Richard
| [[Projets-2015-2016-SSSL| '''Fiche''']] - [[SSSL-UML| '''UML''']]
| [https://github.com/FlorianPO/Speeding-Simplified-Script-Language.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:Groupe2_AIR.pdf|Rapport Consultant]] - [[Media:PresentationProjet.pdf|Presentation]]
|-

!scope="row"| 3
| [[Borne interactive]]
| DUNAND - NAVARRO - REVEL
| Maisonnasse
| [[Projets-2015-2016-Borne-Interactive| '''Fiche''']] - [[Projets-2015-2016-Borne-Interactive-SRS | '''SRS''']] - [[Projets-2015-2016-Borne-Interactive/UML_Diagrams | '''UML''']]
| [https://github.com/Kant73/InteractiveDisplay '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:IPopo.pdf|Rapport Consultant]] - [[Media:PatternDesign.pdf | '''Design Pattern''']] - [[Media:PresentationInteractiveDisplay.pdf|Présentation Intermédiaire]]
|-

!scope="row"| 4
| [[Sonotone]]
| LECORPS, VOUTAT, Hattinguais
| Maisonnasse, Richard
| [[Projets-2015-2016-Sonotone| '''Fiche''']] - [[Projets-2015-2016-Sonotone-SRS | '''SRS''']]
| [https://github.com/Gorgorot38/Sonotone-RICM4 '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SRS_Consultant_Sonotone_4.pdf|Rapport_Consultant]] - [[Media:pattern_sonotone.pdf|Pattern]] - [[Media:Soutenance.pdf|Soutenance_miparcours]]
|-

!scope="row"| 5
| [[Sous-titre_en_temps_r%C3%A9el_d%27un_cours| Sous-titre d'un cours en temps réel]]
| LECHEVALLIER, BUI, OUNISSI
| Maisonnasse
| [[LiveSubtitles| '''Fiche''']]
| [https://github.com/Lechevallier/RealTimeSubtitles '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media: SRS_Groupe_5.pdf| Rapport Consultant]]
|-

!scope="row"| 6
| [[GrenobloisFuté]]
| MOURET, DELAPORTE, Lucidarme
| Nicolas Palix
| [[GrenobleFuté| '''Fiche''']]
| [https://github.com/Lucidarme/Osmand.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_G14.pdf|Rapport Consultant]]
|-

!scope="row"| 7
| [[Streaming en stéréoscopie]]
| ZHAO ZILONG, HAMMOUTI
| Maisonnasse
| [[Projets-2015-2016-Streaming-Stereoscopie| '''Fiche''']] - [[SRS - Streaming en stéréoscopie | '''SRS''']]
| [https://github.com/zhao-zilong/streaming_stereo '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:bruel_medewou_ndiaye.pdf|Rapport_consultant]] - [[Media:streaming.pdf|mi-parcours]]
|-

!scope="row"| 8
| [[PersyCup2016]]
| BIN, ZEGAOUI, ELLAPIN
| Donsez, Maisonnasse
| [[PersyCup| '''Fiche''']]
| [https://github.com/legominstorm/lego '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SoutenanceMiParcours-Persycup2016.pdf|Soutenance Mi-parcours]]
|-

!scope="row"| 9
| [[Services étendus pour le modèle de composants iPOPO pour Python]]
| FOUNAS, HALLAL, GATTAZ
| Calmant & Donsez
| [[Proj-2015-2016-Extensions_IPOPO | '''Fiche''']] - [[Proj-2015-2016-Extensions_IPOPO/SRS | '''SRS''']] - [[Proj-2015-2016-Extensions_IPOPO/UML | '''UML''']]
| [https://github.com/abdelazizFounas/ipopo/tree/tlsremote '''github IPOPO'''] [https://github.com/gattazr/IPOPO-Remote-Client '''github IPOPO Client''']
| [[Media:9_RapportProjet.pdf|Rapport]] - [[Media:9_TransparentsProojet.pdf|Transparents]] - [[Media:9_FlyerProjet.pdf|Flyer]] - [[Media:3-SRS-Pres.pdf| Rapport Consultant]] - [[Media:9_PatternStrat.pdf|Pattern Design]] - [[Media:9_Mid-Presentation.pdf|Mid Presentation]] - [[Media:9_Gantt.pdf|Gantt]]
|-

!scope="row"| 10
| [[IndoorGeoloc2016]]
| ARRADA - CRASTES - FAURE - STOIAN
| Donsez
| [[Proj-2015-2016-IndoorGeoloc/Fiche| '''Fiche''']] - [[Proj-2015-2016-IndoorGeoloc/SRS|SRS]]
| [https://github.com/QuentinFA/Geoloc_Indoor '''github''']
| [[Media:Proj-2015-2016-IndoorGeoloc/RapportProjet.pdf|Rapport]] - [[Media:Proj-2015-2016-IndoorGeoloc/TransparentsProjet.pdf|Transparents]] - [[Media:Proj-2015-2016-IndoorGeoloc/FlyerProjet.pdf|Flyer]] - [[Media: SRSGroupe17.pdf| Rapport Consultant]] - [[Media:Mi_parcours.pdf|Mid presentation]]
|-

!scope="row"| 11
| [[UPnPOpenHAB2016]]
| Medewou , Ndiyae Yacine , Bruel Anna
| Didier Donsez & Jérome Maisonnasse
| [[Proj-Openhab-2016| '''Fiche''']] - [[Proj-2015-2016-Int%C3%A9gration_de_cam%C3%A9ra_de_surveillance_UPnP_%C3%A0_Openhab/SRS| '''SRS''']] - [[Proj-Openhab/UML| '''UML''']]
| [https://github.com/openHab-UPnP '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_ZHAO_HAMMOUTI.pdf|Rapport Consultant]] - [[Media:pattern_ZHAO_HAMMOUTI.pdf|Patterns]] - [[Media:fichier.pdf|Mini soutenance]]
|-

!scope="row"| 12
| [[Sign2Speech]]
| NIOGRET, NOGUERON, TITH
| Didier Donsez
| [[sign2speech_ricm4_2015_2016| '''Fiche''']] - [[SRS - Sign2Speech | '''SRS''']] - [[UML | '''UML''']]
| [https://github.com/SignToSpeech-Project '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:12-Sign2Speech-RapportConsultant.pdf|Rapport Consultant]] - [[Media:12-Sign2Speech-MidPres.pdf|Mid presentation]]
|-

!scope="row"| 13
| [[AstroImage]]
| RACHEX, BLANC, GERRY
| Olivier Richard et Bruno Bzeznik
| [[Proj-2015-2016-Astroimage/Fiche| '''Fiche''']] - [[AstroImage/SRS | '''SRS''']] - [[Media:AstroImage-UML.pdf | '''UML''']]
| [https://github.com/nicolas-blanc/AstroImage '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:13-AstroImage-RapportConsultant.pdf|Rapport Consultant]] - [https://docs.google.com/presentation/d/15F8DRktwmOuSNabdxMASniyr-TIiRzGNNG1mOhcoSnk/edit?usp=sharing '''Patterns''']
|-

!scope="row"| 14
| [[Tachymètre]]
| MACE, NOUGUIER, RAMEL
| Olivier Gattaz
| [[Fiche - Tachymètre | '''Fiche''']] - [[SRS - Tachymètre| '''SRS''']] - [[UML - Tachymètre| '''UML''']]
| [https://github.com/Quego/Tachymetre '''github - Tachymètre''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:srs_tachymetre.pdf|Rapport consultant]] - [[Media:14_PatternDesign.pdf‎ | Pattern Design]] - [[Media:Tachymetre_Presentation.pdf‎ | Présentation]]
|-

!scope="row"| 15
| [[SmartProjector]]
| BRANGER, HABLOT
| Donsez, Maisonnasse
| [[Fiche_SmartProjector_ricm4_2015_2016| '''Fiche''']] - [[SRS - SmartProjector| '''SRS''']] - [[UML - SmartProjector| '''UML''']]
| [https://github.com/P0ppoff/SmartProjector '''github''']
| [[Rapport]] - [[Media:PresentationPorjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:Gl_groupe16.pdf|Rapport Consultant]] - [http://air.imag.fr/index.php/Patron_de_conception_-_SmartProjector patterns]
|-

|}

===Liste de projets===

* [[Dashboard pour gestionnaire de tâches et de ressources]], Olivier Richard
* [[Moteur distribué d'exécution de commande]], Olivier Richard
* [[Environnement d'expérimentation de pour NVIDIA Shield (Tegra X1)]], Olivier Richard
* [[Speeding Simplified Script Language]], Olivier Richard

* Aide (Open-Source)au Handicap Auditif, avec Didier Donsez, Jérome Maisonnasse, Marie-Paule Balicco (SAH UGA) et Nicolas Vuillerme
** [[Borne interactive]] (1 sujet)
** [[Sonotone]] (1 sujet)
** [[Sous-titre en temps réel d'un cours]] (1 sujet)
* [[GrenobloisFuté]] Couche trafic sur OsmAnd avec un greffon. Données dynamique de la métro. Dvp Android. Nicolas Palix.
* [[GeoDiff]] Production, visualisation, fusion de variations (diff) sur de l'information géocodée : Nicolas Palix
* [[Smart campus augmenté et contributif]] Didier Donsez, Vivien Quema

* [[Streaming en stéréoscopie]] sur [[WebRTC]] avec rendu sur [[Oculus]] pour le robot [[RobAIR]], Jérôme Maisonnasse. ([http://gstconf.ubicast.tv/videos/stereoscopic-3d-video/ voir]).
* [[STM32F7]] : Mise en oeuvre de la chaîne de compilation sous Linux avec [[OpenSTM32]] et [[OpenOCD]]. Nicolas Palix
* [[PersyCup2016]] : Persyval Robocup, Didier Donsez, Vivien Quema, Jérome Maisonnasse. (3 étudiants)
* [[Services étendus pour le modèle de composants iPOPO pour Python]], Didier Donsez & Thomas Calmant. (2 étudiants)
* [[SmartClassRoom2016|Développement d'une interface partagée pour tables tactiles (projet SmartClassRoom)]], Didier Donsez, Jérôme Maisonnasse. (2 étudiants)
* [[iRock2016|iRock : surveillance de glissement de terrains]], Didier Donsez & Vivien Quema
* [[IndoorGeoloc2016|Géolocalisation in-door au moyen de balises (beacon) BLE et Wifi à base de STM32 et de balises iBeacon & AltBeacon]], Didier Donsez & Vivien Quema
* [[UPnPOpenHAB2016|Intégration et gestion de caméras de surveillance UPnP dans la plateforme domotique open-source OpenHAB et myOpenHAB]], Didier Donsez & Jérome Maisonnasse.

'''Projets non prioritaires'''

* [[Liveprogramming with Kivy]], Olivier Richard
* [[AstroImage]] production d'image d'astronomie, Olivier Richard et Bruno Bzeznik
* [[G-code Cruncher]] Controle de machine CNC (Nucleo grbl + esp8266 + Sdcard), Olivier Richard
* [[Intégration OpenHAB / OpenTele]] Nicolas Palix

==RICM5==

===Projet Semestre S10===

Enseignant responsable : Didier Donsez

Démarrage : Lundi 25/01 à 10H30-12H30, P253 (Rendez-vous devant la salle AIR) - Visioconf pour Thibaut Cordier

Soutenance : Jeudi 17/03 à 13H00-17H00, salle P043 (Polytech Grenoble)puis en salle C005 (Batiment C)

Etudiants : RICM5 + 8 étudiants Avosti DUT RT

Rappel séances MPI
* Séance 1 : mardi 26 janvier après midi - Stéphanie Diligent
* Séance 2 : mardi 2 février après midi - Stéphanie Diligent
* Séance 3 : lundi 8 février matin - Emmanuelle Tréhoust
* Séance 4 : jeudi 11 février matin - Emmanuelle Tréhoust
* Séance 5 : lundi 21 mars matin - Stéphanie Diligent et Emmanuelle Tréhoust

=====Soutenances=====
Planning:
* Bossa (13H00-13H40 en salle P043)
* Immersion EDF (13H45-14H25 en salle P043)
* IaaS Docker (14H30-15H10 en salle P043)
* SmartCampus (15H15-15H55 en salle P043 et salle P259 AIR)
* SmartClassRoom (16H15-16H55 en C005)
* Pot d' "Au Revoir" (17H00-1800 en C005)

Instructions:
*Chaque soutenance comporte 15 minutes de présentation, 15 minutes de démonstration et 10 minutes de questions. Un transparent doit être consacré au travail confié et réalisé par les étudiants en DUT (AVOSTI).
* Répétez plusieurs fois votre présentation et votre démonstration.
* L'ensemble des documents (y compris photos, vidéos et ''[[Logiciels#Screencast|screencast]]s'') doivent être accessibles depuis le tableau ci-dessous et dans chaque fiche de suivi. Prévoyez une copie sur clé USB.
* Les étudiants vous accompagnent lors de votre soutenance.
* '''TOUT Le matériel prêté devra être rapporté et restitué dans un sac cabas lors de la soutenance.'''

=====Projets=====
{|class="wikitable alternance"
|+ Affectation des projets RICM5 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
!scope="col"| Documents
|-

!scope="row"| 1
| [http://air.imag.fr/index.php/IaaS_collaboratif_avec_Docker IaaS - Docker]
| Eudes Robin, Damotte Alan, Barthelemy Romain, Mammar Malek, Guo Kai
| Didier Donsez
| [[Projets-2015-2016-IaaS_Docker| '''Fiche''']] - [[Projets-2015-2016-IaaS_Docker-SRS| '''SRS''']]
| [https://github.com/EudesRobin/iaas-collaboratif '''github''']
| [[Media:RapportMPI_Iaas.pdf|Rapport MPI]] - [[Media:Transparents_IaaS.pdf|Transparents]] - [[Media:Flyer_IaaS.pdf|Flyer]] - [https://youtu.be/qtqgZNrgcRc '''Screencast''']
|-
!scope="row"| 2
| [http://air.imag.fr/index.php/Portage_de_Bossa Portage de Bossa sur le Kernel Linux 4x]
| Eric Michel Fotsing, Ombeline Rossi, Longfei Yao
| Nicolas Palix, Didier Donsez
| [[Projets-2015-2016-Portage_Bossa| '''Fiche''']] - [[Projets-2015-2016-Portage_Bossa-SRS| '''SRS''']]
| [https://github.com/ZenithKaizer/ '''github''']
| [[Media:Rapport_Bossa.pdf|Rapport]] - [[Media:Transparents_Bossa.pdf|Transparents]] - [[Media:Flyer_Bossa.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 3
| [[Visite immersive en réalité virtuelle dans une usine avec EDF]]
| Adam Christophe, Aissanou Sarah, Klipffel Tararaina, Qian Jean, Zominy Laurent
| Didier Donsez, Georges-Pierre Bonneau, Thibaut Cordier (EDF)
| [[Projets-2015-2016-VisiteImmersiveEDF| '''Fiche''']]
| [https://github.com/VisiteImmersiveEDF '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 4
| [[Contribution à OpenSmartCampus]] (voir http://data.beta.metropolegrenoble.fr/)
| Quentin Torck, Vivien Michel, Jérémy Hammerer, Rama Codazzi, Zhengmeng Zhang
| Didier Donsez, Vivien Quéma
| [[Projets-2015-2016-OpenSmartCampus| '''Fiche''']]
| [https://github.com/quentin74/SmartCampus.git '''github''']
| [[Media:RapportProjetOpenSmartCampus2016.pdf|Rapport]] - [[Media:TransparentsProojetOpenSmartCampus2016.pdf|Transparents]] - [[Media:FlyerProjetOpenSmartCampus2016.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 5
| [[Contribution à SmartClassRoom]] (Interfaces tactiles distribuées et partagées)
| Saussac Thibault, Toussaint Sébastien, Hamdani Youcef, Zoppello Sebastien, Melik sak, Mesnier Vincent
| Jérôme Maisonnasse, Didier Donsez
| [[Projets-2015-2016-SmartClassRoom| '''Fiche''']] - [[Projets-2015-2016-SmartClassRoom/SRS| '''SRS''']]
| [https://github.com/XXXX '''github''']
| [[Media:RapportProjetSmartClassRoom.pdf|Rapport]] - [[Media:TransparentsProjetSmartClassRoom.pdf|Transparents]] - [[Media:FlyerProjetSmartClassRoom.pdf|Flyer]] - Photos - [https://youtu.be/FEwoA4S9rsM '''Screencast/Vidéo''']
|-

|}

===Projets annulés et reportés===
* Projet avec [[Tango Project]] (Annulé)
* Hack the Beam, Didier Donsez & Jérôme Maisonnasse.
* [[Algorithmes de suivi de personnes pour robot de téléprésence RobAIR]] (Jérôme Maisonnasse, Didier Donsez)

=M2PGI=
==[[Projets M2PGI Services Machine-to-Machine|Projet Services Machine-to-Machine]]==
* [[PM2M/2016/TP|Sujet et groupes]]

Projets 2015-2016

2016-03-17T07:38:43Z

Alan.Damotte: /* Projets */

<<[[Projets 2014-2015]] | [[Projets]] | [[Projets 2016-2017]]>>
=RICM=
==RICM3==

==RICM4==
===Projet Semestre S8===

Enseignants responsables : Olivier Richard, Didier Donsez

* '''Evaluation à mi-parcours le lundi 7 mars''': Format: 10min (5min de présentation 3 slides au plus, 5min de discussion). Cette évaluation sera prise en compte dans la note finale.

'''Consignes générales:'''

* '''Vous devez être pro-actifs !!!''': Si des points sont pas ou mals spécifiés, vous le faîtes et vous justifiez vos choix. Pour les problèmes techniques éventuels vous pouvez: vous creusez la question, vous contactez l'auteur du code si il y a lieux, vous faites un rapport de bug ('''Attention:''' ca se prépare !), vous soumettez un patch, vous contactez l'enseignant ou la personne suivant le projet.

* '''Vous devez maintenir une fiche de suivi de projet''': elle doit être mise à jour chaque semaine, elle rassemble les élements essentiels du projet, elle
indique les évolutions du projet et présente sa feuille de route. '''Note:''' le nom de la fiche doit être composé du nom du projet et suffixé par ricm4_2015_2016.

* '''Vous devez utiliser un logiciel de gestion de version''' pour vos développements comme [http://en.wikipedia.org/wiki/Git_%28software%29 git ] et nous vous conseillons d'utiliser le site [https://github.com github] pour l'hébergement de votre dépôt public.

* Les document public (exemple sur github) doivent être rédigés en anglais (README, documentation, commentaires de code, nom de variables et de fonctions). Une bonnification sera accordée si le rapport et les transparents sont en anglais (la soutenance sera en francais).

{|class="wikitable alternance"
|+ Affectation des projets RICM4 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [[Dashboard pour gestionnaire de tâches et de ressources]]
| CROUZET, MATHIEU
| Richard
| [[Projets-2015-2016-DashBoard| '''Fiche''']]
| [https://github.com/MatthieuCrouzet/Projet4A '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_groupe1.pdf|Rapport Consultant]] - [[Media:Paterns.pdf|Patterns]] - [[Media:PresentationDashboard.pdf|Presentation]]
|-

!scope="row"| 2
| [[Speeding Simplified Script Language]]
| POPEK, BERTRAND-DALECHAMPS, WEI
| Richard
| [[Projets-2015-2016-SSSL| '''Fiche''']] - [[SSSL-UML| '''UML''']]
| [https://github.com/FlorianPO/Speeding-Simplified-Script-Language.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:Groupe2_AIR.pdf|Rapport Consultant]] - [[Media:PresentationProjet.pdf|Presentation]]
|-

!scope="row"| 3
| [[Borne interactive]]
| DUNAND - NAVARRO - REVEL
| Maisonnasse
| [[Projets-2015-2016-Borne-Interactive| '''Fiche''']] - [[Projets-2015-2016-Borne-Interactive-SRS | '''SRS''']] - [[Projets-2015-2016-Borne-Interactive/UML_Diagrams | '''UML''']]
| [https://github.com/Kant73/InteractiveDisplay '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:IPopo.pdf|Rapport Consultant]] - [[Media:PatternDesign.pdf | '''Design Pattern''']] - [[Media:PresentationInteractiveDisplay.pdf|Présentation Intermédiaire]]
|-

!scope="row"| 4
| [[Sonotone]]
| LECORPS, VOUTAT, Hattinguais
| Maisonnasse, Richard
| [[Projets-2015-2016-Sonotone| '''Fiche''']] - [[Projets-2015-2016-Sonotone-SRS | '''SRS''']]
| [https://github.com/Gorgorot38/Sonotone-RICM4 '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SRS_Consultant_Sonotone_4.pdf|Rapport_Consultant]] - [[Media:pattern_sonotone.pdf|Pattern]] - [[Media:Soutenance.pdf|Soutenance_miparcours]]
|-

!scope="row"| 5
| [[Sous-titre_en_temps_r%C3%A9el_d%27un_cours| Sous-titre d'un cours en temps réel]]
| LECHEVALLIER, BUI, OUNISSI
| Maisonnasse
| [[LiveSubtitles| '''Fiche''']]
| [https://github.com/Lechevallier/RealTimeSubtitles '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media: SRS_Groupe_5.pdf| Rapport Consultant]]
|-

!scope="row"| 6
| [[GrenobloisFuté]]
| MOURET, DELAPORTE, Lucidarme
| Nicolas Palix
| [[GrenobleFuté| '''Fiche''']]
| [https://github.com/Lucidarme/Osmand.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_G14.pdf|Rapport Consultant]]
|-

!scope="row"| 7
| [[Streaming en stéréoscopie]]
| ZHAO ZILONG, HAMMOUTI
| Maisonnasse
| [[Projets-2015-2016-Streaming-Stereoscopie| '''Fiche''']] - [[SRS - Streaming en stéréoscopie | '''SRS''']]
| [https://github.com/zhao-zilong/streaming_stereo '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:bruel_medewou_ndiaye.pdf|Rapport_consultant]] - [[Media:streaming.pdf|mi-parcours]]
|-

!scope="row"| 8
| [[PersyCup2016]]
| BIN, ZEGAOUI, ELLAPIN
| Donsez, Maisonnasse
| [[PersyCup| '''Fiche''']]
| [https://github.com/legominstorm/lego '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SoutenanceMiParcours-Persycup2016.pdf|Soutenance Mi-parcours]]
|-

!scope="row"| 9
| [[Services étendus pour le modèle de composants iPOPO pour Python]]
| FOUNAS, HALLAL, GATTAZ
| Calmant & Donsez
| [[Proj-2015-2016-Extensions_IPOPO | '''Fiche''']] - [[Proj-2015-2016-Extensions_IPOPO/SRS | '''SRS''']] - [[Proj-2015-2016-Extensions_IPOPO/UML | '''UML''']]
| [https://github.com/abdelazizFounas/ipopo/tree/tlsremote '''github IPOPO'''] [https://github.com/gattazr/IPOPO-Remote-Client '''github IPOPO Client''']
| [[Media:9_RapportProjet.pdf|Rapport]] - [[Media:9_TransparentsProojet.pdf|Transparents]] - [[Media:9_FlyerProjet.pdf|Flyer]] - [[Media:3-SRS-Pres.pdf| Rapport Consultant]] - [[Media:9_PatternStrat.pdf|Pattern Design]] - [[Media:9_Mid-Presentation.pdf|Mid Presentation]] - [[Media:9_Gantt.pdf|Gantt]]
|-

!scope="row"| 10
| [[IndoorGeoloc2016]]
| ARRADA - CRASTES - FAURE - STOIAN
| Donsez
| [[Proj-2015-2016-IndoorGeoloc/Fiche| '''Fiche''']] - [[Proj-2015-2016-IndoorGeoloc/SRS|SRS]]
| [https://github.com/QuentinFA/Geoloc_Indoor '''github''']
| [[Media:Proj-2015-2016-IndoorGeoloc/RapportProjet.pdf|Rapport]] - [[Media:Proj-2015-2016-IndoorGeoloc/TransparentsProjet.pdf|Transparents]] - [[Media:Proj-2015-2016-IndoorGeoloc/FlyerProjet.pdf|Flyer]] - [[Media: SRSGroupe17.pdf| Rapport Consultant]] - [[Media:Mi_parcours.pdf|Mid presentation]]
|-

!scope="row"| 11
| [[UPnPOpenHAB2016]]
| Medewou , Ndiyae Yacine , Bruel Anna
| Didier Donsez & Jérome Maisonnasse
| [[Proj-Openhab-2016| '''Fiche''']] - [[Proj-2015-2016-Int%C3%A9gration_de_cam%C3%A9ra_de_surveillance_UPnP_%C3%A0_Openhab/SRS| '''SRS''']] - [[Proj-Openhab/UML| '''UML''']]
| [https://github.com/openHab-UPnP '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_ZHAO_HAMMOUTI.pdf|Rapport Consultant]] - [[Media:pattern_ZHAO_HAMMOUTI.pdf|Patterns]] - [[Media:fichier.pdf|Mini soutenance]]
|-

!scope="row"| 12
| [[Sign2Speech]]
| NIOGRET, NOGUERON, TITH
| Didier Donsez
| [[sign2speech_ricm4_2015_2016| '''Fiche''']] - [[SRS - Sign2Speech | '''SRS''']] - [[UML | '''UML''']]
| [https://github.com/SignToSpeech-Project '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:12-Sign2Speech-RapportConsultant.pdf|Rapport Consultant]] - [[Media:12-Sign2Speech-MidPres.pdf|Mid presentation]]
|-

!scope="row"| 13
| [[AstroImage]]
| RACHEX, BLANC, GERRY
| Olivier Richard et Bruno Bzeznik
| [[Proj-2015-2016-Astroimage/Fiche| '''Fiche''']] - [[AstroImage/SRS | '''SRS''']] - [[Media:AstroImage-UML.pdf | '''UML''']]
| [https://github.com/nicolas-blanc/AstroImage '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:13-AstroImage-RapportConsultant.pdf|Rapport Consultant]] - [https://docs.google.com/presentation/d/15F8DRktwmOuSNabdxMASniyr-TIiRzGNNG1mOhcoSnk/edit?usp=sharing '''Patterns''']
|-

!scope="row"| 14
| [[Tachymètre]]
| MACE, NOUGUIER, RAMEL
| Olivier Gattaz
| [[Fiche - Tachymètre | '''Fiche''']] - [[SRS - Tachymètre| '''SRS''']] - [[UML - Tachymètre| '''UML''']]
| [https://github.com/Quego/Tachymetre '''github - Tachymètre''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:srs_tachymetre.pdf|Rapport consultant]] - [[Media:14_PatternDesign.pdf‎ | Pattern Design]] - [[Media:Tachymetre_Presentation.pdf‎ | Présentation]]
|-

!scope="row"| 15
| [[SmartProjector]]
| BRANGER, HABLOT
| Donsez, Maisonnasse
| [[Fiche_SmartProjector_ricm4_2015_2016| '''Fiche''']] - [[SRS - SmartProjector| '''SRS''']] - [[UML - SmartProjector| '''UML''']]
| [https://github.com/P0ppoff/SmartProjector '''github''']
| [[Rapport]] - [[Media:PresentationPorjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:Gl_groupe16.pdf|Rapport Consultant]] - [http://air.imag.fr/index.php/Patron_de_conception_-_SmartProjector patterns]
|-

|}

===Liste de projets===

* [[Dashboard pour gestionnaire de tâches et de ressources]], Olivier Richard
* [[Moteur distribué d'exécution de commande]], Olivier Richard
* [[Environnement d'expérimentation de pour NVIDIA Shield (Tegra X1)]], Olivier Richard
* [[Speeding Simplified Script Language]], Olivier Richard

* Aide (Open-Source)au Handicap Auditif, avec Didier Donsez, Jérome Maisonnasse, Marie-Paule Balicco (SAH UGA) et Nicolas Vuillerme
** [[Borne interactive]] (1 sujet)
** [[Sonotone]] (1 sujet)
** [[Sous-titre en temps réel d'un cours]] (1 sujet)
* [[GrenobloisFuté]] Couche trafic sur OsmAnd avec un greffon. Données dynamique de la métro. Dvp Android. Nicolas Palix.
* [[GeoDiff]] Production, visualisation, fusion de variations (diff) sur de l'information géocodée : Nicolas Palix
* [[Smart campus augmenté et contributif]] Didier Donsez, Vivien Quema

* [[Streaming en stéréoscopie]] sur [[WebRTC]] avec rendu sur [[Oculus]] pour le robot [[RobAIR]], Jérôme Maisonnasse. ([http://gstconf.ubicast.tv/videos/stereoscopic-3d-video/ voir]).
* [[STM32F7]] : Mise en oeuvre de la chaîne de compilation sous Linux avec [[OpenSTM32]] et [[OpenOCD]]. Nicolas Palix
* [[PersyCup2016]] : Persyval Robocup, Didier Donsez, Vivien Quema, Jérome Maisonnasse. (3 étudiants)
* [[Services étendus pour le modèle de composants iPOPO pour Python]], Didier Donsez & Thomas Calmant. (2 étudiants)
* [[SmartClassRoom2016|Développement d'une interface partagée pour tables tactiles (projet SmartClassRoom)]], Didier Donsez, Jérôme Maisonnasse. (2 étudiants)
* [[iRock2016|iRock : surveillance de glissement de terrains]], Didier Donsez & Vivien Quema
* [[IndoorGeoloc2016|Géolocalisation in-door au moyen de balises (beacon) BLE et Wifi à base de STM32 et de balises iBeacon & AltBeacon]], Didier Donsez & Vivien Quema
* [[UPnPOpenHAB2016|Intégration et gestion de caméras de surveillance UPnP dans la plateforme domotique open-source OpenHAB et myOpenHAB]], Didier Donsez & Jérome Maisonnasse.

'''Projets non prioritaires'''

* [[Liveprogramming with Kivy]], Olivier Richard
* [[AstroImage]] production d'image d'astronomie, Olivier Richard et Bruno Bzeznik
* [[G-code Cruncher]] Controle de machine CNC (Nucleo grbl + esp8266 + Sdcard), Olivier Richard
* [[Intégration OpenHAB / OpenTele]] Nicolas Palix

==RICM5==

===Projet Semestre S10===

Enseignant responsable : Didier Donsez

Démarrage : Lundi 25/01 à 10H30-12H30, P253 (Rendez-vous devant la salle AIR) - Visioconf pour Thibaut Cordier

Soutenance : Jeudi 17/03 à 13H00-17H00, salle P043 (Polytech Grenoble)puis en salle C005 (Batiment C)

Etudiants : RICM5 + 8 étudiants Avosti DUT RT

Rappel séances MPI
* Séance 1 : mardi 26 janvier après midi - Stéphanie Diligent
* Séance 2 : mardi 2 février après midi - Stéphanie Diligent
* Séance 3 : lundi 8 février matin - Emmanuelle Tréhoust
* Séance 4 : jeudi 11 février matin - Emmanuelle Tréhoust
* Séance 5 : lundi 21 mars matin - Stéphanie Diligent et Emmanuelle Tréhoust

=====Soutenances=====
Planning:
* Bossa (13H00-13H40 en salle P043)
* Immersion EDF (13H45-14H25 en salle P043)
* IaaS Docker (14H30-15H10 en salle P043)
* SmartCampus (15H15-15H55 en salle P043 et salle P259 AIR)
* SmartClassRoom (16H15-16H55 en C005)
* Pot d' "Au Revoir" (17H00-1800 en C005)

Instructions:
*Chaque soutenance comporte 15 minutes de présentation, 15 minutes de démonstration et 10 minutes de questions. Un transparent doit être consacré au travail confié et réalisé par les étudiants en DUT (AVOSTI).
* Répétez plusieurs fois votre présentation et votre démonstration.
* L'ensemble des documents (y compris photos, vidéos et ''[[Logiciels#Screencast|screencast]]s'') doivent être accessibles depuis le tableau ci-dessous et dans chaque fiche de suivi. Prévoyez une copie sur clé USB.
* Les étudiants vous accompagnent lors de votre soutenance.
* '''TOUT Le matériel prêté devra être rapporté et restitué dans un sac cabas lors de la soutenance.'''

=====Projets=====
{|class="wikitable alternance"
|+ Affectation des projets RICM5 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
!scope="col"| Documents
|-

!scope="row"| 1
| [http://air.imag.fr/index.php/IaaS_collaboratif_avec_Docker IaaS - Docker]
| Eudes Robin, Damotte Alan, Barthelemy Romain, Mammar Malek, Guo Kai
| Didier Donsez
| [[Projets-2015-2016-IaaS_Docker| '''Fiche''']] - [[Projets-2015-2016-IaaS_Docker-SRS| '''SRS''']]
| [https://github.com/EudesRobin/iaas-collaboratif '''github''']
| [[Media:RapportMPI_Iaas.pdf|Rapport]] - [[Media:Transparents_IaaS.pdf|Transparents]] - [[Media:Flyer_IaaS.pdf|Flyer]] - [https://youtu.be/qtqgZNrgcRc '''Screencast''']
|-
!scope="row"| 2
| [http://air.imag.fr/index.php/Portage_de_Bossa Portage de Bossa sur le Kernel Linux 4x]
| Eric Michel Fotsing, Ombeline Rossi, Longfei Yao
| Nicolas Palix, Didier Donsez
| [[Projets-2015-2016-Portage_Bossa| '''Fiche''']] - [[Projets-2015-2016-Portage_Bossa-SRS| '''SRS''']]
| [https://github.com/ZenithKaizer/ '''github''']
| [[Media:Rapport_Bossa.pdf|Rapport]] - [[Media:Transparents_Bossa.pdf|Transparents]] - [[Media:Flyer_Bossa.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 3
| [[Visite immersive en réalité virtuelle dans une usine avec EDF]]
| Adam Christophe, Aissanou Sarah, Klipffel Tararaina, Qian Jean, Zominy Laurent
| Didier Donsez, Georges-Pierre Bonneau, Thibaut Cordier (EDF)
| [[Projets-2015-2016-VisiteImmersiveEDF| '''Fiche''']]
| [https://github.com/VisiteImmersiveEDF '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 4
| [[Contribution à OpenSmartCampus]] (voir http://data.beta.metropolegrenoble.fr/)
| Quentin Torck, Vivien Michel, Jérémy Hammerer, Rama Codazzi, Zhengmeng Zhang
| Didier Donsez, Vivien Quéma
| [[Projets-2015-2016-OpenSmartCampus| '''Fiche''']]
| [https://github.com/quentin74/SmartCampus.git '''github''']
| [[Media:RapportProjetOpenSmartCampus2016.pdf|Rapport]] - [[Media:TransparentsProojetOpenSmartCampus2016.pdf|Transparents]] - [[Media:FlyerProjetOpenSmartCampus2016.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 5
| [[Contribution à SmartClassRoom]] (Interfaces tactiles distribuées et partagées)
| Saussac Thibault, Toussaint Sébastien, Hamdani Youcef, Zoppello Sebastien, Melik sak, Mesnier Vincent
| Jérôme Maisonnasse, Didier Donsez
| [[Projets-2015-2016-SmartClassRoom| '''Fiche''']] - [[Projets-2015-2016-SmartClassRoom/SRS| '''SRS''']]
| [https://github.com/XXXX '''github''']
| [[Media:RapportProjetSmartClassRoom.pdf|Rapport]] - [[Media:TransparentsProjetSmartClassRoom.pdf|Transparents]] - [[Media:FlyerProjetSmartClassRoom.pdf|Flyer]] - Photos - [https://youtu.be/FEwoA4S9rsM '''Screencast/Vidéo''']
|-

|}

===Projets annulés et reportés===
* Projet avec [[Tango Project]] (Annulé)
* Hack the Beam, Didier Donsez & Jérôme Maisonnasse.
* [[Algorithmes de suivi de personnes pour robot de téléprésence RobAIR]] (Jérôme Maisonnasse, Didier Donsez)

=M2PGI=
==[[Projets M2PGI Services Machine-to-Machine|Projet Services Machine-to-Machine]]==
* [[PM2M/2016/TP|Sujet et groupes]]

Projets-2015-2016-IaaS Docker

2016-03-16T10:13:30Z

Alan.Damotte: /* How to use the scripts? */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

== Archive content ==

*Scripts:
** coordinatorBuild: used to build coordinator and shinken images
** coordinatorStart: used to start coordinator and shinken images
** coordinatorStop: used to stop coordinator and shinken images
** userAdd: add iaas user to host
** startProvider: main script that create iaas user, build all images, and start coordinator and monitoring containers

*Folders:
** containerManagment: contains all the scripts which are copied to iaas user home and will be used to manage containers
** coordinator: contains Dockerfile and script to initialize coordinator image
** docker_shinken: contains Dockerfiles and scripts to initialize monitoring image (note: we now use cAdvisor to monitor our containers)
** images: contains Dockerfiles and scripts to initialize main images (ubuntu, debian, centos)

== How to use the scripts? ==

* To initialize and start coordinator and monitoring containers:
./startProvider.sh

* To start coordinator and monitoring containers (will have no effect if they are already started, not necessary if startProvider script is used):
./coordinatorStart.sh

* To stop coordinator and monitoring containers:
./coordinatorStop.sh

* If you need to update coordinator or shinken image:

:* Update coordinator image
./coordinatorBuild.sh update coordinator

:* Update shinken image (this is not necessary anymore since we use cAdvisor)
./coordinatorBuild.sh update shinken

:* Update both coordinator and shinken image
./coordinatorBuild.sh update all

There is no need to use the other scripts the archive contains.

= Monitoring with cAdvisor=

We were looking for a light and efficient tool to monitor the instances running on provider side. We first used Shinken but we found this tool way to heavy for what we wanted to do. Then we found cAdvisor.

Google first created cAdvisor to monitor their own lmctfy containers and added Docker container support (those that use the default lib container execdriver). cAdvisor is short for Container Advisor. You can use cAdvisor to get a detailed look into the resource usage and the various performance characteristics of your running containers. cAdvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB.

If you want to monitor all of the containers you have running on a host, you can deploy the cAdvisor image as a container in that same host. It will then have access to the resource usage and performance characteristics of your other containers in that host.

You can check cAdvisor is running by pointing your browser to the host’s IP or hostname and port 8080. If running locally, try http://localhost:8080. This will bring up the built-in Web UI.

[[File:cAdvisor.png|center|thumb|500px|cAdvisor]]

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-16T10:13:09Z

Alan.Damotte: /* How to use the scripts? */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

== Archive content ==

*Scripts:
** coordinatorBuild: used to build coordinator and shinken images
** coordinatorStart: used to start coordinator and shinken images
** coordinatorStop: used to stop coordinator and shinken images
** userAdd: add iaas user to host
** startProvider: main script that create iaas user, build all images, and start coordinator and monitoring containers

*Folders:
** containerManagment: contains all the scripts which are copied to iaas user home and will be used to manage containers
** coordinator: contains Dockerfile and script to initialize coordinator image
** docker_shinken: contains Dockerfiles and scripts to initialize monitoring image (note: we now use cAdvisor to monitor our containers)
** images: contains Dockerfiles and scripts to initialize main images (ubuntu, debian, centos)

== How to use the scripts? ==

* To initialize and start coordinator and monitoring containers:
./startProvider.sh

* To start coordinator and monitoring containers (will have no effect if they are already started, not necessary if startProvider script is used):
./coordinatorStart.sh

* To stop coordinator and monitoring containers:
./coordinatorStop.sh

* If you need to update coordinator or shinken image:

** Update coordinator image
./coordinatorBuild.sh update coordinator

** Update shinken image (this is not necessary anymore since we use cAdvisor)
./coordinatorBuild.sh update shinken

** Update both coordinator and shinken image
./coordinatorBuild.sh update all

There is no need to use the other scripts the archive contains.

= Monitoring with cAdvisor=

We were looking for a light and efficient tool to monitor the instances running on provider side. We first used Shinken but we found this tool way to heavy for what we wanted to do. Then we found cAdvisor.

Google first created cAdvisor to monitor their own lmctfy containers and added Docker container support (those that use the default lib container execdriver). cAdvisor is short for Container Advisor. You can use cAdvisor to get a detailed look into the resource usage and the various performance characteristics of your running containers. cAdvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB.

If you want to monitor all of the containers you have running on a host, you can deploy the cAdvisor image as a container in that same host. It will then have access to the resource usage and performance characteristics of your other containers in that host.

You can check cAdvisor is running by pointing your browser to the host’s IP or hostname and port 8080. If running locally, try http://localhost:8080. This will bring up the built-in Web UI.

[[File:cAdvisor.png|center|thumb|500px|cAdvisor]]

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-16T10:12:49Z

Alan.Damotte: /* How to use the scripts? */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

== Archive content ==

*Scripts:
** coordinatorBuild: used to build coordinator and shinken images
** coordinatorStart: used to start coordinator and shinken images
** coordinatorStop: used to stop coordinator and shinken images
** userAdd: add iaas user to host
** startProvider: main script that create iaas user, build all images, and start coordinator and monitoring containers

*Folders:
** containerManagment: contains all the scripts which are copied to iaas user home and will be used to manage containers
** coordinator: contains Dockerfile and script to initialize coordinator image
** docker_shinken: contains Dockerfiles and scripts to initialize monitoring image (note: we now use cAdvisor to monitor our containers)
** images: contains Dockerfiles and scripts to initialize main images (ubuntu, debian, centos)

== How to use the scripts? ==

* To initialize and start coordinator and monitoring containers:
./startProvider.sh

* To start coordinator and monitoring containers (will have no effect if they are already started, not necessary if startProvider script is used):
./coordinatorStart.sh

* To stop coordinator and monitoring containers:
./coordinatorStop.sh

* If you need to update coordinator or shinken image:

*Update coordinator image
./coordinatorBuild.sh update coordinator

*Update shinken image (this is not necessary anymore since we use cAdvisor)
./coordinatorBuild.sh update shinken

*Update both coordinator and shinken image
./coordinatorBuild.sh update all

There is no need to use the other scripts the archive contains.

= Monitoring with cAdvisor=

We were looking for a light and efficient tool to monitor the instances running on provider side. We first used Shinken but we found this tool way to heavy for what we wanted to do. Then we found cAdvisor.

Google first created cAdvisor to monitor their own lmctfy containers and added Docker container support (those that use the default lib container execdriver). cAdvisor is short for Container Advisor. You can use cAdvisor to get a detailed look into the resource usage and the various performance characteristics of your running containers. cAdvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB.

If you want to monitor all of the containers you have running on a host, you can deploy the cAdvisor image as a container in that same host. It will then have access to the resource usage and performance characteristics of your other containers in that host.

You can check cAdvisor is running by pointing your browser to the host’s IP or hostname and port 8080. If running locally, try http://localhost:8080. This will bring up the built-in Web UI.

[[File:cAdvisor.png|center|thumb|500px|cAdvisor]]

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-16T10:11:23Z

Alan.Damotte: /* Containers' automatic deployment */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

== Archive content ==

*Scripts:
** coordinatorBuild: used to build coordinator and shinken images
** coordinatorStart: used to start coordinator and shinken images
** coordinatorStop: used to stop coordinator and shinken images
** userAdd: add iaas user to host
** startProvider: main script that create iaas user, build all images, and start coordinator and monitoring containers

*Folders:
** containerManagment: contains all the scripts which are copied to iaas user home and will be used to manage containers
** coordinator: contains Dockerfile and script to initialize coordinator image
** docker_shinken: contains Dockerfiles and scripts to initialize monitoring image (note: we now use cAdvisor to monitor our containers)
** images: contains Dockerfiles and scripts to initialize main images (ubuntu, debian, centos)

== How to use the scripts? ==

* To initialize and start coordinator and monitoring containers:
./startProvider.sh

* To start coordinator and monitoring containers (will have no effect if they are already started, not necessary if startProvider script is used):
./coordinatorStart.sh

* To stop coordinator and monitoring containers:
./coordinatorStop.sh

* If you need to update coordinator or shinken image:

** Update coordinator image
./coordinatorBuild.sh update coordinator

** Update shinken image (this is not necessary anymore since we use cAdvisor)
./coordinatorBuild.sh update shinken

**Update both coordinator and shinken image
./coordinatorBuild.sh update all

There is no need to use the other scripts the archive contains.

= Monitoring with cAdvisor=

We were looking for a light and efficient tool to monitor the instances running on provider side. We first used Shinken but we found this tool way to heavy for what we wanted to do. Then we found cAdvisor.

Google first created cAdvisor to monitor their own lmctfy containers and added Docker container support (those that use the default lib container execdriver). cAdvisor is short for Container Advisor. You can use cAdvisor to get a detailed look into the resource usage and the various performance characteristics of your running containers. cAdvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB.

If you want to monitor all of the containers you have running on a host, you can deploy the cAdvisor image as a container in that same host. It will then have access to the resource usage and performance characteristics of your other containers in that host.

You can check cAdvisor is running by pointing your browser to the host’s IP or hostname and port 8080. If running locally, try http://localhost:8080. This will bring up the built-in Web UI.

[[File:cAdvisor.png|center|thumb|500px|cAdvisor]]

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

File:CAdvisor.png

2016-03-16T09:49:35Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker

2016-03-16T09:49:18Z

Alan.Damotte: /* Containers' automatic deployment */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Monitoring with cAdvisor=

We were looking for a light and efficient tool to monitor the instances running on provider side. We first used Shinken but we found this tool way to heavy for what we wanted to do. Then we found cAdvisor.

Google first created cAdvisor to monitor their own lmctfy containers and added Docker container support (those that use the default lib container execdriver). cAdvisor is short for Container Advisor. You can use cAdvisor to get a detailed look into the resource usage and the various performance characteristics of your running containers. cAdvisor includes a simple UI to view the live data, a simple API to retrieve the data programmatically, and the ability to store the data in an external InfluxDB.

If you want to monitor all of the containers you have running on a host, you can deploy the cAdvisor image as a container in that same host. It will then have access to the resource usage and performance characteristics of your other containers in that host.

You can check cAdvisor is running by pointing your browser to the host’s IP or hostname and port 8080. If running locally, try http://localhost:8080. This will bring up the built-in Web UI.

[[File:cAdvisor.png|center|thumb|500px|cAdvisor]]

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-15T08:45:52Z

Alan.Damotte: /* Planning */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - March 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: March 7th - March 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: March 14th - March 17th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again
** Really remove client instances running on provider when he choose to not be available
** Others minor fixes

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets 2015-2016

2016-03-14T14:01:39Z

Alan.Damotte: /* Projets */

<<[[Projets 2014-2015]] | [[Projets]] | [[Projets 2016-2017]]>>
=RICM=
==RICM3==

==RICM4==
===Projet Semestre S8===

Enseignants responsables : Olivier Richard, Didier Donsez

* '''Evaluation à mi-parcours le lundi 7 mars''': Format: 10min (5min de présentation 3 slides au plus, 5min de discussion). Cette évaluation sera prise en compte dans la note finale.

'''Consignes générales:'''

* '''Vous devez être pro-actifs !!!''': Si des points sont pas ou mals spécifiés, vous le faîtes et vous justifiez vos choix. Pour les problèmes techniques éventuels vous pouvez: vous creusez la question, vous contactez l'auteur du code si il y a lieux, vous faites un rapport de bug ('''Attention:''' ca se prépare !), vous soumettez un patch, vous contactez l'enseignant ou la personne suivant le projet.

* '''Vous devez maintenir une fiche de suivi de projet''': elle doit être mise à jour chaque semaine, elle rassemble les élements essentiels du projet, elle
indique les évolutions du projet et présente sa feuille de route. '''Note:''' le nom de la fiche doit être composé du nom du projet et suffixé par ricm4_2015_2016.

* '''Vous devez utiliser un logiciel de gestion de version''' pour vos développements comme [http://en.wikipedia.org/wiki/Git_%28software%29 git ] et nous vous conseillons d'utiliser le site [https://github.com github] pour l'hébergement de votre dépôt public.

* Les document public (exemple sur github) doivent être rédigés en anglais (README, documentation, commentaires de code, nom de variables et de fonctions). Une bonnification sera accordée si le rapport et les transparents sont en anglais (la soutenance sera en francais).

{|class="wikitable alternance"
|+ Affectation des projets RICM4 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [[Dashboard pour gestionnaire de tâches et de ressources]]
| CROUZET, MATHIEU
| Richard
| [[Projets-2015-2016-DashBoard| '''Fiche''']]
| [https://github.com/MatthieuCrouzet/Projet4A '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_groupe1.pdf|Rapport Consultant]] - [[Media:Paterns.pdf|Patterns]] - [[Media:PresentationDashboard.pdf|Presentation]]
|-

!scope="row"| 2
| [[Speeding Simplified Script Language]]
| POPEK, BERTRAND-DALECHAMPS, WEI
| Richard
| [[Projets-2015-2016-SSSL| '''Fiche''']] - [[SSSL-UML| '''UML''']]
| [https://github.com/FlorianPO/Speeding-Simplified-Script-Language.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:Groupe2_AIR.pdf|Rapport Consultant]] - [[Media:PresentationProjet.pdf|Presentation]]
|-

!scope="row"| 3
| [[Borne interactive]]
| DUNAND - NAVARRO - REVEL
| Maisonnasse
| [[Projets-2015-2016-Borne-Interactive| '''Fiche''']] - [[Projets-2015-2016-Borne-Interactive-SRS | '''SRS''']] - [[Projets-2015-2016-Borne-Interactive/UML_Diagrams | '''UML''']]
| [https://github.com/Kant73/InteractiveDisplay '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:IPopo.pdf|Rapport Consultant]] - [[Media:PatternDesign.pdf | '''Design Pattern''']] - [[Media:PresentationInteractiveDisplay.pdf|Présentation Intermédiaire]]
|-

!scope="row"| 4
| [[Sonotone]]
| LECORPS, VOUTAT, Hattinguais
| Maisonnasse, Richard
| [[Projets-2015-2016-Sonotone| '''Fiche''']] - [[Projets-2015-2016-Sonotone-SRS | '''SRS''']]
| [https://github.com/Gorgorot38/Sonotone-RICM4 '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SRS_Consultant_Sonotone_4.pdf|Rapport_Consultant]] - [[Media:pattern_sonotone.pdf|Pattern]] - [[Media:Soutenance.pdf|Soutenance_miparcours]]
|-

!scope="row"| 5
| [[Sous-titre_en_temps_r%C3%A9el_d%27un_cours| Sous-titre d'un cours en temps réel]]
| LECHEVALLIER, BUI, OUNISSI
| Maisonnasse
| [[LiveSubtitles| '''Fiche''']]
| [https://github.com/Lechevallier/RealTimeSubtitles '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media: SRS_Groupe_5.pdf| Rapport Consultant]]
|-

!scope="row"| 6
| [[GrenobloisFuté]]
| MOURET, DELAPORTE, Lucidarme
| Nicolas Palix
| [[GrenobleFuté| '''Fiche''']]
| [https://github.com/Lucidarme/Osmand.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_G14.pdf|Rapport Consultant]]
|-

!scope="row"| 7
| [[Streaming en stéréoscopie]]
| ZHAO ZILONG, HAMMOUTI
| Maisonnasse
| [[Projets-2015-2016-Streaming-Stereoscopie| '''Fiche''']] - [[SRS - Streaming en stéréoscopie | '''SRS''']]
| [https://github.com/zhao-zilong/streaming_stereo '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:bruel_medewou_ndiaye.pdf|Rapport_consultant]] - [[Media:streaming.pdf|mi-parcours]]
|-

!scope="row"| 8
| [[PersyCup2016]]
| BIN, ZEGAOUI, ELLAPIN
| Donsez, Maisonnasse
| [[PersyCup| '''Fiche''']]
| [https://github.com/legominstorm/lego '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SoutenanceMiParcours-Persycup2016.pdf|Soutenance Mi-parcours]]
|-

!scope="row"| 9
| [[Services étendus pour le modèle de composants iPOPO pour Python]]
| FOUNAS, HALLAL, GATTAZ
| Calmant & Donsez
| [[Proj-2015-2016-Extensions_IPOPO | '''Fiche''']] - [[Proj-2015-2016-Extensions_IPOPO/SRS | '''SRS''']] - [[Proj-2015-2016-Extensions_IPOPO/UML | '''UML''']]
| [https://github.com/abdelazizFounas/ipopo/tree/tlsremote '''github IPOPO'''] [https://github.com/gattazr/IPOPO-Remote-Client '''github IPOPO Client''']
| [[Media:9_RapportProjet.pdf|Rapport]] - [[Media:9_TransparentsProojet.pdf|Transparents]] - [[Media:9_FlyerProjet.pdf|Flyer]] - [[Media:3-SRS-Pres.pdf| Rapport Consultant]] - [[Media:9_PatternStrat.pdf|Pattern Design]] - [[Media:9_Mid-Presentation.pdf|Mid Presentation]] - [[Media:9_Gantt.pdf|Gantt]]
|-

!scope="row"| 10
| [[IndoorGeoloc2016]]
| ARRADA - CRASTES - FAURE - STOIAN
| Donsez
| [[Proj-2015-2016-IndoorGeoloc/Fiche| '''Fiche''']] - [[Proj-2015-2016-IndoorGeoloc/SRS|SRS]]
| [https://github.com/QuentinFA/Geoloc_Indoor '''github''']
| [[Media:Proj-2015-2016-IndoorGeoloc/RapportProjet.pdf|Rapport]] - [[Media:Proj-2015-2016-IndoorGeoloc/TransparentsProjet.pdf|Transparents]] - [[Media:Proj-2015-2016-IndoorGeoloc/FlyerProjet.pdf|Flyer]] - [[Media: SRSGroupe17.pdf| Rapport Consultant]] - [[Media:Mi_parcours.pdf|Mid presentation]]
|-

!scope="row"| 11
| [[UPnPOpenHAB2016]]
| Medewou , Ndiyae Yacine , Bruel Anna
| Didier Donsez & Jérome Maisonnasse
| [[Proj-Openhab-2016| '''Fiche''']] - [[Proj-2015-2016-Int%C3%A9gration_de_cam%C3%A9ra_de_surveillance_UPnP_%C3%A0_Openhab/SRS| '''SRS''']] - [[Proj-Openhab/UML| '''UML''']]
| [https://github.com/openHab-UPnP '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_ZHAO_HAMMOUTI.pdf|Rapport Consultant]] - [[Media:pattern_ZHAO_HAMMOUTI.pdf|Patterns]] - [[Media:fichier.pdf|Mini soutenance]]
|-

!scope="row"| 12
| [[Sign2Speech]]
| NIOGRET, NOGUERON, TITH
| Didier Donsez
| [[sign2speech_ricm4_2015_2016| '''Fiche''']] - [[SRS - Sign2Speech | '''SRS''']] - [[UML | '''UML''']]
| [https://github.com/SignToSpeech-Project '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:12-Sign2Speech-RapportConsultant.pdf|Rapport Consultant]] - [[Media:12-Sign2Speech-MidPres.pdf|Mid presentation]]
|-

!scope="row"| 13
| [[AstroImage]]
| RACHEX, BLANC, GERRY
| Olivier Richard et Bruno Bzeznik
| [[Proj-2015-2016-Astroimage/Fiche| '''Fiche''']] - [[AstroImage/SRS | '''SRS''']] - [[Media:AstroImage-UML.pdf | '''UML''']]
| [https://github.com/nicolas-blanc/AstroImage '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:13-AstroImage-RapportConsultant.pdf|Rapport Consultant]] - [https://docs.google.com/presentation/d/15F8DRktwmOuSNabdxMASniyr-TIiRzGNNG1mOhcoSnk/edit?usp=sharing '''Patterns''']
|-

!scope="row"| 14
| [[Tachymètre]]
| MACE, NOUGUIER, RAMEL
| Olivier Gattaz
| [[Fiche - Tachymètre | '''Fiche''']] - [[SRS - Tachymètre| '''SRS''']] - [[UML - Tachymètre| '''UML''']]
| [https://github.com/Quego/Tachymetre '''github - Tachymètre''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:srs_tachymetre.pdf|Rapport consultant]] - [[Media:14_PatternDesign.pdf‎ | Pattern Design]] - [[Media:Tachymetre_Presentation.pdf‎ | Présentation]]
|-

!scope="row"| 15
| [[SmartProjector]]
| BRANGER, HABLOT
| Donsez, Maisonnasse
| [[Fiche_SmartProjector_ricm4_2015_2016| '''Fiche''']] - [[SRS - SmartProjector| '''SRS''']] - [[UML - SmartProjector| '''UML''']]
| [https://github.com/P0ppoff/SmartProjector '''github''']
| [[Rapport]] - [[Media:PresentationPorjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:Gl_groupe16.pdf|Rapport Consultant]] - [http://air.imag.fr/index.php/Patron_de_conception_-_SmartProjector patterns]
|-

|}

===Liste de projets===

* [[Dashboard pour gestionnaire de tâches et de ressources]], Olivier Richard
* [[Moteur distribué d'exécution de commande]], Olivier Richard
* [[Environnement d'expérimentation de pour NVIDIA Shield (Tegra X1)]], Olivier Richard
* [[Speeding Simplified Script Language]], Olivier Richard

* Aide (Open-Source)au Handicap Auditif, avec Didier Donsez, Jérome Maisonnasse, Marie-Paule Balicco (SAH UGA) et Nicolas Vuillerme
** [[Borne interactive]] (1 sujet)
** [[Sonotone]] (1 sujet)
** [[Sous-titre en temps réel d'un cours]] (1 sujet)
* [[GrenobloisFuté]] Couche trafic sur OsmAnd avec un greffon. Données dynamique de la métro. Dvp Android. Nicolas Palix.
* [[GeoDiff]] Production, visualisation, fusion de variations (diff) sur de l'information géocodée : Nicolas Palix
* [[Smart campus augmenté et contributif]] Didier Donsez, Vivien Quema

* [[Streaming en stéréoscopie]] sur [[WebRTC]] avec rendu sur [[Oculus]] pour le robot [[RobAIR]], Jérôme Maisonnasse. ([http://gstconf.ubicast.tv/videos/stereoscopic-3d-video/ voir]).
* [[STM32F7]] : Mise en oeuvre de la chaîne de compilation sous Linux avec [[OpenSTM32]] et [[OpenOCD]]. Nicolas Palix
* [[PersyCup2016]] : Persyval Robocup, Didier Donsez, Vivien Quema, Jérome Maisonnasse. (3 étudiants)
* [[Services étendus pour le modèle de composants iPOPO pour Python]], Didier Donsez & Thomas Calmant. (2 étudiants)
* [[SmartClassRoom2016|Développement d'une interface partagée pour tables tactiles (projet SmartClassRoom)]], Didier Donsez, Jérôme Maisonnasse. (2 étudiants)
* [[iRock2016|iRock : surveillance de glissement de terrains]], Didier Donsez & Vivien Quema
* [[IndoorGeoloc2016|Géolocalisation in-door au moyen de balises (beacon) BLE et Wifi à base de STM32 et de balises iBeacon & AltBeacon]], Didier Donsez & Vivien Quema
* [[UPnPOpenHAB2016|Intégration et gestion de caméras de surveillance UPnP dans la plateforme domotique open-source OpenHAB et myOpenHAB]], Didier Donsez & Jérome Maisonnasse.

'''Projets non prioritaires'''

* [[Liveprogramming with Kivy]], Olivier Richard
* [[AstroImage]] production d'image d'astronomie, Olivier Richard et Bruno Bzeznik
* [[G-code Cruncher]] Controle de machine CNC (Nucleo grbl + esp8266 + Sdcard), Olivier Richard
* [[Intégration OpenHAB / OpenTele]] Nicolas Palix

==RICM5==

===Projet Semestre S10===

Enseignant responsable : Didier Donsez

Démarrage : Lundi 25/01 à 10H30-12H30, P253 (Rendez-vous devant la salle AIR) - Visioconf pour Thibaut Cordier

Soutenance : Jeudi 17/03 à 13H00-17H00, salle P043 (Polytech Grenoble)puis en salle C005 (Batiment C)

Etudiants : RICM5 + 8 étudiants Avosti DUT RT

Rappel séances MPI
* Séance 1 : mardi 26 janvier après midi - Stéphanie Diligent
* Séance 2 : mardi 2 février après midi - Stéphanie Diligent
* Séance 3 : lundi 8 février matin - Emmanuelle Tréhoust
* Séance 4 : jeudi 11 février matin - Emmanuelle Tréhoust
* Séance 5 : lundi 21 mars matin - Stéphanie Diligent et Emmanuelle Tréhoust

=====Soutenances=====
Planning:
* Immersion EDF (13H00-13H40 en salle P043)
* Bossa (13H45-14H25 en salle P043)
* IaaS Docker (14H30-15H10 en salle P043)
* SmartCampus (15H15-15H55 en salle P043 et salle P259 AIR)
* SmartClassRoom (16H15-16H55 en C005)
* Pot d' "Au Revoir" (17H00-1800 en C005)

Instructions:
*Chaque soutenance comporte 15 minutes de présentation, 15 minutes de démonstration et 10 minutes de questions. Un transparent doit être consacré au travail confié et réalisé par les étudiants en DUT (AVOSTI).
* Répétez plusieurs fois votre présentation et votre démonstration.
* L'ensemble des documents (y compris photos, vidéos et ''[[Logiciels#Screencast|screencast]]s'') doivent être accessibles depuis le tableau ci-dessous et dans chaque fiche de suivi. Prévoyez une copie sur clé USB.
* '''TOUT Le matériel prêté devra être rapporté et restitué dans un sac cabas lors de la soutenance.'''

=====Projets=====
{|class="wikitable alternance"
|+ Affectation des projets RICM5 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
!scope="col"| Documents
|-

!scope="row"| 1
| [http://air.imag.fr/index.php/IaaS_collaboratif_avec_Docker IaaS - Docker]
| Eudes Robin, Damotte Alan, Barthelemy Romain, Mammar Malek, Guo Kai, Bonnard Loïc, Caperan Théo
| Didier Donsez
| [[Projets-2015-2016-IaaS_Docker| '''Fiche''']] - [[Projets-2015-2016-IaaS_Docker-SRS| '''SRS''']]
| [https://github.com/EudesRobin/iaas-collaboratif '''github''']
| [[Media:Rapport_IaaS.pdf|Rapport]] - [[Media:Transparents_IaaS.pdf|Transparents]] - [[Media:Flyer_IaaS.pdf|Flyer]]
|-
!scope="row"| 2
| [http://air.imag.fr/index.php/Portage_de_Bossa Portage de Bossa sur le Kernel Linux 4x]
| Eric Michel Fotsing, Ombeline Rossi, Longfei Yao
| Nicolas Palix, Didier Donsez
| [[Projets-2015-2016-Portage_Bossa| '''Fiche''']]
| [https://github.com/ZenithKaizer/ '''github''']
| [[Media:Rapport_Bossa.pdf|Rapport]] - [[Media:Transparents_Bossa.pdf|Transparents]] - [[Media:Flyer_Bossa.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 3
| [[Visite immersive en réalité virtuelle dans une usine avec EDF]]
| Adam Christophe, Aissanou Sarah, Klipffel Tararaina, Qian Jean, Zominy Laurent
| Didier Donsez, Georges-Pierre Bonneau, Thibaut Cordier (EDF)
| [[Projets-2015-2016-VisiteImmersiveEDF| '''Fiche''']]
| [https://github.com/VisiteImmersiveEDF '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 4
| [[Contribution à OpenSmartCampus]] (voir http://data.beta.metropolegrenoble.fr/)
| Quentin Torck, Vivien Michel, Jérémy Hammerer, Rama Codazzi, Zhengmeng Zhang
| Didier Donsez, Vivien Quéma
| [[Projets-2015-2016-OpenSmartCampus| '''Fiche''']]
| [https://github.com/quentin74/SmartCampus.git '''github''']
| [[Media:RapportProjetOpenSmartCampus2016.pdf|Rapport]] - [[Media:TransparentsProojetOpenSmartCampus2016.pdf|Transparents]] - [[Media:FlyerProjetOpenSmartCampus2016.pdf|Flyer]] - Photos - Vidéos
|-

!scope="row"| 5
| [[Contribution à SmartClassRoom]] (Interfaces tactiles distribuées et partagées)
| Saussac Thibault, Toussaint Sébastien, Hamdani Youcef, Zoppello Sebastien, Melik sak, Mesnier Vincent
| Jérôme Maisonnasse, Didier Donsez
| [[Projets-2015-2016-SmartClassRoom| '''Fiche''']]
| [https://github.com/XXXX '''github''']
| [[Media:RapportProjetSmartClassRoom.pdf|Rapport]] - [[Media:TransparentsProjetSmartClassRoom.pdf|Transparents]] - [[Media:FlyerProjetSmartClassRoom.pdf|Flyer]] - Photos - Vidéos
|-

|}

===Projets annulés et reportés===
* Projet avec [[Tango Project]] (Annulé)
* Hack the Beam, Didier Donsez & Jérôme Maisonnasse.
* [[Algorithmes de suivi de personnes pour robot de téléprésence RobAIR]] (Jérôme Maisonnasse, Didier Donsez)

=M2PGI=
==[[Projets M2PGI Services Machine-to-Machine|Projet Services Machine-to-Machine]]==
* [[PM2M/2016/TP|Sujet et groupes]]

Projets-2015-2016-IaaS Docker

2016-03-14T14:00:54Z

Alan.Damotte: /* System Architecture */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

== Database schema ==

[[File:Database.jpg|center|thumb|1000px|Database diagram]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-14T13:43:41Z

Alan.Damotte: /* Week 8: Mars 14th - Mars 18th */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===
* Finish presentation preparation
* Fix some bugs:
** Start instances if they already exist
** Stop instances and no longer remove them in order to be reused/started again

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use too much space disk of the provider. To do so, we implemented a watchdog that checks every 30 seconds the disk usage of each container and stops them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper and to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-14T09:51:09Z

Alan.Damotte: /* Containers' automatic deployment */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|500px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-14T09:50:53Z

Alan.Damotte: /* Containers' automatic deployment */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|600px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allows us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes its port 22000 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allows us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set a memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

File:Provider functioning.png

2016-03-14T09:50:20Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Provider functioning.png"

File:Provider functioning.png

2016-03-14T09:49:10Z

Alan.Damotte: Alan.Damotte uploaded a new version of "File:Provider functioning.png"

Projets-2015-2016-IaaS Docker

2016-03-14T09:47:14Z

Alan.Damotte: /* Build and run */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task needs to be done from the host. That's why the first step is to create a new user on provider machine that we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (cAdvisor) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker

2016-03-14T08:57:47Z

Alan.Damotte: /* Week 7: Mars 7th - Mars 13th */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decides to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Specification of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which makes it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatizes user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish a connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implement bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployment:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contain only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now its behaviour allows us to have different disk usage for each instance. Now we use a cron job, we won't need anymore to launch the script by ourselves
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough informations about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually develop some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Write report for our last MPI course
* End of Rabbitmq set up on front-end
* Test complete loop:
** Create profile
** Set required information (ssh public key)
** As a provider, give settings of the provided machine
** As a client, ask for an instance
** As a client connect to the instance (ssh)
** Check that Rabbitmq is correctly tracing back information about containers/instances
* Add a rating system which will be used to give a mark to providers.
* Start preparing the presentation

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Global Architecture ==
[[File:General_schema_IaaS.png|center|thumb|1000px|Global architecture]]

== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Provider and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients and their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]
* [https://github.com/meteorhacks/npm meteorhacks:npm Installation Instructions]

== RabbitMQ related ==
* [https://www.npmjs.com/package/amqplib amqlib , the library used to publish]
* [https://www.rabbitmq.com/getstarted.html RabbitMQ Tutorials]

Projets-2015-2016-IaaS Docker-SRS

2016-03-11T10:22:42Z

Alan.Damotte:

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| February 2016
| Romain Barthelemy, Alan Damotte, Robin Eudes, Kai Guo, Malek Mammar
| Collaborative IaaS using Docker
| Pierre-Yves Gibello
| February 2016

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:* The system must allow client to ask for an instance precising required resources
:* The system must allow client to launch an instance
:* The system must allow client to connect to running instance
:* The system must allow client to stop an instance
:* The system must allow client to remove an instance
:* The system should allow client to give a mark to a provider

:* The system must allow client to download required files
:* The system must allow client to launch coordinator system
:* The system must allow client to provide some of his resources
:* The system must allow client to start providing
:* The system must allow client to stop providing
:* The system should allow client to see resources consumption of the instances running on his machine

==Performance requirements==

==Design constraints==

==Logical database requirement==
[[File:Database_diagram.png|center|thumb|1000px|Database diagram]]

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
* Clients could only connect to their instances
* Clients could know status of their instances
* Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-11T10:22:06Z

Alan.Damotte:

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| March 2016
| Romain Barthelemy, Alan Damotte, Robin Eudes, Kai Guo, Malek Mammar
| Collaborative IaaS using Docker
| Didier Donsez
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:* The system must allow client to ask for an instance precising required resources
:* The system must allow client to launch an instance
:* The system must allow client to connect to running instance
:* The system must allow client to stop an instance
:* The system must allow client to remove an instance
:* The system should allow client to give a mark to a provider

:* The system must allow client to download required files
:* The system must allow client to launch coordinator system
:* The system must allow client to provide some of his resources
:* The system must allow client to start providing
:* The system must allow client to stop providing
:* The system should allow client to see resources consumption of the instances running on his machine

==Performance requirements==

==Design constraints==

==Logical database requirement==
[[File:Database_diagram.png|center|thumb|1000px|Database diagram]]

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
* Clients could only connect to their instances
* Clients could know status of their instances
* Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker

2016-03-09T15:16:01Z

Alan.Damotte: /* The team */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker-SRS

2016-03-09T12:57:16Z

Alan.Damotte: /* Reliability */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:* The system must allow client to ask for an instance precising required resources
:* The system must allow client to launch an instance
:* The system must allow client to connect to running instance
:* The system must allow client to stop an instance
:* The system must allow client to remove an instance
:* The system should allow client to give a mark to a provider

:* The system must allow client to download required files
:* The system must allow client to launch coordinator system
:* The system must allow client to provide some of his resources
:* The system must allow client to start providing
:* The system must allow client to stop providing
:* The system should allow client to see resources consumption of the instances running on his machine

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
* Clients could only connect to their instances
* Clients could know status of their instances
* Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-09T12:56:35Z

Alan.Damotte: /* Functional requirements */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:* The system must allow client to ask for an instance precising required resources
:* The system must allow client to launch an instance
:* The system must allow client to connect to running instance
:* The system must allow client to stop an instance
:* The system must allow client to remove an instance
:* The system should allow client to give a mark to a provider

:* The system must allow client to download required files
:* The system must allow client to launch coordinator system
:* The system must allow client to provide some of his resources
:* The system must allow client to start providing
:* The system must allow client to stop providing
:* The system should allow client to see resources consumption of the instances running on his machine

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
** Clients could only connect to their instances
** Clients could know status of their instances
** Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-09T12:50:50Z

Alan.Damotte: /* Functional requirements */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile

:* The sk for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
** Clients could only connect to their instances
** Clients could know status of their instances
** Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-09T12:49:31Z

Alan.Damotte: /* General constraints */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)
:** The provider must have Docker installed

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:*

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
** Clients could only connect to their instances
** Clients could know status of their instances
** Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-09T12:47:10Z

Alan.Damotte: /* Specific requirements, covering functional, non-functional and interface requirements */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==Functional requirements==
:* The system must allow user to create their profile
:*

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

The system must deliver correct informations all the time, so that :
** Clients could only connect to their instances
** Clients could know status of their instances
** Providers could know status of their machine

===Availability===

The system must be available 24h/24, 7days/7 since both providers and clients should be able to use it.

===Security===

The security of the service must be optimal, clients should not be able to access information on instances of other clients. Furthermore, providers should also not be able to access container's information.

===Maintainability===

Updates have to be easy to do, in order to add new functionalities, improve the service easily.

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

File:RapportMPI Iaas.pdf

2016-03-09T09:47:11Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker

2016-03-09T09:46:50Z

Alan.Damotte: /* Deliverables */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

[[Media:RapportMPI_Iaas.pdf|Management of innovative projects (MPI) report (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T10:18:04Z

Alan.Damotte: /* Product functions */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start providing
:** Stop providing
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==External interface requirements==

==Functional requirements==

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

===Availability===

===Security===

===Maintainability===

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T10:14:40Z

Alan.Damotte: /* Product functions */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :
:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :
:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start to provide
:** Stop to provide
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==External interface requirements==

==Functional requirements==

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

===Availability===

===Security===

===Maintainability===

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T10:12:54Z

Alan.Damotte: /* Scope of the product */

The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard.

'''Read first:'''
* http://www.cs.st-andrews.ac.uk/~ifs/Books/SE9/Presentations/PPTX/Ch4.pptx
* http://en.wikipedia.org/wiki/Software_requirements_specification
* [http://www.cse.msu.edu/~chengb/RE-491/Papers/IEEE-SRS-practice.pdf IEEE Recommended Practice for Software Requirements Specifications IEEE Std 830-1998]

{|class="wikitable alternance"
|+ Document History
|-
|
!scope="col"| Version
!scope="col"| Date
!scope="col"| Authors
!scope="col"| Description
!scope="col"| Validator
!scope="col"| Validation Date
|-
!scope="row" |
| 0.1.0
| TBC
| TBC
| TBC
| TBC
| TBC

|}
=Introduction=

==Purpose of the requirements document==

:*This Software Requirements Specification (SRS) identifies the requirements for the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project .
:*This document is a guideline about the functionalities offered and the problems that the system solves.

==Scope of the product==
The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

==Definitions, acronyms and abbreviations==

==References==

:*The main page of the project: [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]]<br\>

==Overview of the remainder of the document==

:The rest of the SRS examines the specifications of the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project in details. Section two of the SRS presents the general factors that affect the [[Projets-2015-2016-IaaS Docker | Collaborative Iaas]] project and its requirements, such as user characteristics and project constraints. Section three outlines the detailed, specific and functional requirements, performance, system and other related requirements of the project. Supporting information about appendices is provided in Section three.

=General description=

==Product perspective==
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

==Product functions==

:*Client :

:** Register to the service
:** Ask for an instance precising required resources
:** Launch an instance
:** Connect to running instance
:** Stop an instance
:** Remove an instance
:** Give a mark to a provider

:*Provider :

:** Register to the service
:** Download required files
:** Launch coordinator system
:** Provide some of his resources
:** Start to provide
:** Stop to provide
:** See resources consumption of the instances running on his machine

== User characteristics==
:* The client/provider doesn't have to be familiar with programming
:* The client/provider should know unix basics
:* The client/provider should know how ssh works
:* The provider should know how to launch a script in a terminal

==General constraints==
:*System constraint:
:** The provider's machine must run on a unix system (Ubuntu for example)

:*Environment constraint:
:** Internet access is required to use the service

==Assumptions and dependencies==
:* The client/provider have an internet access

=Specific requirements, covering functional, non-functional and interface requirements=

==External interface requirements==

==Functional requirements==

==Performance requirements==

==Design constraints==

==Logical database requirement==

==Software System attributes==

===Reliability===

===Availability===

===Security===

===Maintainability===

===Portability===
:For the moment, the system will be available in Linux only for provider side
:However, if packages are available on other systems, we might release the system on other OS later.

==Other requirements==
:*The system must be able to run on Linux 14 or higher
:*The system must not consume too much CPU
:*The system must not consume too much Memory

=Product evolution=

=Appendices=

==Specification ==
* The global project's page can be found [[Projets-2015-2016-IaaS Docker | here]].

==Licensing Requirements==

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T10:12:08Z

Alan.Damotte: /* General description */

File:IaasUseCase.png

2016-03-08T09:54:34Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker

2016-03-08T09:54:23Z

Alan.Damotte: /* Product perspective */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

[[File:IaasUseCase.png|center|thumb|1000px|Use case diagram]]

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

File:IaasContextDiagram.png

2016-03-08T09:45:47Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker

2016-03-08T09:45:35Z

Alan.Damotte: /* Product perspective */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =
[[File:IaasContextDiagram.png|center|thumb|1000px|Context diagram]]

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker

2016-03-08T09:44:55Z

Alan.Damotte: /* Ship More Software Faster */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= Product perspective =

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T09:39:35Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker-SRS

2016-03-08T09:38:30Z

Alan.Damotte: Created page with "The document provides a template of the Software Requirements Specification (SRS). It is inspired of the IEEE/ANSI 830-1998 Standard. '''Read first:''' * http://www.cs.st-an..."

Projets 2015-2016

2016-03-08T09:31:37Z

Alan.Damotte: /* Projet Semestre S10 */

<<[[Projets 2014-2015]] | [[Projets]] | [[Projets 2016-2017]]>>
=RICM=
==RICM3==

==RICM4==
===Projet Semestre S8===

Enseignants responsables : Olivier Richard, Didier Donsez

* '''Evaluation à mi-parcours le lundi 7 mars''': Format: 10min (5min de présentation 3 slides au plus, 5min de discussion). Cette évaluation sera prise en compte dans la note finale.

'''Consignes générales:'''

* '''Vous devez être pro-actifs !!!''': Si des points sont pas ou mals spécifiés, vous le faîtes et vous justifiez vos choix. Pour les problèmes techniques éventuels vous pouvez: vous creusez la question, vous contactez l'auteur du code si il y a lieux, vous faites un rapport de bug ('''Attention:''' ca se prépare !), vous soumettez un patch, vous contactez l'enseignant ou la personne suivant le projet.

* '''Vous devez maintenir une fiche de suivi de projet''': elle doit être mise à jour chaque semaine, elle rassemble les élements essentiels du projet, elle
indique les évolutions du projet et présente sa feuille de route. '''Note:''' le nom de la fiche doit être composé du nom du projet et suffixé par ricm4_2015_2016.

* '''Vous devez utiliser un logiciel de gestion de version''' pour vos développements comme [http://en.wikipedia.org/wiki/Git_%28software%29 git ] et nous vous conseillons d'utiliser le site [https://github.com github] pour l'hébergement de votre dépôt public.

* Les document public (exemple sur github) doivent être rédigés en anglais (README, documentation, commentaires de code, nom de variables et de fonctions). Une bonnification sera accordée si le rapport et les transparents sont en anglais (la soutenance sera en francais).

{|class="wikitable alternance"
|+ Affectation des projets RICM4 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [[Dashboard pour gestionnaire de tâches et de ressources]]
| CROUZET, MATHIEU
| Richard
| [[Projets-2015-2016-DashBoard| '''Fiche''']]
| [https://github.com/MatthieuCrouzet/Projet4A '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_groupe1.pdf|Rapport Consultant]] - [[Media:Paterns.pdf|Patterns]] - [[Media:PresentationDashboard.pdf|Presentation]]
|-

!scope="row"| 2
| [[Speeding Simplified Script Language]]
| POPEK, BERTRAND-DALECHAMPS, WEI
| Richard
| [[Projets-2015-2016-SSSL| '''Fiche''']] - [[SSSL-UML| '''UML''']]
| [https://github.com/FlorianPO/Speeding-Simplified-Script-Language.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:Groupe2_AIR.pdf|Rapport Consultant]] - [[Media:PresentationProjet.pdf|Presentation]]
|-

!scope="row"| 3
| [[Borne interactive]]
| DUNAND - NAVARRO - REVEL
| Maisonnasse
| [[Projets-2015-2016-Borne-Interactive| '''Fiche''']] - [[Projets-2015-2016-Borne-Interactive-SRS | '''SRS''']] - [[Projets-2015-2016-Borne-Interactive/UML_Diagrams | '''UML''']]
| [https://github.com/Kant73/InteractiveDisplay '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:IPopo.pdf|Rapport Consultant]] - [[Media:PatternDesign.pdf | '''Design Pattern''']] - [[Media:PresentationInteractiveDisplay.pdf|Présentation Intermédiaire]]
|-

!scope="row"| 4
| [[Sonotone]]
| LECORPS, VOUTAT, Hattinguais
| Maisonnasse, Richard
| [[Projets-2015-2016-Sonotone| '''Fiche''']] - [[Projets-2015-2016-Sonotone-SRS | '''SRS''']]
| [https://github.com/Gorgorot38/Sonotone-RICM4 '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SRS_Consultant_Sonotone_4.pdf|Rapport_Consultant]] - [[Media:pattern_sonotone.pdf|Pattern]] - [[Media:Soutenance.pdf|Soutenance]]
|-

!scope="row"| 5
| [[Sous-titre_en_temps_r%C3%A9el_d%27un_cours| Sous-titre d'un cours en temps réel]]
| LECHEVALLIER, BUI, OUNISSI
| Maisonnasse
| [[LiveSubtitles| '''Fiche''']]
| [https://github.com/Lechevallier/RealTimeSubtitles '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media: SRS_Groupe_5.pdf| Rapport Consultant]]
|-

!scope="row"| 6
| [[GrenobloisFuté]]
| MOURET, DELAPORTE, Lucidarme
| Nicolas Palix
| [[GrenobleFuté| '''Fiche''']]
| [https://github.com/Lucidarme/Osmand.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_G14.pdf|Rapport Consultant]]
|-

!scope="row"| 7
| [[Streaming en stéréoscopie]]
| ZHAO ZILONG, HAMMOUTI
| Maisonnasse
| [[Projets-2015-2016-Streaming-Stereoscopie| '''Fiche''']] - [[SRS - Streaming en stéréoscopie | '''SRS''']]
| [https://github.com/zhao-zilong/streaming_stereo '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:bruel_medewou_ndiaye.pdf|Rapport_consultant]] - [[Media:streaming.pdf|mi-parcours]]
|-

!scope="row"| 8
| [[PersyCup2016]]
| BIN, ZEGAOUI, ELLAPIN
| Donsez, Maisonnasse
| [[PersyCup| '''Fiche''']]
| [https://github.com/legominstorm/lego '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SoutenanceMiParcours-Persycup2016.pdf|Soutenance Mi-parcours]]
|-

!scope="row"| 9
| [[Services étendus pour le modèle de composants iPOPO pour Python]]
| FOUNAS, HALLAL, GATTAZ
| Calmant & Donsez
| [[Proj-2015-2016-Extensions_IPOPO | '''Fiche''']] - [[Proj-2015-2016-Extensions_IPOPO/SRS | '''SRS''']] - [[Proj-2015-2016-Extensions_IPOPO/UML | '''UML''']]
| [https://github.com/abdelazizFounas/ipopo/tree/tlsremote '''github IPOPO'''] [https://github.com/gattazr/IPOPO-Remote-Client '''github IPOPO Client''']
| [[Media:9_RapportProjet.pdf|Rapport]] - [[Media:9_TransparentsProojet.pdf|Transparents]] - [[Media:9_FlyerProjet.pdf|Flyer]] - [[Media:3-SRS-Pres.pdf| Rapport Consultant]] - [[Media:9_PatternStrat.pdf|Pattern Design]] - [[Media:9_Mid-Presentation.pdf|Mid Presentation]] - [[Media:9_Gantt.pdf|Gantt]]
|-

!scope="row"| 10
| [[IndoorGeoloc2016]]
| ARRADA - CRASTES - FAURE - STOIAN
| Donsez
| [[Proj-2015-2016-IndoorGeoloc/Fiche| '''Fiche''']] - [[Proj-2015-2016-IndoorGeoloc/SRS|SRS]]
| [https://github.com/xxx '''github''']
| [[Media:Proj-2015-2016-IndoorGeoloc/RapportProjet.pdf|Rapport]] - [[Media:Proj-2015-2016-IndoorGeoloc/TransparentsProjet.pdf|Transparents]] - [[Media:Proj-2015-2016-IndoorGeoloc/FlyerProjet.pdf|Flyer]] - [[Media: SRSGroupe17.pdf| Rapport Consultant]]
|-

!scope="row"| 11
| [[UPnPOpenHAB2016]]
| Medewou , Ndiyae Yacine , Bruel Anna
| Didier Donsez & Jérome Maisonnasse
| [[Proj-Openhab-2016| '''Fiche''']] - [[Proj-2015-2016-Int%C3%A9gration_de_cam%C3%A9ra_de_surveillance_UPnP_%C3%A0_Openhab/SRS| '''SRS''']] - [[Proj-Openhab/UML| '''UML''']]
| [https://github.com/openHab-UPnP '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_ZHAO_HAMMOUTI.pdf|Rapport Consultant]] - [[Media:pattern_ZHAO_HAMMOUTI.pdf|Patterns]] - [[Media:fichier.pdf|Mini soutenance]]
|-

!scope="row"| 12
| [[Sign2Speech]]
| NIOGRET, NOGUERON, TITH
| Didier Donsez
| [[sign2speech_ricm4_2015_2016| '''Fiche''']] - [[SRS - Sign2Speech | '''SRS''']] - [[UML | '''UML''']]
| [https://github.com/SignToSpeech-Project '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:12-Sign2Speech-RapportConsultant.pdf|Rapport Consultant]] - [[Media:12-Sign2Speech-MidPres.pdf|Mid presentation]]
|-

!scope="row"| 13
| [[AstroImage]]
| RACHEX, BLANC, GERRY
| Olivier Richard et Bruno Bzeznik
| [[Proj-2015-2016-Astroimage/Fiche| '''Fiche''']] - [[AstroImage/SRS | '''SRS''']] - [[Media:AstroImage-UML.pdf | '''UML''']]
| [https://github.com/nicolas-blanc/AstroImage '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:13-AstroImage-RapportConsultant.pdf|Rapport Consultant]] - [https://docs.google.com/presentation/d/15F8DRktwmOuSNabdxMASniyr-TIiRzGNNG1mOhcoSnk/edit?usp=sharing '''Patterns''']
|-

!scope="row"| 14
| [[Tachymètre]]
| MACE, NOUGUIER, RAMEL
| Olivier Gattaz
| [[Fiche - Tachymètre | '''Fiche''']] - [[SRS - Tachymètre| '''SRS''']] - [[UML - Tachymètre| '''UML''']]
| [https://github.com/Quego/Tachymetre '''github - Tachymètre''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:srs_tachymetre.pdf|Rapport consultant]] - [[Media:14_PatternDesign.pdf‎ | Pattern Design]] - [[Media:Tachymetre_Presentation.pdf‎ | Présentation]]
|-

!scope="row"| 15
| [[SmartProjector]]
| BRANGER, HABLOT
| Donsez, Maisonnasse
| [[Fiche_SmartProjector_ricm4_2015_2016| '''Fiche''']] - [[SRS - SmartProjector| '''SRS''']] - [[UML - SmartProjector| '''UML''']]
| [https://github.com/P0ppoff/SmartProjector '''github''']
| [[Rapport]] - [[Media:PresentationPorjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:Gl_groupe16.pdf|Rapport Consultant]] - [http://air.imag.fr/index.php/Patron_de_conception_-_SmartProjector patterns]
|-

|}

===Liste de projets===

* [[Dashboard pour gestionnaire de tâches et de ressources]], Olivier Richard
* [[Moteur distribué d'exécution de commande]], Olivier Richard
* [[Environnement d'expérimentation de pour NVIDIA Shield (Tegra X1)]], Olivier Richard
* [[Speeding Simplified Script Language]], Olivier Richard

* Aide (Open-Source)au Handicap Auditif, avec Didier Donsez, Jérome Maisonnasse, Marie-Paule Balicco (SAH UGA) et Nicolas Vuillerme
** [[Borne interactive]] (1 sujet)
** [[Sonotone]] (1 sujet)
** [[Sous-titre en temps réel d'un cours]] (1 sujet)
* [[GrenobloisFuté]] Couche trafic sur OsmAnd avec un greffon. Données dynamique de la métro. Dvp Android. Nicolas Palix.
* [[GeoDiff]] Production, visualisation, fusion de variations (diff) sur de l'information géocodée : Nicolas Palix
* [[Smart campus augmenté et contributif]] Didier Donsez, Vivien Quema

* [[Streaming en stéréoscopie]] sur [[WebRTC]] avec rendu sur [[Oculus]] pour le robot [[RobAIR]], Jérôme Maisonnasse. ([http://gstconf.ubicast.tv/videos/stereoscopic-3d-video/ voir]).
* [[STM32F7]] : Mise en oeuvre de la chaîne de compilation sous Linux avec [[OpenSTM32]] et [[OpenOCD]]. Nicolas Palix
* [[PersyCup2016]] : Persyval Robocup, Didier Donsez, Vivien Quema, Jérome Maisonnasse. (3 étudiants)
* [[Services étendus pour le modèle de composants iPOPO pour Python]], Didier Donsez & Thomas Calmant. (2 étudiants)
* [[SmartClassRoom2016|Développement d'une interface partagée pour tables tactiles (projet SmartClassRoom)]], Didier Donsez, Jérôme Maisonnasse. (2 étudiants)
* [[iRock2016|iRock : surveillance de glissement de terrains]], Didier Donsez & Vivien Quema
* [[IndoorGeoloc2016|Géolocalisation in-door au moyen de balises (beacon) BLE et Wifi à base de STM32 et de balises iBeacon & AltBeacon]], Didier Donsez & Vivien Quema
* [[UPnPOpenHAB2016|Intégration et gestion de caméras de surveillance UPnP dans la plateforme domotique open-source OpenHAB et myOpenHAB]], Didier Donsez & Jérome Maisonnasse.

'''Projets non prioritaires'''

* [[Liveprogramming with Kivy]], Olivier Richard
* [[AstroImage]] production d'image d'astronomie, Olivier Richard et Bruno Bzeznik
* [[G-code Cruncher]] Controle de machine CNC (Nucleo grbl + esp8266 + Sdcard), Olivier Richard
* [[Intégration OpenHAB / OpenTele]] Nicolas Palix

==RICM5==

===Projet Semestre S10===

Enseignant responsable : Didier Donsez

Démarrage : Lundi 25/01 à 10H30-12H30, P253 (Rendez-vous devant la salle AIR) - Visioconf pour Thibaut Cordier

Soutenance : Vendredi 18/03 à 8H30-12H30, P257

Etudiants : RICM5 + 8 étudiants Avosti DUT RT

Rappel séances MPI
* Séance 1 : mardi 26 janvier après midi - Stéphanie Diligent
* Séance 2 : mardi 2 février après midi - Stéphanie Diligent
* Séance 3 : lundi 8 février matin - Emmanuelle Tréhoust
* Séance 4 : jeudi 11 février matin - Emmanuelle Tréhoust
* Séance 5 : lundi 21 mars matin - Stéphanie Diligent et Emmanuelle Tréhoust

Planning soutenance (à venir).
* Bossa
* IaaS Docker
* Immersion EDF
* SmartCampus
* SmartClassRoom (en C005)
* Pot d' "Au Revoir"

{|class="wikitable alternance"
|+ Affectation des projets RICM5 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [http://air.imag.fr/index.php/IaaS_collaboratif_avec_Docker IaaS - Docker]
| Eudes Robin, Damotte Alan, Barthelemy Romain, Mammar Malek, Guo Kai, Bonnard Loïc, Caperan Théo
| Didier Donsez
| [[Projets-2015-2016-IaaS_Docker| '''Fiche''']] - [[Projets-2015-2016-IaaS_Docker-SRS| '''SRS''']]
| [https://github.com/EudesRobin/iaas-collaboratif '''github''']
| [[Media:Rapport_IaaS.pdf|Rapport]] - [[Media:Transparents_IaaS.pdf|Transparents]] - [[Media:Flyer_IaaS.pdf|Flyer]]
|-
!scope="row"| 2
| [http://air.imag.fr/index.php/Portage_de_Bossa Portage de Bossa sur le Kernel Linux 4x]
| Eric Michel Fotsing, Ombeline Rossi, Longfei Yao
| Nicolas Palix, Didier Donsez
| [[Projets-2015-2016-Portage_Bossa| '''Fiche''']]
| [https://github.com/ZenithKaizer/ '''github''']
| [[Media:Rapport_Bossa.pdf|Rapport]] - [[Media:Transparents_Bossa.pdf|Transparents]] - [[Media:Flyer_Bossa.pdf|Flyer]]
|-

!scope="row"| 3
| [[Visite immersive en réalité virtuelle dans une usine avec EDF]]
| Adam Christophe, Aissanou Sarah, Klipffel Tararaina, Qian Jean, Zominy Laurent
| Didier Donsez, Georges-Pierre Bonneau, Thibaut Cordier (EDF)
| [[Projets-2015-2016-VisiteImmersiveEDF| '''Fiche''']]
| [https://github.com/VisiteImmersiveEDF '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]]
|-

!scope="row"| 4
| [[Contribution à OpenSmartCampus]] (voir http://data.beta.metropolegrenoble.fr/)
| Quentin Torck, Vivien Michel, Jérémy Hammerer, Rama Codazzi, Zhengmeng Zhang
| Didier Donsez, Vivien Quéma
| [[Projets-2015-2016-OpenSmartCampus| '''Fiche''']]
| [https://github.com/quentin74/SmartCampus.git '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]]
|-

!scope="row"| 5
| [[Contribution à SmartClassRoom]] (Interfaces tactiles distribuées et partagées)
| Saussac Thibault, Toussaint Sébastien, Hamdani Youcef, Zoppello Sebastien, Melik sak, Mesnier Vincent
| Jérôme Maisonnasse, Didier Donsez
| [[Projets-2015-2016-SmartClassRoom| '''Fiche''']]
| [https://github.com/XXXX '''github''']
| [[Media:RapportProjetSmartClassRoom.pdf|Rapport]] - [[Media:TransparentsProjetSmartClassRoom.pdf|Transparents]] - [[Media:FlyerProjetSmartClassRoom.pdf|Flyer]]
|-

|}

===Projets annulés et reportés===
* Projet avec [[Tango Project]] (Annulé)
* Hack the Beam, Didier Donsez & Jérôme Maisonnasse.
* [[Algorithmes de suivi de personnes pour robot de téléprésence RobAIR]] (Jérôme Maisonnasse, Didier Donsez)

=M2PGI=
==[[Projets M2PGI Services Machine-to-Machine|Projet Services Machine-to-Machine]]==
* [[PM2M/2016/TP|Sujet et groupes]]

Projets 2015-2016

2016-03-08T09:31:13Z

Alan.Damotte: /* Projet Semestre S10 */

<<[[Projets 2014-2015]] | [[Projets]] | [[Projets 2016-2017]]>>
=RICM=
==RICM3==

==RICM4==
===Projet Semestre S8===

Enseignants responsables : Olivier Richard, Didier Donsez

* '''Evaluation à mi-parcours le lundi 7 mars''': Format: 10min (5min de présentation 3 slides au plus, 5min de discussion). Cette évaluation sera prise en compte dans la note finale.

'''Consignes générales:'''

* '''Vous devez être pro-actifs !!!''': Si des points sont pas ou mals spécifiés, vous le faîtes et vous justifiez vos choix. Pour les problèmes techniques éventuels vous pouvez: vous creusez la question, vous contactez l'auteur du code si il y a lieux, vous faites un rapport de bug ('''Attention:''' ca se prépare !), vous soumettez un patch, vous contactez l'enseignant ou la personne suivant le projet.

* '''Vous devez maintenir une fiche de suivi de projet''': elle doit être mise à jour chaque semaine, elle rassemble les élements essentiels du projet, elle
indique les évolutions du projet et présente sa feuille de route. '''Note:''' le nom de la fiche doit être composé du nom du projet et suffixé par ricm4_2015_2016.

* '''Vous devez utiliser un logiciel de gestion de version''' pour vos développements comme [http://en.wikipedia.org/wiki/Git_%28software%29 git ] et nous vous conseillons d'utiliser le site [https://github.com github] pour l'hébergement de votre dépôt public.

* Les document public (exemple sur github) doivent être rédigés en anglais (README, documentation, commentaires de code, nom de variables et de fonctions). Une bonnification sera accordée si le rapport et les transparents sont en anglais (la soutenance sera en francais).

{|class="wikitable alternance"
|+ Affectation des projets RICM4 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [[Dashboard pour gestionnaire de tâches et de ressources]]
| CROUZET, MATHIEU
| Richard
| [[Projets-2015-2016-DashBoard| '''Fiche''']]
| [https://github.com/MatthieuCrouzet/Projet4A '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_groupe1.pdf|Rapport Consultant]] - [[Media:Paterns.pdf|Patterns]] - [[Media:PresentationDashboard.pdf|Presentation]]
|-

!scope="row"| 2
| [[Speeding Simplified Script Language]]
| POPEK, BERTRAND-DALECHAMPS, WEI
| Richard
| [[Projets-2015-2016-SSSL| '''Fiche''']] - [[SSSL-UML| '''UML''']]
| [https://github.com/FlorianPO/Speeding-Simplified-Script-Language.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:Groupe2_AIR.pdf|Rapport Consultant]] - [[Media:PresentationProjet.pdf|Presentation]]
|-

!scope="row"| 3
| [[Borne interactive]]
| DUNAND - NAVARRO - REVEL
| Maisonnasse
| [[Projets-2015-2016-Borne-Interactive| '''Fiche''']] - [[Projets-2015-2016-Borne-Interactive-SRS | '''SRS''']] - [[Projets-2015-2016-Borne-Interactive/UML_Diagrams | '''UML''']]
| [https://github.com/Kant73/InteractiveDisplay '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:IPopo.pdf|Rapport Consultant]] - [[Media:PatternDesign.pdf | '''Design Pattern''']] - [[Media:PresentationInteractiveDisplay.pdf|Présentation Intermédiaire]]
|-

!scope="row"| 4
| [[Sonotone]]
| LECORPS, VOUTAT, Hattinguais
| Maisonnasse, Richard
| [[Projets-2015-2016-Sonotone| '''Fiche''']] - [[Projets-2015-2016-Sonotone-SRS | '''SRS''']]
| [https://github.com/Gorgorot38/Sonotone-RICM4 '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SRS_Consultant_Sonotone_4.pdf|Rapport_Consultant]] - [[Media:pattern_sonotone.pdf|Pattern]] - [[Media:Soutenance.pdf|Soutenance]]
|-

!scope="row"| 5
| [[Sous-titre_en_temps_r%C3%A9el_d%27un_cours| Sous-titre d'un cours en temps réel]]
| LECHEVALLIER, BUI, OUNISSI
| Maisonnasse
| [[LiveSubtitles| '''Fiche''']]
| [https://github.com/Lechevallier/RealTimeSubtitles '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media: SRS_Groupe_5.pdf| Rapport Consultant]]
|-

!scope="row"| 6
| [[GrenobloisFuté]]
| MOURET, DELAPORTE, Lucidarme
| Nicolas Palix
| [[GrenobleFuté| '''Fiche''']]
| [https://github.com/Lucidarme/Osmand.git '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_G14.pdf|Rapport Consultant]]
|-

!scope="row"| 7
| [[Streaming en stéréoscopie]]
| ZHAO ZILONG, HAMMOUTI
| Maisonnasse
| [[Projets-2015-2016-Streaming-Stereoscopie| '''Fiche''']] - [[SRS - Streaming en stéréoscopie | '''SRS''']]
| [https://github.com/zhao-zilong/streaming_stereo '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:bruel_medewou_ndiaye.pdf|Rapport_consultant]] - [[Media:streaming.pdf|mi-parcours]]
|-

!scope="row"| 8
| [[PersyCup2016]]
| BIN, ZEGAOUI, ELLAPIN
| Donsez, Maisonnasse
| [[PersyCup| '''Fiche''']]
| [https://github.com/legominstorm/lego '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:SoutenanceMiParcours-Persycup2016.pdf|Soutenance Mi-parcours]]
|-

!scope="row"| 9
| [[Services étendus pour le modèle de composants iPOPO pour Python]]
| FOUNAS, HALLAL, GATTAZ
| Calmant & Donsez
| [[Proj-2015-2016-Extensions_IPOPO | '''Fiche''']] - [[Proj-2015-2016-Extensions_IPOPO/SRS | '''SRS''']] - [[Proj-2015-2016-Extensions_IPOPO/UML | '''UML''']]
| [https://github.com/abdelazizFounas/ipopo/tree/tlsremote '''github IPOPO'''] [https://github.com/gattazr/IPOPO-Remote-Client '''github IPOPO Client''']
| [[Media:9_RapportProjet.pdf|Rapport]] - [[Media:9_TransparentsProojet.pdf|Transparents]] - [[Media:9_FlyerProjet.pdf|Flyer]] - [[Media:3-SRS-Pres.pdf| Rapport Consultant]] - [[Media:9_PatternStrat.pdf|Pattern Design]] - [[Media:9_Mid-Presentation.pdf|Mid Presentation]] - [[Media:9_Gantt.pdf|Gantt]]
|-

!scope="row"| 10
| [[IndoorGeoloc2016]]
| ARRADA - CRASTES - FAURE - STOIAN
| Donsez
| [[Proj-2015-2016-IndoorGeoloc/Fiche| '''Fiche''']] - [[Proj-2015-2016-IndoorGeoloc/SRS|SRS]]
| [https://github.com/xxx '''github''']
| [[Media:Proj-2015-2016-IndoorGeoloc/RapportProjet.pdf|Rapport]] - [[Media:Proj-2015-2016-IndoorGeoloc/TransparentsProjet.pdf|Transparents]] - [[Media:Proj-2015-2016-IndoorGeoloc/FlyerProjet.pdf|Flyer]] - [[Media: SRSGroupe17.pdf| Rapport Consultant]]
|-

!scope="row"| 11
| [[UPnPOpenHAB2016]]
| Medewou , Ndiyae Yacine , Bruel Anna
| Didier Donsez & Jérome Maisonnasse
| [[Proj-Openhab-2016| '''Fiche''']] - [[Proj-2015-2016-Int%C3%A9gration_de_cam%C3%A9ra_de_surveillance_UPnP_%C3%A0_Openhab/SRS| '''SRS''']] - [[Proj-Openhab/UML| '''UML''']]
| [https://github.com/openHab-UPnP '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:gl_ZHAO_HAMMOUTI.pdf|Rapport Consultant]] - [[Media:pattern_ZHAO_HAMMOUTI.pdf|Patterns]] - [[Media:fichier.pdf|Mini soutenance]]
|-

!scope="row"| 12
| [[Sign2Speech]]
| NIOGRET, NOGUERON, TITH
| Didier Donsez
| [[sign2speech_ricm4_2015_2016| '''Fiche''']] - [[SRS - Sign2Speech | '''SRS''']] - [[UML | '''UML''']]
| [https://github.com/SignToSpeech-Project '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:12-Sign2Speech-RapportConsultant.pdf|Rapport Consultant]] - [[Media:12-Sign2Speech-MidPres.pdf|Mid presentation]]
|-

!scope="row"| 13
| [[AstroImage]]
| RACHEX, BLANC, GERRY
| Olivier Richard et Bruno Bzeznik
| [[Proj-2015-2016-Astroimage/Fiche| '''Fiche''']] - [[AstroImage/SRS | '''SRS''']] - [[Media:AstroImage-UML.pdf | '''UML''']]
| [https://github.com/nicolas-blanc/AstroImage '''github''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:13-AstroImage-RapportConsultant.pdf|Rapport Consultant]] - [https://docs.google.com/presentation/d/15F8DRktwmOuSNabdxMASniyr-TIiRzGNNG1mOhcoSnk/edit?usp=sharing '''Patterns''']
|-

!scope="row"| 14
| [[Tachymètre]]
| MACE, NOUGUIER, RAMEL
| Olivier Gattaz
| [[Fiche - Tachymètre | '''Fiche''']] - [[SRS - Tachymètre| '''SRS''']] - [[UML - Tachymètre| '''UML''']]
| [https://github.com/Quego/Tachymetre '''github - Tachymètre''']
| [[Media:RapportProjet.pdf|Rapport]] - [[Media:TransparentsProojet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:srs_tachymetre.pdf|Rapport consultant]] - [[Media:14_PatternDesign.pdf‎ | Pattern Design]] - [[Media:Tachymetre_Presentation.pdf‎ | Présentation]]
|-

!scope="row"| 15
| [[SmartProjector]]
| BRANGER, HABLOT
| Donsez, Maisonnasse
| [[Fiche_SmartProjector_ricm4_2015_2016| '''Fiche''']] - [[SRS - SmartProjector| '''SRS''']] - [[UML - SmartProjector| '''UML''']]
| [https://github.com/P0ppoff/SmartProjector '''github''']
| [[Rapport]] - [[Media:PresentationPorjet.pdf|Transparents]] - [[Media:FlyerProjet.pdf|Flyer]] - [[Media:Gl_groupe16.pdf|Rapport Consultant]] - [http://air.imag.fr/index.php/Patron_de_conception_-_SmartProjector patterns]
|-

|}

===Liste de projets===

* [[Dashboard pour gestionnaire de tâches et de ressources]], Olivier Richard
* [[Moteur distribué d'exécution de commande]], Olivier Richard
* [[Environnement d'expérimentation de pour NVIDIA Shield (Tegra X1)]], Olivier Richard
* [[Speeding Simplified Script Language]], Olivier Richard

* Aide (Open-Source)au Handicap Auditif, avec Didier Donsez, Jérome Maisonnasse, Marie-Paule Balicco (SAH UGA) et Nicolas Vuillerme
** [[Borne interactive]] (1 sujet)
** [[Sonotone]] (1 sujet)
** [[Sous-titre en temps réel d'un cours]] (1 sujet)
* [[GrenobloisFuté]] Couche trafic sur OsmAnd avec un greffon. Données dynamique de la métro. Dvp Android. Nicolas Palix.
* [[GeoDiff]] Production, visualisation, fusion de variations (diff) sur de l'information géocodée : Nicolas Palix
* [[Smart campus augmenté et contributif]] Didier Donsez, Vivien Quema

* [[Streaming en stéréoscopie]] sur [[WebRTC]] avec rendu sur [[Oculus]] pour le robot [[RobAIR]], Jérôme Maisonnasse. ([http://gstconf.ubicast.tv/videos/stereoscopic-3d-video/ voir]).
* [[STM32F7]] : Mise en oeuvre de la chaîne de compilation sous Linux avec [[OpenSTM32]] et [[OpenOCD]]. Nicolas Palix
* [[PersyCup2016]] : Persyval Robocup, Didier Donsez, Vivien Quema, Jérome Maisonnasse. (3 étudiants)
* [[Services étendus pour le modèle de composants iPOPO pour Python]], Didier Donsez & Thomas Calmant. (2 étudiants)
* [[SmartClassRoom2016|Développement d'une interface partagée pour tables tactiles (projet SmartClassRoom)]], Didier Donsez, Jérôme Maisonnasse. (2 étudiants)
* [[iRock2016|iRock : surveillance de glissement de terrains]], Didier Donsez & Vivien Quema
* [[IndoorGeoloc2016|Géolocalisation in-door au moyen de balises (beacon) BLE et Wifi à base de STM32 et de balises iBeacon & AltBeacon]], Didier Donsez & Vivien Quema
* [[UPnPOpenHAB2016|Intégration et gestion de caméras de surveillance UPnP dans la plateforme domotique open-source OpenHAB et myOpenHAB]], Didier Donsez & Jérome Maisonnasse.

'''Projets non prioritaires'''

* [[Liveprogramming with Kivy]], Olivier Richard
* [[AstroImage]] production d'image d'astronomie, Olivier Richard et Bruno Bzeznik
* [[G-code Cruncher]] Controle de machine CNC (Nucleo grbl + esp8266 + Sdcard), Olivier Richard
* [[Intégration OpenHAB / OpenTele]] Nicolas Palix

==RICM5==

===Projet Semestre S10===

Enseignant responsable : Didier Donsez

Démarrage : Lundi 25/01 à 10H30-12H30, P253 (Rendez-vous devant la salle AIR) - Visioconf pour Thibaut Cordier

Soutenance : Vendredi 18/03 à 8H30-12H30, P257

Etudiants : RICM5 + 8 étudiants Avosti DUT RT

Rappel séances MPI
* Séance 1 : mardi 26 janvier après midi - Stéphanie Diligent
* Séance 2 : mardi 2 février après midi - Stéphanie Diligent
* Séance 3 : lundi 8 février matin - Emmanuelle Tréhoust
* Séance 4 : jeudi 11 février matin - Emmanuelle Tréhoust
* Séance 5 : lundi 21 mars matin - Stéphanie Diligent et Emmanuelle Tréhoust

Planning soutenance (à venir).
* Bossa
* IaaS Docker
* Immersion EDF
* SmartCampus
* SmartClassRoom (en C005)
* Pot d' "Au Revoir"

{|class="wikitable alternance"
|+ Affectation des projets RICM5 2015-2016
|-
|
!scope="col"| Sujet
!scope="col"| Etudiants
!scope="col"| Enseignant(s)
!scope="col"| Fiche de suivi
!scope="col"| Dépot git
|-

!scope="row"| 1
| [http://air.imag.fr/index.php/IaaS_collaboratif_avec_Docker IaaS - Docker]
| Eudes Robin, Damotte Alan, Barthelemy Romain, Mammar Malek, Guo Kai, Bonnard Loïc, Caperan Théo
| Didier Donsez
| [[Projets-2015-2016-IaaS_Docker| '''Fiche'''] - [Projets-2015-2016-IaaS_Docker-SRS| '''SRS''']]
| [https://github.com/EudesRobin/iaas-collaboratif '''github''']
| [[Media:Rapport_IaaS.pdf|Rapport]] - [[Media:Transparents_IaaS.pdf|Transparents]] - [[Media:Flyer_IaaS.pdf|Flyer]]
|-
!scope="row"| 2
| [http://air.imag.fr/index.php/Portage_de_Bossa Portage de Bossa sur le Kernel Linux 4x]
| Eric Michel Fotsing, Ombeline Rossi, Longfei Yao
| Nicolas Palix, Didier Donsez
| [[Projets-2015-2016-Portage_Bossa| '''Fiche''']]
| [https://github.com/ZenithKaizer/ '''github''']
| [[Media:Rapport_Bossa.pdf|Rapport]] - [[Media:Transparents_Bossa.pdf|Transparents]] - [[Media:Flyer_Bossa.pdf|Flyer]]
|-

!scope="row"| 3
| [[Visite immersive en réalité virtuelle dans une usine avec EDF]]
| Adam Christophe, Aissanou Sarah, Klipffel Tararaina, Qian Jean, Zominy Laurent
| Didier Donsez, Georges-Pierre Bonneau, Thibaut Cordier (EDF)
| [[Projets-2015-2016-VisiteImmersiveEDF| '''Fiche''']]
| [https://github.com/VisiteImmersiveEDF '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]]
|-

!scope="row"| 4
| [[Contribution à OpenSmartCampus]] (voir http://data.beta.metropolegrenoble.fr/)
| Quentin Torck, Vivien Michel, Jérémy Hammerer, Rama Codazzi, Zhengmeng Zhang
| Didier Donsez, Vivien Quéma
| [[Projets-2015-2016-OpenSmartCampus| '''Fiche''']]
| [https://github.com/quentin74/SmartCampus.git '''github''']
| [[Media:RapportProjetX.pdf|Rapport]] - [[Media:TransparentsProojetX.pdf|Transparents]] - [[Media:FlyerProjetX.pdf|Flyer]]
|-

!scope="row"| 5
| [[Contribution à SmartClassRoom]] (Interfaces tactiles distribuées et partagées)
| Saussac Thibault, Toussaint Sébastien, Hamdani Youcef, Zoppello Sebastien, Melik sak, Mesnier Vincent
| Jérôme Maisonnasse, Didier Donsez
| [[Projets-2015-2016-SmartClassRoom| '''Fiche''']]
| [https://github.com/XXXX '''github''']
| [[Media:RapportProjetSmartClassRoom.pdf|Rapport]] - [[Media:TransparentsProjetSmartClassRoom.pdf|Transparents]] - [[Media:FlyerProjetSmartClassRoom.pdf|Flyer]]
|-

|}

===Projets annulés et reportés===
* Projet avec [[Tango Project]] (Annulé)
* Hack the Beam, Didier Donsez & Jérôme Maisonnasse.
* [[Algorithmes de suivi de personnes pour robot de téléprésence RobAIR]] (Jérôme Maisonnasse, Didier Donsez)

=M2PGI=
==[[Projets M2PGI Services Machine-to-Machine|Projet Services Machine-to-Machine]]==
* [[PM2M/2016/TP|Sujet et groupes]]

File:Flyer IaaS.pdf

2016-03-08T09:29:08Z

Alan.Damotte:

Projets-2015-2016-IaaS Docker

2016-03-08T09:16:23Z

Alan.Damotte: /* Roadmap */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients
* Implement public profile: at the moment, users can only access their private profile. We imagine that we can consult providers profile to see which one is best rated
* Add the possibility for clients to use the rating system to choose only the best rated providers (special package, more expensive of course)

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network): it's better for both clients and providers to be on the same geographical area
* Implement active replication in case a provider suddenly stops his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains (difficult): since providers are admin of their machine they can see what the containers contain, or enter the containers. It would be good to guarantee clients that their instances are totally safe, and no one, including the provider, can access their information.

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker

2016-03-08T09:07:39Z

Alan.Damotte: /* Week 7: Mars 7th - Mars 13th */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network)
* Implement active replication in case a provider suddenly stop his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===
* Finish creating the flyer
* Start to write report for our last MPI course
* End of Rabbitmq set up on front-end
* Add a rating system which will be use to give a mark to providers.

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]

Projets-2015-2016-IaaS Docker

2016-03-08T09:06:09Z

Alan.Damotte: /* Roadmap */

[[Image:collaborativIaas.jpg|right|400px]]

= Project presentation =
== Introduction ==

The objective of this project is to allow a user group (member) to pool their laptops or desktop in order to calculate big data of few users. To do so, the solution should work with Docker to virtualize user machines and control the use of resources of each machine.

Project under GPLv3 licence : https://www.gnu.org/licenses/gpl-3.0.fr.html

== The team ==
'''RICM5 students'''

* EUDES Robin
* DAMOTTE Alan
* BARTHELEMY Romain
* MAMMAR Malek
* GUO Kai

'''DUT Students'''

* BONNARD Loïc
* CAPERAN Théo

'''Supervisors'''
Pierre-Yves Gibello (Linagora), Vincent Zurczak (Linagora), Didier Donsez

== Deliverables ==
[https://github.com/EudesRobin/iaas-collaboratif Github repository]

[https://waffle.io/EudesRobin/iaas-collaboratif Waffle.io]

[[Media:CahierdeschargesIaas.pdf|Specifications (written in French)]]

= Roadmap =
Our waffle shows our current roadmap and the different tasks we are working on.
The aim of this section is to gather all the ideas we have which would be good to implement in the future to improve the service (after the end of our project).

'''User experience:'''
* Add a way to report bad behaviour of providers or clients

'''Algorithms:'''
* Implement algorithm that optimize geographic allocation between providers and clients (better network)
* Implement active replication in case a provider suddenly stop his machine
* Reallocate the instances to another provider when the first one decide to cleanly stop his machine (docker commit/docker pull)
* Optimize disk usage and bandwidth allocation

'''Security:'''
* Find way to prevent provider to enter in the instance and do whatever he wants, or see what each instances running on his machines contains

'''Monetary system:'''
* Implement monetary system for providers and clients
* Set different possible packages at different prices and for different levels of service

= Planning =

[[File:gantt_iaas.png|center|thumb|1000px|Preliminary Gantt chart]]
[[File:gantt0309_iaas.png|center|thumb|1000px|Gantt chart at March 9th]]

=== Week 1: January 25th - January 31th ===
* Getting familiar with Docker (for some of the group members)
* Fix Docker's DNS issue using public network (wifi-campus/eduroam)
* Contacting our supervisors
* First thoughts on this project, what we could do
* Redaction of specifications, creation of architecture diagrams
* Create scripts that start/stop containers automatically (some modifications still need to be done)

=== Week 2: February 1st - February 7th ===
* Manage and limit space disk usage of each container, limit resources allocation at containers' launch.
** CPU and memory allocation: ok
** Docker doesn't seem to implement easy way to limit container's disk usage: implementing a watchdog (script) which will check container's disk usage and stop those that exceed a limit
* Think about restricted access to Docker containers: for the moment, providers are admin and can easily access to containers
* See how instances can easily give their network information to coordinator
* Get familiar with Shinken and study the possibilities
* Speficication of technologies used
* End of specification redaction + feedback from tutors
* Start to work on Meteor-AngularJS tutorials
* Configure a personal VM for the frontend & setup meteor-angular on it

=== Week 3: February 8th - February 14th ===
* '''Objective for this week:''' get a prototype that contains a basic front-end which make it possible to launch remote Docker instance.
* Container deployment:
** Deploy all containers on the same network: that allows us to connect to the instances from the coordinator
** Create user on host: will be used to connect ourselves in ssh from coordinator instance to host and launch deployment scripts
** Create script that totally automatize user creation, images creation and build, coordinator's and shinken's containers launch
* At the end of the week, the prototype is working: we can launch an instance an a provider machine from the front-end. We still need to establish and test the connection between a client and his instance. We have a good cornerstone of our project yet.

=== Week 4: February 15th - February 21st ===
* Try to establish connection between a client and his container
* Continue client/provider's web page development on front-end
* Start editing help page
* Correct some responsive effects on the site
* Container deployment:
** Implements bandwidth restriction
** Create script that automatically set client public key in container's authorized_keys file, modify some script to automatically delete client public key in coordinator's authorized_keys file
* Start to study and set up Rabbitmq (publish from provider to front-end for example)

=== Week 5: February 22nd - February 28th (Vacation) ===
* Update wiki/help page, work on some responsive issues on the website
* Establish script that automatically create SSH-jump config for the client
* Work on foreign keys and database (front-end side)
* Continue front-end development
* Establish rabbitmq on both front-end side and provider side

=== Week 6: February 29th - Mars 6th ===
* Container's deployement:
** Modify coordinator Dockerfile to install nodejs
** Create a cron job that will run a command every 30 seconds: that command will be used to send the file that contains container's information to rabbitmq server
** Modify coordinator to set up 2 users: one for the front-end and one for the clients. Each one will contains only the public key they need in authorized_keys' file
** Modify startProvider script to check is ssh-server is installed and running on provider, and change default port (22 to 22000)
** Modify watchdog functioning: up to now, the script was just checking if each instance was respecting a limit. Now it's behaviour allow us to have different disk usage for each instance. Now use a cron job, we won't need anymore to launch the script by ourself
** Change monitoring system: we found an other monitoring system for Docker called cAdvisor which gives us enough information about containers.

* Frontend dev:
** Generate a proper & unique instance name : <username>-<provider_domain_name>-<num_instance_user_at_provider> eg. : toto-domain1-0
** Add form to modify provider machines informations
** Fix warning "CSS file deliver as html file" by Meteor
** Add README to explain how to use scripts, how files are organized (for github branch : frontendWebui , docker , master )
** Improve user feedback (notifications) on errors/success
** Proper parameters to start/stop instances
** Add username field in profile
** Resolve bugs occurring when the machines allocate resources from a different user

*Test and feedback:
** Set up the main test: container deployment and access to instance from the client
** Some permissions on coordinator instance needed to be changed
** SSH default configuration needed to be changed to: disable root login and authentication by password
** Connection from client to his instance is working
=> The main development phase is finished since we have a working base. We still need to improve some things, eventually development some advanced functionalities during the last two weeks.

=== Week 7: Mars 7th - Mars 13th ===

=== Week 8: Mars 14th - Mars 18th ===

=What is Docker?=
Docker allows you to package an application with all of its dependencies into a standardized unit for software development.
Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

'''Lightweight:'''
Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient.

'''Open:'''
Docker containers are based on open standards allowing containers to run on all major Linux distributions and Microsoft operating systems with support for every infrastructure.

'''Secure:'''
Containers isolate applications from each other and the underlying infrastructure while providing an added layer of protection for the application.

==How is this different from virtual machines?==
[[File:VM_vsContainer.png|200px|thumb|Virtual machines]][[File:Container_vsVM.png|200px|thumb|Docker containers]]

Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

'''Virtual Machines:'''
Each virtual machine includes the application, the necessary binaries and libraries and an entire guest operating system - all of which may be tens of GBs in size.

'''Containers:'''
Containers include the application and all of its dependencies, but share the kernel with other containers. They run as an isolated process in userspace on the host operating system. They’re also not tied to any specific infrastructure – Docker containers run on any computer, on any infrastructure and in any cloud.

==How does this help you build better software?==
When your app is in Docker containers, you don’t have to worry about setting up and maintaining different environments or different tooling for each language. Focus on creating new features, fixing issues and shipping software.

'''Accelerate Developer Onboarding:'''
Stop wasting hours trying to setup developer environments, spin up new instances and make copies of production code to run locally. With Docker, you can easily take copies of your live environment and run on any new endpoint running Docker.

'''Empower Developer Creativity:'''
The isolation capabilities of Docker containers free developers from the worries of using “approved” language stacks and tooling. Developers can use the best language and tools for their application service without worrying about causing conflict issues.

'''Eliminate Environment Inconsistencies:'''
By packaging up the application with its configs and dependencies together and shipping as a container, the application will always work as designed locally, on another machine, in test or production. No more worries about having to install the same configs into a different environment.

==Easily Share and Collaborate on Applications==

Docker creates a common framework for developers and sysadmins to work together on distributed applications

'''Distribute and share content:'''
Store, distribute and manage your Docker images in your Docker Hub with your team. Image updates, changes and history are automatically shared across your organization.

'''Simply share your application with others:'''
Ship one or many containers to others or downstream service teams without worrying about different environment dependencies creating issues with your application. Other teams can easily link to or test against your app without having to learn or worry about how it works.

==Ship More Software Faster==

Docker allows you to dynamically change your application like never before from adding new capabilities, scaling out services to quickly changing problem areas.

'''Ship 7X More:'''
Docker users on average ship software 7X more after deploying Docker in their environment. More frequent updates provide more value to your customers faster.

'''Quickly Scale:'''
Docker containers spin up and down in seconds making it easy to scale an application service at any time to satisfy peak customer demand, then just as easily spin down those containers to only use the resources you need when you need it

'''Easily Remediate Issues:'''
Docker make it easy to identify issues and isolate the problem container, quickly roll back to make the necessary changes then push the updated container into production. The isolation between containers make these changes less disruptive than traditional software models.

= System Architecture =
== Instances allocation ==
[[File:Infrastructure_globale.png|center|thumb|1000px|Global infrastructure]]

== SSH connections to allocated instances ==
[[File:Infra_generale_network.png|center|thumb|1000px|Network global infrastructure]]

[[File:Legend_infra.png|center|thumb|1000px|Caption]]

== Coordinator and Frontend details ==

[[File:Coordinator.png]][[File:Frontend.png]]

= Containers' automatic deployment =

The aim of this part is to automatize the containers deployment on provider side. This includes launching coordinator instance and monitoring instance (shinken). The coordinator instance will allow us to launch new containers and establish the link between clients et their containers.

[[File:Provider_functioning.png|center|thumb|1000px|Provider functioning]]

== Build and run ==

'''First step: user creation'''

Since we can only interact with the coordinator instance from the front-end, we need a way to launch new container. It's not possible to do so from a container, and that task need to be done from the host. That's why the first step is to create a new user on provider machine we will use to launch new containers or stop them. The moment it is done, we deploy necessary scripts in this user's home. Those scripts are necessary to launch and stop new containers. It is simpler for us to do so than transferring those files from the coordinator to the host when the connection is established.

'''Second step: images creation'''

Then, the second step consists in building coordinator and monitoring (shinken) images. To do so we use Dockerfile that allow us to build a container containing all we need. The coordinator instance just contains a ssh web-server. That container exposes it's port 22 and will be used as a jump host to connect the front-end/clients to the other instances.

'''Third step: coordinator and monitoring instance deployment'''

Finally, when the images are successfully built, we can run these containers on Docker deamon. We are now able to connect the front-end to the coordinator instance and deploy instances.

== Resources management ==

Docker already provides some functionalities which allow us to restrict CPU and memory usage. However, we needed to implement some functionalities ourselves like space disk usage and bandwidth restriction.

'''CPU:''' To restrict CPU usage, we just need to know the hyper-threading coefficient and remember which CPU is already used. There is a Docker option we can use while launching container that allow us to choose which CPU the container will use to run.
The example below shows how this works with 4 CPU (and hyper-threading coefficient is 2).

[[File:CPUShare.png|center|thumb|1000px|CPU share]]

'''Memory:''' While launching a container, we set memory soft limit as the value required/reserved by the client. The hard limit is set as the maximum memory made available by the provider In doing so, a container can use more memory that his soft limit. But if several containers are running on the same host, Docker will ensure that each container doesn't consume more memory than his soft limit.

'''Disk:''' Docker doesn't seem to provide a functionality to restrict disk usage. And yet, it's really important for us to make sure that a client will not use to much space disk of the provider. To do so, we implemented a watchdog that check every 30 seconds the disk usage of each container and stop them if they reach the limit defined by the provider. We also use that watchdog to inspect and save container's information that will be used on the front-end to display container's state and space disk usage. Thanks to that, clients will know if they are about to reach the limit.

'''Bandwidth:''' Since all the containers run on the same Docker network, we are able to use Wondershaper to set a limit for bandwidth usage. Then, Docker takes care to divide equitably the available bandwidth to each container.

= Useful links =

== Network related ==
* [https://wiki.gentoo.org/wiki/SSH_jump_host SSH Jump Host]

== Git related ==
* [https://www.atlassian.com/git/tutorials/ Git tutorials]
* [https://help.github.com/articles/changing-a-remote-s-url/ Switch https to ssh - remote url github]
* [https://gist.github.com/jexchan/2351996 Multiple SSH keys for github]

== Docker related ==
* [https://docs.docker.com/ Docker official website]
* [http://2.bp.blogspot.com/-ZGXYBT4l9II/U5BFJwe_jWI/AAAAAAAAGB4/-Le5-NavlGg/s1600/docker-cycle.png Docker cycle]
* [https://www.wanadev.fr/tuto-debuter-et-comprendre-docker/ Understand how Docker works]
* [http://www.occitech.fr/blog/2014/10/tuto-docker-hello-world/ Docker tutorial - Hello world (understand basic commands)]
* [https://robinwinslow.uk/2014/08/27/fix-docker-networking/ Fix Docker's DNS issue with public network]

== Meteor / MongoDB related ==
* [http://www.angular-meteor.com/ Angular Meteor official website]
* [https://www.mongodb.org/ MongoDB, the database used]
* [https://github.com/aldeed/meteor-collection2 Collection2 - A Meteor package that allows you to attach a schema to a Mongo.Collection]
* [https://github.com/laverdet/node-fibers#futures Asynchronous call in Meteor with fibers/future]
* [http://bootstrap-notify.remabledesigns.com/ Notification module used to display pretty notifications]