Skip to content

UBUNTU LTS

I’ve discussed before about the new Ubuntu LTS upgrade. I can say now some tricks I’ve learned.

First of all, going to PHP7 can be frustrating. First because the PHP-FastCGI is installed by default during the upgrdae. This is mainly to run PHP7 with the nginx server. When you come from PHP5 and a running install of Apache 2.4, you don’t want this stuff. Then I stopped the PHP Fast-CGI which is basically a PHP server and I’ve configured my Apache 2.4 to run correctly.

Due to this configuration, PHP was “out-of-service” but not the PHP pages. I have exposed the code source of my pages to the world. Then be careful… On my side, nothing to regret: all the sensible files containing the configuration are stored outside the public area of the site. Just some users have seen some very surprising pages. Note Google has also indexed these pages. Then having your sensible data accessible from the public area is very dangerous. Note the users of WordPress can store their config.php file a folder above the public area. This is a very good advice.

Then, after this, I needed to re-install all the extensions including the MongoDB in his new release. Basically, the new version of the driver is easy to install if you are prepared. Better to test with a PHP5 working installation.

The last (but very annoying) thing about the update, this is the rotation file for Apache logs. Edit the “/etc/logrotate.d/apache2″ file. I was surprised (some time after) that only one week of logs were retained. Not enough for me. I changed it manually.

The very bad news is concerning MongoDB. First of all, MongoDB goes to version 2.6. Which is quite ridiculous. For a long-term release, a version 3.0 was expected. Nevertheless, you can update your database to a recent (a decent) version with the MongoDB repository. Note this version 2.6 do NOT start automatically when the server boots. I don’t know why. As I restart my server twice a year, I don’t mind really and I will upgrade to version 3 instead of looking for a solution.

Note the upgrade was running during one hour but the server can continue to serve your webpages through Apache2 because there is no many reboots and things are conceived to go straight.

have a good upgrade (if you have not done it yet).

 

Robots.txt

Beaucoup de gens connaissent le fichier robots.txt. Placé dans le répertoire racine du serveur WEB, il permet d’indiquer aux moteurs de recherche les répertoires à éviter.

Bien entendu, ce fichier n’est donné qu’à titre indicatif et seuls les moteurs de recherche sérieux le prennent en compte. Il est relativement facile de créer un fichier robots.txt

Dans le cas de mon site WEB, celui contient les informations suivantes:

User-agent: *
Disallow: /cgi-bin/
SITEMAP: http://www.koikonfait.com/sitemap.xml

Pas davantage. Pour mieux comprendre:

La partie “User-agent” permet de définir quels user agents a le droit de passer et lequel sera interdit (comprendre: persona non grata). À priori, il est contre-productif d’interdire l’accès à certains moteurs de recherche. Si vous aimez la politique du fou, vous pouvez interdire à Google d’indexer votre site. Pourquoi pas? Google respecte vos prescriptions à la lettre.

Idem pour les autres moteurs de recherches. Encore faut-il avoir avoir accès aux logs de votre serveur pour connaître la liste des robots qui passent sur votre site. Si votre site est hébergé, il est probable que vous n’ayez pas accès à l’information. Dans le cas où vous héberger votre site, cette information se trouve généralement dans /var/logs/apache2.

Certains robots semblent stupides voire gênants (je citerai majestic par exemple). D’autres peuvent surprendre (comme Baidu, le moteur de recherche chinois). D’autres plus justifiés comme BingBot (le moteur de recherche de Microsoft). Personnellement, je ne fais pas de sectarisme: tout le monde a accès.

En revanche, j’ai interdit l’accès au répertoire “/cgi-bin/” qui est censé recevoir des programmes exécutables. C’est une vieille habitude qui n’a aucun sens car ce répertoire est vide sur mon serveur!

Il existe également une directive très intéressante que, personnellement, je n’utilise pas:

Crawl-delay: 10

Cette information permet d’indiquer aux moteurs de recherche d’attendre 10 secondes entre chaque requête. Cela permet de limiter la charge du serveur. Personnellement, je n’ai pas pris la peine de régler ce paramètre. Je ne suis pas sûr qu’il soit respecté. Mais la raison principale, c’est que mon serveur est prévu “pour tenir le choc”. Est-ce que les moteurs de recherche sont des sauvages?

Oui pour Bing, le moteur de recherche de Microsoft. Il est capable d’envoyer 5 requêtes dans la même seconde. Pas très fair-play. De son côté, Google va être incroyable. Ses requêtes sont régulièrement espacées. Généralement d’au moins 5 secondes (avec des exceptions). Dans le cas de redirection (type 301 ou 302), il ne va pas se forcer à lire la page immédiatement, il va la mettre dans sa liste “à scanner”. Je n’imagine pas l’algorithme mis en place mais il présente de gros avantages: fair-play avec le site WEB et économique pour la bande passante Google. Une stratégie “gagnant-gagnant”.

 

Ubuntu 16.04 LTS

For those who uses an Ubuntu server 14.04 LTS, you will be able to move forward… To the 16.04 LTS. Quite easily.

You will be able to move from the old 14.04 to the new 16.04 in the next days (when the version 16.04.1 will be out, it is planned for July 21).

Why upgrading the system? To keep your machine with recent versions. From my point of view, the main differences will be PHP (from 5.5 to 7! Not a big deal.. But much faster) and MongoDB (expected to go from 2.4 to 3.2). Concerning my server, this is the main expectation. Having a Mongo 3.2 with WireTiger could be a very good deal. Note, the distribution upgrade will NOT upgrade the Mongo file system: you have to do it manually by first backuping your data then modifying the data directory and the engine for mongo. Then restoring the data.

For the release itself. I just put the way to do (if you have a server and do the upgrade through ssh). NOTE: YOU HAVE FIRST TO BACKUP YOUR SYSTEM. My provider gives me the capability to backup the disks without fear. Then I consider everything done about backup.

First step, install the “update manager”. It is very easy:

sudo apt-get install update-manager-core

Starting form July 21, check if you have the brand new version:

do-release-upgrade -c

If yes, do the backup. Then run the upgrade:

sudo do-release-upgrade

Wait and see… In my next post, I will give you my point of view and the time needed to upgrade. And if I failed… basically, the upgrade should be done next week (July 26th or 27th, perhaps later). Currently, I serve more than 150 pages per day and a mobile application. Stopping the server for upgrade should be done in less than 1 hour… We will see.

IMPORTANT NOTE: there is a change about ssh connection (I think the default connection will be without password). If you upgrade through the network, be very careful.

There is a page about upgrading than giving more or less the same advices: How to upgrade to Ubuntu 16.04 LTS .

Good luck…

NOTE 27/07: it seems the upgrade for servers is not ready. See bug #1605474 for more information. The upgrade could be available by the end of July. Using the “-d” option is not recommended. If you use a LTS version, I guess you are patient. 

 

Du concept au produit: fait!

Le concept était simple: trouver une activité à proximité de chez soi dans moins de 2h.

Ben voilà. Simple.

Le produit: un site WEB (juin 2015). Simple: 1 mois de codage (en PHP et MongoDB parce que ça permet de coder vite). Et relativement pas trop mal (moi qui suis architecte et développeur JAVA, je ne devrais pas dire cela).

Juillet 2015: les cinémas et quefaire.paris.fr disponibles.

Puis l’application sur iPhone (et iPad): une carte qui affiche les activités pour les deux prochaines heures.

Puis l’application sur Android. Pareil: une carte.

Septembre 2015: un site d’administration pour me donner quelques statistiques.

Octobre 2015: les brocantes.

Janvier 2016: BilletRéduc (excellente idée) et EventBrite (le site de billeterie).

10000 évènements disponibles principalement sur Paris et Genève en février 2016. Des versions améliorées des applications.

Février 2016: arrivée de la FNAC. On passe la barre des 20000 évènements.

Mai 2016: on retire les doublons… Parce que nos fournisseurs nous renvoient parfois les mêmes infos (concerts, théâtre…). On atteint les 40000 évènements dans la base avec MeetUp.

Juin 2016: 50000 évènements. France, Belgique et Suisse. Nouvelle page d’accueil.

Juillet 2016: 75000 évènements. Avec Facebook, on sature la pauvre machine avec son 1Go de mémoire et les temps de réponses faiblissent mais c’est sans compter sur quelques astuces de programmation qui se profilent à l’horizon…

En 1 année, un concept est devenu une machine de guerre. koikonfait.com est passé de concept à réalité. Pas de morale à donner. Pas de conclusion. Une constatation. Avec des outils Open Source, un serveur VPS et un budget de 30 euros/mois (l’hébergement, le nom de domaine, l’abonnement Apple Store pour l’application iPhone…), je suis arrivé à mes fins. Je peux prendre mon appli et décider de ma soirée: ce sera sans doute un concert quelque part à Genève…

Capture d’écran 2016-07-13 à 00.18.49.png

Répartition des activités sur la zone francophone

 

 

 

Now Unix works in Windows 10

I’m very impressed by the demo: you can get a Linux box in a Windows 10 environment. Just seen the demo and I am very excited. It is not a joke, it a something like a Linux emulator (basically, the kernel calls have been mapped into the Windows NT DLL).

It is not completed yet but I fell like we had run a MS/DOS emulation under Solaris back to 90’s. Full of hope. This is the end of the war between Linux and Microsoft, at last.

Just type “bash” in a simple command line window…! Well, you have to wait a little for the deployment in Windows 10 or subscribe for Developper Preview

Git for Subversion users

First of all: GIT is NOT Subversion. The commands are NOT the same and the behaviour of the same commands are different. A “commit” has NOT the same meaning in the two software. Then the best is to think differently. But sometimes, we could imagine a one-to-one guide. Let’s try.

svn checkout is git clone

As you checkout the central repository in subversion, you just make a copy of the parent one in git. This is why the command is not the same. Of course, I consider you have a read/write access to the “parent” repository you try to copy. This is to keep the same behaviour than for Subversion.

svn checkout <repo> creates the local copy of the repository on your disk (you have the last version). Note sometimes you write svn checkout <repo>/trunk <repo> to only copy the development branch.

git clone <repo> will clone the repository on you local disk (you have all the history of the project including the last version but also all the branches). This is why the merge is so simple in git: all the history is available.

svn propedit svn:ignore is .gitignore

If you need to remove some files from the source system (object files, compiled results, logs…), you just have to create a .gitignore file at the root of the project. No need to use the “propset” (or “propedit”) commands, one per directory.

See git documentation for some subtle notation.

svn revert is git checkout

What a shame! Reverting to the repository last version is made through a checkout in git! What that fuck? Well, checkout is getting the last version in Subversion. Same on git. This is why there is no revert command in git. Think you can achieve a revert by forcing the checkout of the last version in subversion.

Note a “revert” command could be a good way to “match” Subversion and Git but also is quite inadmissible for git purists.

svn update is git pull origin master

To get the last version from svn, just write svn update

But, git is a multiple repository system and multiple branch system. You must give these 2 information. Then git pull origin master. Because you get the master (i.e. trunk) branch from the “origin” repository. But you could do the same with another parent repository. Well, of course, it is out of scope of this post because svn has a unique repository.

Subversion has branches but managed as separate projects (usually using “tags”, “branches” and “trunk” directories). git is more like the CVS ancestor: branches are part of the tool.

svn move is mv

There is no “move” command to rename a file locally (and absolutely not possibility to rename a file remotely). You simple move your file with the “mv” command on Unix or rename it through the Explorer under Windows (or the Finder for Mac users).

Because there is no tracking like in Subversion, git has no need for marking files changed. It will see it. Note that that for copying or deleting a file, just do the same: use the operating system commands.

svn status is git status

Well, not so surprising. You are checking the “local” status of your files. Note git has a slightly different notation than Subversion but you can understand easily what has changed and what added.

The main difference is about the management of changes. In Subversion, you can commit changes made on files. You just have to add manually new files. git is just about “changes” including new files. But there are good news: renaming a file in git is simply renaming the file. git will understand a file has been renamed and will do the job.

svn commit is git add + commit + push

Because your local directory is a git repository (yes, it is), commit has the sense of “commit” on your repository. Then to send back to the repository (in the sense of Subversion), you have to push.

Because there is no renaming or moving and because your work is a complete repository and history, you have to first add the files and push back all of them to the origin master.

git add --all
git commit -m "<my comments about changes>"
git push origin master

An important point

Never forget that git will NEVER store an empty directory. If you need an empty directory to be saved in the git repository, you need to add a fake file inside (usually a README.md because it is a good practice).

Conclusion

This post is NOT to force you to go to a git repository but to make the adoption simple. I don’t want to change your source control from subversion to git. Personally, I use both. Sometimes subversion is better than git depending of your organization, your management and so on. And this post is the very minimal of the minimal. If you don’t understand the philosophy behind git, you will just continue to suppose these tools are the same. But not: they are really different.

Redirecting in Unix

Well, sometimes we can use a post for ourselves. This post is part of those.

Sometimes, redirecting is a little mess. Because I wanted to create backups of my database and the output goes right to a file and stderr is just an informative output. This is achieved by the commands curl and mongoexport among others.

Then what is I want to have the stderr sent to the stdout?

Well, quite simple. Follow the example below:

( mongoexport -db mydb --collection my 2>&3 | gzip -c > my.json.gz ) 3>&1

 

The explanation: you send the output to a “third” channel (achieved by 2>&3). Then simply pipe to the compression made through gzip. Don’t forget to make it when a high priority (using the parenthesis) and send back the channel 3 to the stdout (channel 1) by adding 3>&1.

That’s magic but sometimes I need to look on the internet for this trick. And I loose my time googling. Now I know where to find the info! If someone can achieve this in Windows10 on the command line, just drop me the way to do in the comments.