From 23a53661e05761a475a5f032755cca148df24682 Mon Sep 17 00:00:00 2001 From: Marc Tobias Metten Date: Thu, 25 Jun 2015 04:21:32 +0200 Subject: [PATCH] first version of Vagrant installation instructions --- VAGRANT.md | 167 +++++++++++++++++++++++++++++++++++++++++++ Vagrantfile | 49 +++++++++++++ vagrant-provision.sh | 166 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 382 insertions(+) create mode 100644 VAGRANT.md create mode 100644 Vagrantfile create mode 100755 vagrant-provision.sh diff --git a/VAGRANT.md b/VAGRANT.md new file mode 100644 index 00000000..26b0ee75 --- /dev/null +++ b/VAGRANT.md @@ -0,0 +1,167 @@ +# Install Nominatim in a virtual machine for development and testing + +This document describes how you can install Nominatim inside a Ubuntu 14 +virtual machine on your desktop/laptop (host machine). The goal is to give +you a development environment to easily edit code and run the test suite +without affecting the rest of your system. + +The installation can run largely unsupervised. You should expect 1-2h from +start to finish depending on how fast your computer and download speed +is. + +## Prerequisites + +1. [Virtualbox](https://www.virtualbox.org/wiki/Downloads) + +2. [Vagrant](https://www.vagrantup.com/downloads.html) + +3. Nominatim + + git clone --recursive https://github.com/twain47/Nominatim.git + + If you haven't used `--recursive`, then you can load the submodules using + + git submodule init + git submodule update + + + +## Installation + +1. Start the virtual machine + + vagrant up + +2. Log into the virtual machine + + vagrant ssh + +3. Import a small country (Monaco) + + You need to give the virtual machine more memory (2GB) for an import, see `Vagrantfile`. + + See the FAQ how to skip this step and point Nominatim to an existing database. + + ``` + # inside the virtual machine: + cd Nominatim + wget --no-verbose --output-document=data/monaco.osm.pbf http://download.geofabrik.de/europe/monaco-latest.osm.pbf + utils/setup.php --osm-file data/monaco.osm.pbf --osm2pgsql-cache 1000 --all | tee monaco.$$.log + ./utils/specialphrases.php --countries > data/specialphrases_countries.sql + psql -d nominatim -f data/specialphrases_countries.sql + ``` + + To repeat an import you'd need to delete the database first + + dropdb --username postgres -if-exists nominatim + + + +## Development + +Vagrant maps the virtual machine's port 8089 to your host machine. Thus you can +see Nominatim in action on [locahost:8089](http://localhost:8089/nominatim/). + +You edit code on your host machine in any editor you like. There is no need to +restart any software: just refresh your browser window. + +PHP errors are written to `/var/log/apache2/error.log`. + +With `echo` and `var_dump()` you write into the output (HTML/XML/JSON) when +you either add `&debug=1` to the URL (preferred) or set +`@define('CONST_Debug', true);` in `settings/local.php`. + + + + +## Running functional tests + +Tests in `/features/db` and `/features/osm2pgsql` have to pass 100%. Other +tests might require full planet-wide data. Sadly even if you have your own +planet-wide data there will be enough differences to the openstreetmap.org +installation to cause false positives in the other tests (see FAQ). + +To run the full test suite + + cd ~/Nominatim/tests + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features + +To run a single file + + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce features/api/reverse.feature + +To run specific tests you can add tags just before the `Scenario line`, e.g. + + @bug-34 + Scenario: address lookup for non-existing or invalid node, way, relation + +and then + + NOMINATIM_SERVER=http://localhost:8089/nominatim lettuce -t bug-34 + + +## Running unit tests + + cd ~/Nominatim/tests + phpunit + + + + + + +## FAQ + +##### Will it run on Windows? + +Yes, Vagrant and Virtualbox can be installed on MS Windows just fine. You need a 64bit +version of Windows. + + +##### Why Monaco, can I use another country? + +Of course! The Monaco import takes less than 30 minutes and works with 2GB RAM. + +##### Will the results be the same as those from nominatim.openstreetmap.org? + +No. Long running Nominatim installations will differ once new import features (or +bug fixes) get added since those usually only get applied to new/changed data. + +Also this document skips the optional Wikipedia data import which affects ranking +of search results. See [Nominatim instllation](http://wiki.openstreetmap.org/wiki/Nominatim/Installation) for details. + +##### Why Ubuntu, can I test CentOS/CoreOS/FreeBSD? + +In general Nominatim will run in all these environment. The installation steps +are slightly different, e.g. the name of the package manager, Apache2 package +name, location of files. We chose Ubuntu because that is closest to the +nominatim.openstreetmap.org production environment. + +You can configure/download other Vagrant boxes from [vagrantbox.es](http://www.vagrantbox.es/). + + +##### How can I connect to an existing database? + +Let's say you have a Postgres database named `nominatim_it` on server `your-server.com` and port `5432`. The Postgres username is `postgres`. You can edit `settings/local.php` and point Nominatim to it. + + pgsql://postgres@your-server.com:5432/nominatim_it + +No data import necessary, no restarting necessary. + +If the Postgres installation is behind a firewall, you can try + + ssh -L 9999:localhost:5432 your-username@your-server.com + +inside the virtual machine. It will map the port to `localhost:9999` and then +you edit `settings/local.php` with + + pgsql://postgres@localhost:9999/nominatim_it + + +##### My computer is slow and the import takes too long. Can I start the virtual machine "in the cloud"? + +Yes. It's possible to start the virtual machine on [Amazon AWS (plugin)](https://github.com/mitchellh/vagrant-aws) or [DigitalOcean (plugin)](https://github.com/smdahlen/vagrant-digitalocean). + + + + diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 00000000..042148f7 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,49 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + + # Ubuntu LTS 14.04 + config.vm.box = "ubuntu/trusty64" + + # Apache webserver + config.vm.network "forwarded_port", guest: 8089, host: 8089 + + + # If true, then any SSH connections made will enable agent forwarding. + config.ssh.forward_agent = true + + config.vm.synced_folder ".", "/home/vagrant/Nominatim" + + + + # provision using a simple shell script + config.vm.provision :shell, :path => "vagrant-provision.sh" + + + # configure shared package cache if possible + if Vagrant.has_plugin?("vagrant-cachier") + config.cache.enable :apt + config.cache.scope = :box + end + + + config.vm.provider "virtualbox" do |vb| + vb.gui = false + vb.customize ["modifyvm", :id, "--memory", "1024"] + end + + + config.vm.provider :digital_ocean do |provider, override| + override.ssh.private_key_path = '~/.ssh/id_rsa' + override.vm.box = 'digital_ocean' + override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box" + + provider.token = 'e74c5c15501bd2782e689b0805ad577f954bfd39ea1fde3f17ff62835f2fc6c1' + # provider.token = 'YOUR TOKEN' + provider.image = 'ubuntu-14-04-x64' + provider.region = 'nyc2' + provider.size = '512mb' + end + +end diff --git a/vagrant-provision.sh b/vagrant-provision.sh new file mode 100755 index 00000000..aad75443 --- /dev/null +++ b/vagrant-provision.sh @@ -0,0 +1,166 @@ +#!/bin/bash + + +## During 'vagrant provision' this script runs as root and the current +## directory is '/root' +USERNAME=vagrant + +### +### maybe create ubuntu user +### + +# if [[ ! `id -u $USERNAME` ]]; then +# useradd $USERNAME --create-home --shell /bin/bash +# +# # give sudo power +# echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99-$USERNAME-user +# chmod 0440 /etc/sudoers.d/99-$USERNAME-user +# service sudo restart +# +# # add basic .profile +# cp -r .ssh .profile .bashrc /home/$USERNAME/ +# chown -R $USERNAME /home/$USERNAME/.* +# chgrp -R $USERNAME /home/$USERNAME/.* +# +# # now ideally login as $USERNAME and continue +# su $USERNAME -l +# fi + + +sudo apt-get update -qq +sudo apt-get upgrade -y +# sudo apt-get install -y git-core screen +sudo apt-get install -y build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev \ + libtool automake libproj-dev libboost-dev libboost-system-dev \ + libboost-filesystem-dev libboost-thread-dev +sudo apt-get autoremove -y + +# get arrow-keys working in terminal (e.g. editing in vi) +echo 'stty sane' >> ~/.bash_profile +echo 'export TERM=linux' >> ~/.bash_profile +source ~/.bash_profile + + +### +### PostgreSQL 9.3 + PostGIS 2.1 +### + +sudo apt-get install -y postgresql-9.3-postgis-2.1 postgresql-contrib-9.3 postgresql-server-dev-9.3 +# already included: proj-bin libgeos-dev + +# make sure OS-authenticated users (e.g. $USERNAME) can access +sudo sed -i "s/ident/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo sed -i "s/md5/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo sed -i "s/peer/trust/" /etc/postgresql/9.3/main/pg_hba.conf +sudo /etc/init.d/postgresql restart + +# creates the role +sudo -u postgres createuser -s $USERNAME + + + +### +### PHP for frontend +### +sudo apt-get install -y php5 php5-pgsql php-pear +sudo pear install DB + + +# get rid of some warning +# where is the ini file? 'php --ini' +echo "date.timezone = 'Etc/UTC'" | sudo tee /etc/php5/cli/conf.d/99-timezone.ini > /dev/null + + + +### +### Nominatim +### +sudo apt-get install -y libprotobuf-c0-dev protobuf-c-compiler \ + libgeos-c1 libgeos++-dev \ + lua5.2 liblua5.2-dev + +# git clone --recursive https://github.com/twain47/Nominatim.git + + +# now ideally login as $USERNAME and continue +su $USERNAME -l +pwd +ls -la /home/vagrant +cd /home/vagrant/Nominatim + +# cd ~/Nominatim +./autogen.sh +./configure +make +chmod +x ./ +chmod +x ./module + + +# IP=`curl -s http://bot.whatismyipaddress.com` +IP=localhost +echo " settings/local.php + + + + + + + +### +### Setup Apache/website +### + +createuser -SDR www-data + +echo ' +Listen 8089 + + # DirectoryIndex index.html + # ErrorDocument 403 /index.html + + DocumentRoot "/var/www/" + + + Options FollowSymLinks MultiViews + AddType text/html .php + + +' | sudo tee /etc/apache2/sites-enabled/nominatim.conf > /dev/null + + +sudo apache2ctl graceful + + +sudo mkdir -m 755 /var/www/nominatim +sudo chown $USERNAME /var/www/nominatim +./utils/setup.php --threads 1 --create-website /var/www/nominatim + + +# if you get 'permission denied for relation word', then try +# GRANT usage ON SCHEMA public TO "www-data"; +# GRANT SELECT ON ALL TABLES IN SCHEMA public TO "www-data"; + +## +## Test suite (Python) +## https://github.com/twain47/Nominatim/tree/master/tests +## +sudo apt-get install -y python-dev python-pip python-Levenshtein tidy +sudo pip install lettuce nose pytidylib haversine psycopg2 shapely + +## +## Test suite (PHP) +## https://github.com/twain47/Nominatim/tree/master/tests-php +## +wget --no-clobber -q https://phar.phpunit.de/phpunit.phar +chmod +x phpunit.phar +sudo mv phpunit.phar /usr/local/bin/phpunit + + -- 2.45.1