From: Sarah Hoffmann Date: Wed, 13 Nov 2019 09:30:22 +0000 (+0100) Subject: Merge pull request #1557 from mtmail/document-boundingbox X-Git-Tag: v3.5.0~133 X-Git-Url: https://git.openstreetmap.org/nominatim.git/commitdiff_plain/6f2e767c77a7efbc326fdb69287d72f062cec5b0?hp=db6da756832798e02a7a21afab43d2b2562bb947 Merge pull request #1557 from mtmail/document-boundingbox documentation: add note what bounding box can be used for --- diff --git a/docs/admin/Faq.md b/docs/admin/Faq.md index 485ba25b..2e5a7c43 100644 --- a/docs/admin/Faq.md +++ b/docs/admin/Faq.md @@ -26,14 +26,16 @@ If the reported rank is 26 or higher, you can also safely add `--index-noanalyse PHP Warning: file_get_contents(): open_basedir restriction in effect. -You need to adjust the [open_basedir](https://www.php.net/manual/en/ini.core.php#ini.open-basedir) setting -in your PHP configuration (`php.ini file`). By default this setting may look like this: +You need to adjust the +[open_basedir](https://www.php.net/manual/en/ini.core.php#ini.open-basedir) +setting in your PHP configuration (`php.ini` file). By default this setting may +look like this: open_basedir = /srv/http/:/home/:/tmp/:/usr/share/pear/ -Either add reported directories to the list or disable this setting temporarily by -dding ";" at the beginning of the line. Don't forget to enable this setting again -once you are done with the PHP command line operations. +Either add reported directories to the list or disable this setting temporarily +by adding ";" at the beginning of the line. Don't forget to enable this setting +again once you are done with the PHP command line operations. ### PHP timzeone warnings @@ -107,10 +109,11 @@ to get the full error message. `could not connect to server: No such file or directory` -On CentOS v7 the PostgreSQL server is started with `systemd`. -Check if `/usr/lib/systemd/system/httpd.service` contains a line `PrivateTmp=true`. -If so then Apache cannot see the `/tmp/.s.PGSQL.5432` file. It's a good security feature, -so use the [preferred solution](../appendix/Install-on-Centos-7/#adding-selinux-security-settings). +On CentOS v7 the PostgreSQL server is started with `systemd`. Check if +`/usr/lib/systemd/system/httpd.service` contains a line `PrivateTmp=true`. If +so then Apache cannot see the `/tmp/.s.PGSQL.5432` file. It's a good security +feature, so use the +[preferred solution](../appendix/Install-on-Centos-7/#adding-selinux-security-settings). However, you can solve this the quick and dirty way by commenting out that line and then run @@ -118,14 +121,12 @@ However, you can solve this the quick and dirty way by commenting out that line sudo systemctl restart httpd -### "must be an array or an object that implements Countable" warning in /usr/share/pear/DB.php - -The warning started with PHP 7.2. Make sure you have at least [version 1.9.3 of PEAR DB](https://github.com/pear/DB/releases) -installed. - ### Website reports "DB Error: insufficient permissions" -The user the webserver, e.g. Apache, runs under needs to have access to the Nominatim database. You can find the user like [this](https://serverfault.com/questions/125865/finding-out-what-user-apache-is-running-as), for default Ubuntu operating system for example it's `www-data`. +The user the webserver, e.g. Apache, runs under needs to have access to the +Nominatim database. You can find the user like +[this](https://serverfault.com/questions/125865/finding-out-what-user-apache-is-running-as), +for default Ubuntu operating system for example it's `www-data`. 1. Repeat the `createuser` step of the installation instructions. @@ -164,18 +165,8 @@ When running SELinux, make sure that the ### Setup.php fails with "DB Error: extension not found" Make sure you have the PostgreSQL extensions "hstore" and "postgis" installed. -See the installation instruction for a full list of required packages. - - -### Setup.php reports "Cannot redeclare getDB()" - -`Cannot redeclare getDB() (previously declared in /your/path/Nominatim/lib/db.php:4)` - -The message is a bit misleading as PHP needs to load the file `DB.php` and -instead re-loads Nominatim's `db.php`. To solve this make sure you -have the [Pear module 'DB'](https://pear.php.net/package/DB/) installed. +See the installation instructions for a full list of required packages. - sudo pear install DB ### I forgot to delete the flatnodes file before starting an import. diff --git a/docs/develop/Documentation.md b/docs/develop/Documentation.md index 6e792c25..df8d2b1a 100644 --- a/docs/develop/Documentation.md +++ b/docs/develop/Documentation.md @@ -1,36 +1,36 @@ # Documentation Pages -The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/]() +The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/) -To preview local changes: +To preview local changes, first install MkDocs -1. Install MkDocs +``` +pip3 install --user mkdocs +``` - ``` - pip3 install --user mkdocs - ``` +Then go to the build directory and run -2. In build directory run +``` +make doc +INFO - Cleaning site directory +INFO - Building documentation to directory: /home/vagrant/build/site-html +``` - ``` - make doc - INFO - Cleaning site directory - INFO - Building documentation to directory: /home/vagrant/build/site-html - ``` +This runs `mkdocs build` plus extra transformation of some files and adds +symlinks (see `CMakeLists.txt` for the exact steps). - This runs `mkdocs build` plus extra transformion of some files and adds symlinks (see `CMakeLists.txt` for the exact steps). +Now you can start webserver for local testing +``` +build> mkdocs serve +[server:296] Serving on http://127.0.0.1:8000 +[handlers:62] Start watching changes +``` -3. Start webserver for local testing +If you develop inside a Vagrant virtual machine: - ``` - mkdocs serve - [server:296] Serving on http://127.0.0.1:8000 - [handlers:62] Start watching changes - ``` - - If you develop inside a Vagrant virtual machine: - * add port forwarding to your Vagrantfile, e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000` - * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost - IP does not get forwarded. + * add port forwarding to your Vagrantfile, + e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000` + * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost + IP does not get forwarded. diff --git a/docs/develop/Postcodes.md b/docs/develop/Postcodes.md new file mode 100644 index 00000000..ff36b0dd --- /dev/null +++ b/docs/develop/Postcodes.md @@ -0,0 +1,45 @@ +# Postcodes in Nominatim + +The blog post +[Nominatim and Postcodes](https://www.openstreetmap.org/user/lonvia/diary/43143) +describes the handling implemented since Nominatim 3.1. + +Postcode centroids (aka 'calculated postcodes') are generated by looking at all +postcodes of a country, grouping them and calculating the geometric centroid. +There is currently no logic to deal with extreme outliers (typos or other +mistakes in OSM data). There is also no check if a postcodes adheres to a +country's format, e.g. if Swiss postcodes are 4 digits. + + +## Regular updating calculated postcodes + +The script to rerun the calculation is +`build/utils/update.php --calculate-postcodes` +and runs once per night on nominatim.openstreetmap.org. + + +## Finding places that share a specific postcode + +In the Nominatim database run + +```sql +SELECT address->'postcode' as pc, + osm_type, osm_id, class, type, + st_x(centroid) as lon, st_y(centroid) as lat +FROM placex +WHERE country_code='fr' + AND upper(trim (both ' ' from address->'postcode')) = '33210'; +``` + +Alternatively on [Overpass](https://overpass-turbo.eu/) run the following query + +``` +[out:json][timeout:250]; +area["name"="France"]->.boundaryarea; +( +nwr(area.boundaryarea)["addr:postcode"="33210"]; +); +out body; +>; +out skel qt; +``` diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index cb00e2e8..5cf63f83 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -22,6 +22,7 @@ pages: - 'Overview' : 'develop/overview.md' - 'OSM Data Import' : 'develop/Import.md' - 'Place Ranking' : 'develop/Ranking.md' + - 'Postcodes' : 'develop/Postcodes.md' - 'Documentation' : 'develop/Documentation.md' - 'External Data Sources': - 'Overview' : 'data-sources/overview.md' diff --git a/lib/DB.php b/lib/DB.php index fe2529b2..e4aa4349 100644 --- a/lib/DB.php +++ b/lib/DB.php @@ -284,7 +284,7 @@ class DB { // https://secure.php.net/manual/en/ref.pdo-pgsql.connection.php $aInfo = array(); - if (preg_match('/^pgsql:(.+)/', $sDSN, $aMatches)) { + if (preg_match('/^pgsql:(.+)$/', $sDSN, $aMatches)) { foreach (explode(';', $aMatches[1]) as $sKeyVal) { list($sKey, $sVal) = explode('=', $sKeyVal, 2); if ($sKey == 'host') $sKey = 'hostspec'; diff --git a/lib/cmd.php b/lib/cmd.php index 32fdc857..77878c15 100644 --- a/lib/cmd.php +++ b/lib/cmd.php @@ -148,12 +148,14 @@ function runSQLScript($sScript, $bfatal = true, $bVerbose = false, $bIgnoreError // Convert database DSN to psql parameters $aDSNInfo = \Nominatim\DB::parseDSN(CONST_Database_DSN); if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432; - $sCMD = 'psql -p '.$aDSNInfo['port'].' -d '.$aDSNInfo['database']; + $sCMD = 'psql' + .' -p '.escapeshellarg($aDSNInfo['port']) + .' -d '.escapeshellarg($aDSNInfo['database']); if (isset($aDSNInfo['hostspec']) && $aDSNInfo['hostspec']) { - $sCMD .= ' -h ' . $aDSNInfo['hostspec']; + $sCMD .= ' -h ' . escapeshellarg($aDSNInfo['hostspec']); } if (isset($aDSNInfo['username']) && $aDSNInfo['username']) { - $sCMD .= ' -U ' . $aDSNInfo['username']; + $sCMD .= ' -U ' . escapeshellarg($aDSNInfo['username']); } $aProcEnv = null; if (isset($aDSNInfo['password']) && $aDSNInfo['password']) { diff --git a/lib/setup/SetupClass.php b/lib/setup/SetupClass.php index a26b7dae..818aeeb7 100755 --- a/lib/setup/SetupClass.php +++ b/lib/setup/SetupClass.php @@ -80,13 +80,15 @@ class SetupFunctions fail('database already exists ('.CONST_Database_DSN.')'); } - $sCreateDBCmd = 'createdb -E UTF-8 -p '.$this->aDSNInfo['port'].' '.$this->aDSNInfo['database']; + $sCreateDBCmd = 'createdb -E UTF-8' + .' -p '.escapeshellarg($this->aDSNInfo['port']) + .' '.escapeshellarg($this->aDSNInfo['database']); if (isset($this->aDSNInfo['username'])) { - $sCreateDBCmd .= ' -U '.$this->aDSNInfo['username']; + $sCreateDBCmd .= ' -U '.escapeshellarg($this->aDSNInfo['username']); } if (isset($this->aDSNInfo['hostspec'])) { - $sCreateDBCmd .= ' -h '.$this->aDSNInfo['hostspec']; + $sCreateDBCmd .= ' -h '.escapeshellarg($this->aDSNInfo['hostspec']); } $result = $this->runWithPgEnv($sCreateDBCmd); @@ -178,30 +180,30 @@ class SetupFunctions fail("osm2pgsql not found in '$osm2pgsql'"); } - $osm2pgsql .= ' -S '.CONST_Import_Style; + $osm2pgsql .= ' -S '.escapeshellarg(CONST_Import_Style); if (!is_null(CONST_Osm2pgsql_Flatnode_File) && CONST_Osm2pgsql_Flatnode_File) { - $osm2pgsql .= ' --flat-nodes '.CONST_Osm2pgsql_Flatnode_File; + $osm2pgsql .= ' --flat-nodes '.escapeshellarg(CONST_Osm2pgsql_Flatnode_File); } if (CONST_Tablespace_Osm2pgsql_Data) - $osm2pgsql .= ' --tablespace-slim-data '.CONST_Tablespace_Osm2pgsql_Data; + $osm2pgsql .= ' --tablespace-slim-data '.escapeshellarg(CONST_Tablespace_Osm2pgsql_Data); if (CONST_Tablespace_Osm2pgsql_Index) - $osm2pgsql .= ' --tablespace-slim-index '.CONST_Tablespace_Osm2pgsql_Index; + $osm2pgsql .= ' --tablespace-slim-index '.escapeshellarg(CONST_Tablespace_Osm2pgsql_Index); if (CONST_Tablespace_Place_Data) - $osm2pgsql .= ' --tablespace-main-data '.CONST_Tablespace_Place_Data; + $osm2pgsql .= ' --tablespace-main-data '.escapeshellarg(CONST_Tablespace_Place_Data); if (CONST_Tablespace_Place_Index) - $osm2pgsql .= ' --tablespace-main-index '.CONST_Tablespace_Place_Index; + $osm2pgsql .= ' --tablespace-main-index '.escapeshellarg(CONST_Tablespace_Place_Index); $osm2pgsql .= ' -lsc -O gazetteer --hstore --number-processes 1'; - $osm2pgsql .= ' -C '.$this->iCacheMemory; - $osm2pgsql .= ' -P '.$this->aDSNInfo['port']; + $osm2pgsql .= ' -C '.escapeshellarg($this->iCacheMemory); + $osm2pgsql .= ' -P '.escapeshellarg($this->aDSNInfo['port']); if (isset($this->aDSNInfo['username'])) { - $osm2pgsql .= ' -U '.$this->aDSNInfo['username']; + $osm2pgsql .= ' -U '.escapeshellarg($this->aDSNInfo['username']); } if (isset($this->aDSNInfo['hostspec'])) { - $osm2pgsql .= ' -H '.$this->aDSNInfo['hostspec']; + $osm2pgsql .= ' -H '.escapeshellarg($this->aDSNInfo['hostspec']); } - $osm2pgsql .= ' -d '.$this->aDSNInfo['database'].' '.$sOSMFile; + $osm2pgsql .= ' -d '.escapeshellarg($this->aDSNInfo['database']).' '.escapeshellarg($sOSMFile); $this->runWithPgEnv($osm2pgsql); @@ -599,13 +601,15 @@ class SetupFunctions public function index($bIndexNoanalyse) { $sOutputFile = ''; - $sBaseCmd = CONST_InstallPath.'/nominatim/nominatim -i -d '.$this->aDSNInfo['database'].' -P ' - .$this->aDSNInfo['port'].' -t '.$this->iInstances.$sOutputFile; + $sBaseCmd = CONST_InstallPath.'/nominatim/nominatim -i' + .' -d '.escapeshellarg($this->aDSNInfo['database']) + .' -P '.escapeshellarg($this->aDSNInfo['port']) + .' -t '.escapeshellarg($this->iInstances.$sOutputFile); if (isset($this->aDSNInfo['hostspec'])) { - $sBaseCmd .= ' -H '.$this->aDSNInfo['hostspec']; + $sBaseCmd .= ' -H '.escapeshellarg($this->aDSNInfo['hostspec']); } if (isset($this->aDSNInfo['username'])) { - $sBaseCmd .= ' -U '.$this->aDSNInfo['username']; + $sBaseCmd .= ' -U '.escapeshellarg($this->aDSNInfo['username']); } info('Index ranks 0 - 4'); @@ -742,15 +746,18 @@ class SetupFunctions private function pgsqlRunDropAndRestore($sDumpFile) { - $sCMD = 'pg_restore -p '.$this->aDSNInfo['port'].' -d '.$this->aDSNInfo['database'].' --no-owner -Fc --clean '.$sDumpFile; + $sCMD = 'pg_restore' + .' -p '.escapeshellarg($this->aDSNInfo['port']) + .' -d '.escapeshellarg($this->aDSNInfo['database']) + .' --no-owner -Fc --clean '.escapeshellarg($sDumpFile); if ($this->oDB->getPostgresVersion() >= 9.04) { $sCMD .= ' --if-exists'; } if (isset($this->aDSNInfo['hostspec'])) { - $sCMD .= ' -h '.$this->aDSNInfo['hostspec']; + $sCMD .= ' -h '.escapeshellarg($this->aDSNInfo['hostspec']); } if (isset($this->aDSNInfo['username'])) { - $sCMD .= ' -U '.$this->aDSNInfo['username']; + $sCMD .= ' -U '.escapeshellarg($this->aDSNInfo['username']); } $this->runWithPgEnv($sCMD); @@ -814,15 +821,17 @@ class SetupFunctions { if (!file_exists($sFilename)) fail('unable to find '.$sFilename); - $sCMD = 'psql -p '.$this->aDSNInfo['port'].' -d '.$this->aDSNInfo['database']; + $sCMD = 'psql' + .' -p '.escapeshellarg($this->aDSNInfo['port']) + .' -d '.escapeshellarg($this->aDSNInfo['database']); if (!$this->bVerbose) { $sCMD .= ' -q'; } if (isset($this->aDSNInfo['hostspec'])) { - $sCMD .= ' -h '.$this->aDSNInfo['hostspec']; + $sCMD .= ' -h '.escapeshellarg($this->aDSNInfo['hostspec']); } if (isset($this->aDSNInfo['username'])) { - $sCMD .= ' -U '.$this->aDSNInfo['username']; + $sCMD .= ' -U '.escapeshellarg($this->aDSNInfo['username']); } $aProcEnv = null; if (isset($this->aDSNInfo['password'])) { @@ -835,12 +844,12 @@ class SetupFunctions 1 => array('pipe', 'w'), 2 => array('file', '/dev/null', 'a') ); - $hGzipProcess = proc_open('zcat '.$sFilename, $aDescriptors, $ahGzipPipes); + $hGzipProcess = proc_open('zcat '.escapeshellarg($sFilename), $aDescriptors, $ahGzipPipes); if (!is_resource($hGzipProcess)) fail('unable to start zcat'); $aReadPipe = $ahGzipPipes[1]; fclose($ahGzipPipes[0]); } else { - $sCMD .= ' -f '.$sFilename; + $sCMD .= ' -f '.escapeshellarg($sFilename); $aReadPipe = array('pipe', 'r'); } $aDescriptors = array( diff --git a/vagrant/Install-on-Centos-7.sh b/vagrant/Install-on-Centos-7.sh index 19b7ff84..b456fb74 100755 --- a/vagrant/Install-on-Centos-7.sh +++ b/vagrant/Install-on-Centos-7.sh @@ -17,12 +17,20 @@ sudo yum install -y epel-release +# More repositories for postgresql 11 (CentOS default 'postgresql' is 9.2), postgis +# and llvm-toolset (https://github.com/theory/pg-semver/issues/35) + + sudo yum install -y https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm + sudo yum install -y centos-release-scl-rh + # Now you can install all packages needed for Nominatim: #DOCS: :::sh - sudo yum install -y postgresql-server postgresql-contrib postgresql-devel \ - postgis postgis-utils \ + + sudo yum install -y postgresql11-server postgresql11-contrib postgresql11-devel \ + postgis25_11 postgis25_11-utils \ wget git cmake make gcc gcc-c++ libtool policycoreutils-python \ + devtoolset-7 llvm-toolset-7 \ php-pgsql php php-intl libpqxx-devel \ proj-epsg bzip2-devel proj-devel libxml2-devel boost-devel \ expat-devel zlib-devel @@ -77,8 +85,11 @@ sudo chown vagrant /srv/nominatim #DOCS: # CentOS does not automatically create a database cluster. Therefore, start # with initializing the database, then enable the server to start at boot: - sudo postgresql-setup initdb - sudo systemctl enable postgresql + echo 'PATH=$PATH:/usr/pgsql-11/bin' > .bash_profile + source .bash_profile + + sudo /usr/pgsql-11/bin/postgresql-11-setup initdb + sudo systemctl enable postgresql-11 # # Next tune the postgresql configuration, which is located in @@ -88,7 +99,7 @@ sudo chown vagrant /srv/nominatim #DOCS: # # Now start the postgresql service after updating this config file. - sudo systemctl restart postgresql + sudo systemctl restart postgresql-11 # # Finally, we need to add two postgres users: one for the user that does @@ -150,7 +161,7 @@ fi #DOCS: # download the country grid: if [ ! -f data/country_osm_grid.sql.gz ]; then #DOCS: :::sh - wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz + wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz fi #DOCS: # The code must be built in a separate directory. Create this directory,