]> git.openstreetmap.org Git - nominatim.git/log
nominatim.git
3 years agoactions: remove install directories before import
Sarah Hoffmann [Wed, 10 Feb 2021 15:35:50 +0000 (16:35 +0100)]
actions: remove install directories before import

This ensures that any dangling references to the build
or source directory are caught by the CI.

3 years agoneed test database for analysing cli test
Sarah Hoffmann [Wed, 10 Feb 2021 15:19:51 +0000 (16:19 +0100)]
need test database for analysing cli test

3 years agoactions: test import with installed version of Nominatim
Sarah Hoffmann [Wed, 10 Feb 2021 15:02:14 +0000 (16:02 +0100)]
actions: test import with installed version of Nominatim

3 years agoadd 'make install' to installation instructions
Sarah Hoffmann [Wed, 10 Feb 2021 10:15:21 +0000 (11:15 +0100)]
add 'make install' to installation instructions

3 years agotest for existance of country grid in cmake already
Sarah Hoffmann [Wed, 10 Feb 2021 09:40:36 +0000 (10:40 +0100)]
test for existance of country grid in cmake already

Given that the file potentially gets installed, it needs to be
present during build time already. Checking during the import
is too late.

3 years agomake installation targets conditional to what is built
Sarah Hoffmann [Wed, 10 Feb 2021 09:04:07 +0000 (10:04 +0100)]
make installation targets conditional to what is built

3 years agoadd make install target
Sarah Hoffmann [Tue, 9 Feb 2021 20:04:42 +0000 (21:04 +0100)]
add make install target

Installation includes PHP andPython libraries, settings, the basic
country data, the postgresql module and our custom version of
osm2pgsql. The latter is installed in our private library directory
so that it does not get in the way of a potentially installed
osm2pgsql from the distribution.

3 years agoadapt unit tests to new directory structure
Sarah Hoffmann [Tue, 9 Feb 2021 19:13:00 +0000 (20:13 +0100)]
adapt unit tests to new directory structure

3 years agouse DataDir constant for data only
Sarah Hoffmann [Tue, 9 Feb 2021 19:04:08 +0000 (20:04 +0100)]
use DataDir constant for data only

So far the data directory constant has pointed to the source
directory to be usable with different subdirectories. Now only
the data subdirectory itself is being used with the constant,
so point to the directory directly.

3 years agoforgot to replace one occurence of sql_dir
Sarah Hoffmann [Tue, 9 Feb 2021 18:32:05 +0000 (19:32 +0100)]
forgot to replace one occurence of sql_dir

3 years agoremove unused BINDIR constant
Sarah Hoffmann [Tue, 9 Feb 2021 18:30:31 +0000 (19:30 +0100)]
remove unused BINDIR constant

3 years agointroduce constant for configuration directory
Sarah Hoffmann [Tue, 9 Feb 2021 17:45:45 +0000 (18:45 +0100)]
introduce constant for configuration directory

This replaces {data_dir}/settings throughout the code, so that
the configuration may be placed somewhere else in the directory
structure (e.g. in /etc).

3 years agorename sql directory to lib-sql
Sarah Hoffmann [Tue, 9 Feb 2021 14:26:56 +0000 (15:26 +0100)]
rename sql directory to lib-sql

Also introduces a separate constant for the sql directory, so that
it can be put separately from the rest of the data if required.

3 years agorename lib to lib-php
Sarah Hoffmann [Tue, 9 Feb 2021 10:52:07 +0000 (11:52 +0100)]
rename lib to lib-php

3 years agoupdate osm2pgsql (disable install rule)
Sarah Hoffmann [Tue, 9 Feb 2021 08:48:50 +0000 (09:48 +0100)]
update osm2pgsql (disable install rule)

3 years agointegrate analyse of indexing into nominatim tool
Sarah Hoffmann [Mon, 8 Feb 2021 21:21:57 +0000 (22:21 +0100)]
integrate analyse of indexing into nominatim tool

3 years agoconsolidate warm and db-check into single admin command
Sarah Hoffmann [Mon, 8 Feb 2021 20:05:06 +0000 (21:05 +0100)]
consolidate warm and db-check into single admin command

3 years agosplit cli.py by subcommands
Sarah Hoffmann [Mon, 8 Feb 2021 16:23:05 +0000 (17:23 +0100)]
split cli.py by subcommands

Reduces file size below 1000 lines.

3 years agomove website into php library directory
Sarah Hoffmann [Mon, 8 Feb 2021 11:00:34 +0000 (12:00 +0100)]
move website into php library directory

3 years agomove postcode table setup to sql/
Sarah Hoffmann [Mon, 8 Feb 2021 10:48:45 +0000 (11:48 +0100)]
move postcode table setup to sql/

Also moves the call to the setup from the setup-db
step to the calculate-postcodes step. The tables also need
no longer be accessible by the webservice.

3 years agoremove unused SQL script
Sarah Hoffmann [Mon, 8 Feb 2021 10:28:24 +0000 (11:28 +0100)]
remove unused SQL script

3 years agofix off-by-one error in replication download
Sarah Hoffmann [Thu, 4 Feb 2021 16:04:04 +0000 (17:04 +0100)]
fix off-by-one error in replication download

3 years agoMerge pull request #2161 from lonvia/timeout-for-replication
Sarah Hoffmann [Thu, 4 Feb 2021 15:52:24 +0000 (16:52 +0100)]
Merge pull request #2161 from lonvia/timeout-for-replication

Reintroduce timeout for replication file download

3 years agoreintroduce timeout for replication file download
Sarah Hoffmann [Thu, 4 Feb 2021 10:47:11 +0000 (11:47 +0100)]
reintroduce timeout for replication file download

This ports the --socket-timeout parameter from
pyosmium-get-changes which ensures that the update
process eventually times out on hanging network connections.

3 years agoMerge pull request #2160 from lonvia/introduce-project-dir
Sarah Hoffmann [Thu, 4 Feb 2021 08:52:59 +0000 (09:52 +0100)]
Merge pull request #2160 from lonvia/introduce-project-dir

Officially introduce and recommend use of a project directory

3 years agoreplace make serve with nominatim serve command
Sarah Hoffmann [Wed, 3 Feb 2021 15:17:46 +0000 (16:17 +0100)]
replace make serve with nominatim serve command

With the website directory now tied to the project directory instead
of the build directory, it is no longer possible to use make for
running the web server.

3 years agoadd migration info for status table
Sarah Hoffmann [Wed, 3 Feb 2021 13:13:09 +0000 (14:13 +0100)]
add migration info for status table

3 years agoRevert "increase splitting for large geometries"
Sarah Hoffmann [Wed, 3 Feb 2021 09:23:38 +0000 (10:23 +0100)]
Revert "increase splitting for large geometries"

This reverts commit 559fe513fa8ffa3a5bc3a7d07c252a872614c073.

Increasing the splitting results in geometries where with rounding
issues at the split points, so that contain operations do not
work as expected anymore.

Fixes #2137.

3 years agoadapt quick start instructions in README to project dir
Sarah Hoffmann [Wed, 3 Feb 2021 09:17:22 +0000 (10:17 +0100)]
adapt quick start instructions in README to project dir

3 years agodo not overwrite custom set module paths
Sarah Hoffmann [Tue, 2 Feb 2021 16:33:41 +0000 (17:33 +0100)]
do not overwrite custom set module paths

Given that the module is now copied to the project directory
when no module path is set, we need the information that the
module path is empty. Therefore hand in the default module path
in a separate variable.

3 years agoprint project directory before running any command
Sarah Hoffmann [Tue, 2 Feb 2021 10:19:31 +0000 (11:19 +0100)]
print project directory before running any command

3 years agomake phpcs happy
Sarah Hoffmann [Tue, 2 Feb 2021 10:15:56 +0000 (11:15 +0100)]
make phpcs happy

3 years agoactions: switch to using separate project dir
Sarah Hoffmann [Tue, 2 Feb 2021 10:03:09 +0000 (11:03 +0100)]
actions: switch to using separate project dir

Also fixes reverse-only import which not run at all.

3 years agodocs: add hint about putting the nominatim tool into the PATH
Sarah Hoffmann [Tue, 2 Feb 2021 09:51:03 +0000 (10:51 +0100)]
docs: add hint about putting the nominatim tool into the PATH

3 years agocopy database module on install
Sarah Hoffmann [Tue, 2 Feb 2021 09:20:05 +0000 (10:20 +0100)]
copy database module on install

When no explicity database module is configured, then the
module is now copied into the project directory and used from
there. This means that Nominatim can be updated to a new
version of the module while existing installation keep their
version of normalisation.

3 years agohand in absolute path to nominatim tool to php scripts
Sarah Hoffmann [Mon, 1 Feb 2021 16:44:13 +0000 (17:44 +0100)]
hand in absolute path to nominatim tool to php scripts

3 years agorefer to new nominatim tool in configuration comments
Sarah Hoffmann [Mon, 1 Feb 2021 16:27:29 +0000 (17:27 +0100)]
refer to new nominatim tool in configuration comments

3 years agochange the default location for external data to project dir
Sarah Hoffmann [Mon, 1 Feb 2021 16:27:01 +0000 (17:27 +0100)]
change the default location for external data to project dir

3 years agoactions: revert to reletive paths for caching
Sarah Hoffmann [Tue, 2 Feb 2021 09:37:18 +0000 (10:37 +0100)]
actions: revert to reletive paths for caching

3 years agoactions: remove setting custom .env
Sarah Hoffmann [Tue, 2 Feb 2021 09:35:30 +0000 (10:35 +0100)]
actions: remove setting custom .env

It only set the pyosmium-get-changes binary which is no longer
needed.

3 years agoremove pyosmium-get-changes detection from cmake
Sarah Hoffmann [Tue, 2 Feb 2021 09:33:15 +0000 (10:33 +0100)]
remove pyosmium-get-changes detection from cmake

pyosmium-get-changes is not longer used.

3 years agoMerge pull request #2155 from lonvia/port-regresh-to-python
Sarah Hoffmann [Mon, 1 Feb 2021 10:50:05 +0000 (11:50 +0100)]
Merge pull request #2155 from lonvia/port-regresh-to-python

Port replication and part of the refrsh function to native Python

3 years agoprint nice summary after updates
Sarah Hoffmann [Sat, 30 Jan 2021 19:32:46 +0000 (20:32 +0100)]
print nice summary after updates

3 years agofix linting issues
Sarah Hoffmann [Sat, 30 Jan 2021 15:42:25 +0000 (16:42 +0100)]
fix linting issues

3 years agodisable warning about non-toplevel import
Sarah Hoffmann [Sat, 30 Jan 2021 15:27:02 +0000 (16:27 +0100)]
disable warning about non-toplevel import

They are needed here so nominatim can be run when osmium
is not installed. Everything except replication will work fine.

3 years agointroduce custom UsageError
Sarah Hoffmann [Sat, 30 Jan 2021 15:20:10 +0000 (16:20 +0100)]
introduce custom UsageError

This is a exception to be thrown when the error occures because
of bad user data. We don't want to print a full stack trace in
these cases but just tell the user what went wrong.

3 years agoremove setting for PYOSMIUM_BINARY
Sarah Hoffmann [Sat, 30 Jan 2021 14:55:04 +0000 (15:55 +0100)]
remove setting for PYOSMIUM_BINARY

pyosmium is now called as a library from the python code,
so that pyosmium-get-changes is no longer needed.

3 years agowhitelist pyosmium for pylint
Sarah Hoffmann [Sat, 30 Jan 2021 14:52:49 +0000 (15:52 +0100)]
whitelist pyosmium for pylint

3 years agoport replication update function to python
Sarah Hoffmann [Sat, 30 Jan 2021 14:50:34 +0000 (15:50 +0100)]
port replication update function to python

3 years agoport check-for-update function to python
Sarah Hoffmann [Thu, 28 Jan 2021 13:34:17 +0000 (14:34 +0100)]
port check-for-update function to python

3 years agoMerge pull request #2153 from rizkyarlin/patch-1
Sarah Hoffmann [Thu, 28 Jan 2021 08:06:33 +0000 (09:06 +0100)]
Merge pull request #2153 from rizkyarlin/patch-1

fix indentation

3 years agofix indentation
Muh. Rizky Eka Arlin [Thu, 28 Jan 2021 06:21:54 +0000 (14:21 +0800)]
fix indentation

3 years agoport replication initialisation to Python
Sarah Hoffmann [Tue, 26 Jan 2021 21:45:24 +0000 (22:45 +0100)]
port replication initialisation to Python

3 years agoconvert functon creation to python
Sarah Hoffmann [Sun, 24 Jan 2021 13:35:35 +0000 (14:35 +0100)]
convert functon creation to python

The new functions always creates normal and partitioned functions.
Also adds specialised connection and cursor classes for adding
frequently used helper functions.

3 years agoport address level computation to Python
Sarah Hoffmann [Sat, 23 Jan 2021 16:25:14 +0000 (17:25 +0100)]
port address level computation to Python

Also adds simple tests for correct table creation.

3 years agomove update code for postcode and word count to Python
Sarah Hoffmann [Fri, 22 Jan 2021 22:25:37 +0000 (23:25 +0100)]
move update code for postcode and word count to Python

Adds also tests for the new function to execute a SQL script.

3 years agocli: import python modules for commands on demand
Sarah Hoffmann [Thu, 21 Jan 2021 09:19:38 +0000 (10:19 +0100)]
cli: import python modules for commands on demand

Given that only one command will be executed in the end, it is
not necessary to import what amounts to the whole library. This
becomes in particular important for update functions that have
a dependency on pyosmium. The dependency can remain optional for
people not using updates.

3 years agoremove API comparison util
Sarah Hoffmann [Tue, 26 Jan 2021 21:46:35 +0000 (22:46 +0100)]
remove API comparison util

This is outdated and unmaintained. There are tools out there
which can do this better. Try, for example
https://github.com/radarlabs/api-diff

3 years agoMerge pull request #2147 from lonvia/tests-for-python-code
Sarah Hoffmann [Thu, 21 Jan 2021 09:07:50 +0000 (10:07 +0100)]
Merge pull request #2147 from lonvia/tests-for-python-code

Add basic set of tests for Python code

3 years agocli indexer tests need a fake database
Sarah Hoffmann [Wed, 20 Jan 2021 19:24:39 +0000 (20:24 +0100)]
cli indexer tests need a fake database

The Indexer constructor opens a connection to the given database.

3 years agoadd more tests for CLI parameter parser
Sarah Hoffmann [Wed, 20 Jan 2021 15:43:06 +0000 (16:43 +0100)]
add more tests for CLI parameter parser

3 years agoadd tests for indexer
Sarah Hoffmann [Wed, 20 Jan 2021 14:42:58 +0000 (15:42 +0100)]
add tests for indexer

3 years agoadd simple tests for CLI parsing
Sarah Hoffmann [Wed, 20 Jan 2021 08:53:47 +0000 (09:53 +0100)]
add simple tests for CLI parsing

3 years agooptionally hand in command line arguments to CLI functions
Sarah Hoffmann [Wed, 20 Jan 2021 08:04:04 +0000 (09:04 +0100)]
optionally hand in command line arguments to CLI functions

Allows easier testing.

3 years agouse pytest mocking functions for manipulating os.environ
Sarah Hoffmann [Wed, 20 Jan 2021 08:03:09 +0000 (09:03 +0100)]
use pytest mocking functions for manipulating os.environ

3 years agoadd tests for Python exec_utils
Sarah Hoffmann [Wed, 20 Jan 2021 08:02:45 +0000 (09:02 +0100)]
add tests for Python exec_utils

3 years agoMerge pull request #2146 from mtmail/two-typos
Sarah Hoffmann [Wed, 20 Jan 2021 20:29:49 +0000 (21:29 +0100)]
Merge pull request #2146 from mtmail/two-typos

correct parameter name in query CLI

3 years agocorrect parameter name in query CLI
marc tobias [Wed, 20 Jan 2021 20:09:41 +0000 (21:09 +0100)]
correct parameter name in query CLI

3 years agoMerge pull request #2145 from lonvia/cli-query-functions
Sarah Hoffmann [Wed, 20 Jan 2021 08:00:45 +0000 (09:00 +0100)]
Merge pull request #2145 from lonvia/cli-query-functions

Add interface to search via command line tool

3 years agoadd API functions to nominatim tool
Sarah Hoffmann [Tue, 19 Jan 2021 10:05:02 +0000 (11:05 +0100)]
add API functions to nominatim tool

3 years agorename nominatim.admin to nominatim.tools
Sarah Hoffmann [Mon, 18 Jan 2021 14:57:32 +0000 (15:57 +0100)]
rename nominatim.admin to nominatim.tools

3 years agoprobe for php_cgi in cmake to be used for querying
Sarah Hoffmann [Sun, 17 Jan 2021 21:03:54 +0000 (22:03 +0100)]
probe for php_cgi in cmake to be used for querying

3 years agoMerge pull request #2143 from lonvia/integrate-indexer-into-nominatim-tool
Sarah Hoffmann [Tue, 19 Jan 2021 07:42:22 +0000 (08:42 +0100)]
Merge pull request #2143 from lonvia/integrate-indexer-into-nominatim-tool

Integrate indexer into nominatim tool

3 years agoremove old nominatim.py in favour of 'nominatim index'
Sarah Hoffmann [Sun, 17 Jan 2021 20:02:50 +0000 (21:02 +0100)]
remove old nominatim.py in favour of 'nominatim index'

The PHP scripts need to know the position of the nominatim
tool in order to call it. This is handed in as environment
variable, so it can be set by the Python script.

3 years agoimplementaion of 'nominatim index'
Sarah Hoffmann [Sun, 17 Jan 2021 19:05:41 +0000 (20:05 +0100)]
implementaion of 'nominatim index'

3 years agomove indexing function into its own Python module
Sarah Hoffmann [Sun, 17 Jan 2021 16:19:17 +0000 (17:19 +0100)]
move indexing function into its own Python module

This makes it mow a standard function of our new Python
library instead of a stand-alone program.

3 years agoadd function to get a DSN for psycopg
Sarah Hoffmann [Sun, 17 Jan 2021 16:06:18 +0000 (17:06 +0100)]
add function to get a DSN for psycopg

Converts the PHP DSN syntax into psycopg syntax when necessary.

3 years agoMerge pull request #2142 from lonvia/update-bdd-api-tests
Sarah Hoffmann [Mon, 18 Jan 2021 14:40:50 +0000 (15:40 +0100)]
Merge pull request #2142 from lonvia/update-bdd-api-tests

Update BDD API tests

3 years agobdd: complete coverage for API tests
Sarah Hoffmann [Sat, 16 Jan 2021 21:20:23 +0000 (22:20 +0100)]
bdd: complete coverage for API tests

Also removes some functions that are no longer used and
fixes debug output where the tests found an issue.

3 years agoremove unused output formatting functions
Sarah Hoffmann [Sat, 16 Jan 2021 16:39:49 +0000 (17:39 +0100)]
remove unused output formatting functions

3 years agobdd: remove duplicated tests
Sarah Hoffmann [Sat, 16 Jan 2021 15:57:28 +0000 (16:57 +0100)]
bdd: remove duplicated tests

3 years agobdd: clean up and extend API tests for details
Sarah Hoffmann [Sat, 16 Jan 2021 11:04:13 +0000 (12:04 +0100)]
bdd: clean up and extend API tests for details

- remove duplicates created by replacing HTML tests
  with JSON tests
- add tests for newer functions for returning geometries
  and hierarchies

3 years agocollect coverage for /website directory as well
Sarah Hoffmann [Fri, 15 Jan 2021 19:27:14 +0000 (20:27 +0100)]
collect coverage for /website directory as well

3 years agoMerge pull request #2139 from lonvia/add-pytest
Sarah Hoffmann [Fri, 15 Jan 2021 16:37:36 +0000 (17:37 +0100)]
Merge pull request #2139 from lonvia/add-pytest

Introduce unit testing for Python code

3 years agopytest may also be installed as py-test[-3]
Sarah Hoffmann [Fri, 15 Jan 2021 15:45:37 +0000 (16:45 +0100)]
pytest may also be installed as py-test[-3]

3 years agoenable pytest testing in CI
Sarah Hoffmann [Fri, 15 Jan 2021 14:22:21 +0000 (15:22 +0100)]
enable pytest testing in CI

3 years agoadd documentation for new pytest tests
Sarah Hoffmann [Fri, 15 Jan 2021 14:18:45 +0000 (15:18 +0100)]
add documentation for new pytest tests

3 years agoadd pytest as a test goal in cmake
Sarah Hoffmann [Fri, 15 Jan 2021 14:09:36 +0000 (15:09 +0100)]
add pytest as a test goal in cmake

3 years agoadd initial pytest test for Configuration
Sarah Hoffmann [Fri, 15 Jan 2021 13:42:03 +0000 (14:42 +0100)]
add initial pytest test for Configuration

3 years agoMerge pull request #2136 from lonvia/introduce-pylint
Sarah Hoffmann [Fri, 15 Jan 2021 13:39:26 +0000 (14:39 +0100)]
Merge pull request #2136 from lonvia/introduce-pylint

Introduce pylint for code style checking for Python.

3 years agopylint may be available as pylint3 or pylint
Sarah Hoffmann [Fri, 15 Jan 2021 09:52:25 +0000 (10:52 +0100)]
pylint may be available as pylint3 or pylint

3 years agouse check parameter of subprocess.run
Sarah Hoffmann [Fri, 15 Jan 2021 09:38:28 +0000 (10:38 +0100)]
use check parameter of subprocess.run

...instead of checking on our own.

Also increase required version of Python to 3.5 because of
subprocess.run().

3 years agoadd pylint to list of required linting tools
Sarah Hoffmann [Fri, 15 Jan 2021 08:04:04 +0000 (09:04 +0100)]
add pylint to list of required linting tools

With pylint being run in the CI, passing it is required now.

3 years agofix errors reported by pylint
Sarah Hoffmann [Thu, 14 Jan 2021 20:36:31 +0000 (21:36 +0100)]
fix errors reported by pylint

3 years agoMerge pull request #2135 from lonvia/python-frontend
Sarah Hoffmann [Fri, 15 Jan 2021 07:56:07 +0000 (08:56 +0100)]
Merge pull request #2135 from lonvia/python-frontend

Introduce new 'nominatim' all-in-one command-line tool

3 years agofix typo
Sarah Hoffmann [Thu, 14 Jan 2021 19:30:27 +0000 (20:30 +0100)]
fix typo

3 years agoreplace Symfony dotenv dependency with Python dotenv
Sarah Hoffmann [Thu, 14 Jan 2021 17:31:18 +0000 (18:31 +0100)]
replace Symfony dotenv dependency with Python dotenv

3 years agobdd: directly call utility scripts in lib
Sarah Hoffmann [Thu, 14 Jan 2021 17:19:22 +0000 (18:19 +0100)]
bdd: directly call utility scripts in lib

This removes the dependency on php-symfony-dotenv for the tests.

3 years agomove dotenv parsing to installed PHP scripts
Sarah Hoffmann [Thu, 14 Jan 2021 16:09:22 +0000 (17:09 +0100)]
move dotenv parsing to installed PHP scripts

This means that the php-symfony-dotenv library is now only needed
when using the legacy scripts. This includes the BDD tests which
currently still rely on the PHP utils.

3 years agouse cli tool for github CI
Sarah Hoffmann [Thu, 14 Jan 2021 13:37:52 +0000 (14:37 +0100)]
use cli tool for github CI