]> git.openstreetmap.org Git - nominatim.git/log
nominatim.git
3 years agouse pytest mocking functions for manipulating os.environ
Sarah Hoffmann [Wed, 20 Jan 2021 08:03:09 +0000 (09:03 +0100)]
use pytest mocking functions for manipulating os.environ

3 years agoadd tests for Python exec_utils
Sarah Hoffmann [Wed, 20 Jan 2021 08:02:45 +0000 (09:02 +0100)]
add tests for Python exec_utils

3 years agoMerge pull request #2146 from mtmail/two-typos
Sarah Hoffmann [Wed, 20 Jan 2021 20:29:49 +0000 (21:29 +0100)]
Merge pull request #2146 from mtmail/two-typos

correct parameter name in query CLI

3 years agocorrect parameter name in query CLI
marc tobias [Wed, 20 Jan 2021 20:09:41 +0000 (21:09 +0100)]
correct parameter name in query CLI

3 years agoMerge pull request #2145 from lonvia/cli-query-functions
Sarah Hoffmann [Wed, 20 Jan 2021 08:00:45 +0000 (09:00 +0100)]
Merge pull request #2145 from lonvia/cli-query-functions

Add interface to search via command line tool

3 years agoadd API functions to nominatim tool
Sarah Hoffmann [Tue, 19 Jan 2021 10:05:02 +0000 (11:05 +0100)]
add API functions to nominatim tool

3 years agorename nominatim.admin to nominatim.tools
Sarah Hoffmann [Mon, 18 Jan 2021 14:57:32 +0000 (15:57 +0100)]
rename nominatim.admin to nominatim.tools

3 years agoprobe for php_cgi in cmake to be used for querying
Sarah Hoffmann [Sun, 17 Jan 2021 21:03:54 +0000 (22:03 +0100)]
probe for php_cgi in cmake to be used for querying

3 years agoMerge pull request #2143 from lonvia/integrate-indexer-into-nominatim-tool
Sarah Hoffmann [Tue, 19 Jan 2021 07:42:22 +0000 (08:42 +0100)]
Merge pull request #2143 from lonvia/integrate-indexer-into-nominatim-tool

Integrate indexer into nominatim tool

3 years agoremove old nominatim.py in favour of 'nominatim index'
Sarah Hoffmann [Sun, 17 Jan 2021 20:02:50 +0000 (21:02 +0100)]
remove old nominatim.py in favour of 'nominatim index'

The PHP scripts need to know the position of the nominatim
tool in order to call it. This is handed in as environment
variable, so it can be set by the Python script.

3 years agoimplementaion of 'nominatim index'
Sarah Hoffmann [Sun, 17 Jan 2021 19:05:41 +0000 (20:05 +0100)]
implementaion of 'nominatim index'

3 years agomove indexing function into its own Python module
Sarah Hoffmann [Sun, 17 Jan 2021 16:19:17 +0000 (17:19 +0100)]
move indexing function into its own Python module

This makes it mow a standard function of our new Python
library instead of a stand-alone program.

3 years agoadd function to get a DSN for psycopg
Sarah Hoffmann [Sun, 17 Jan 2021 16:06:18 +0000 (17:06 +0100)]
add function to get a DSN for psycopg

Converts the PHP DSN syntax into psycopg syntax when necessary.

3 years agoMerge pull request #2142 from lonvia/update-bdd-api-tests
Sarah Hoffmann [Mon, 18 Jan 2021 14:40:50 +0000 (15:40 +0100)]
Merge pull request #2142 from lonvia/update-bdd-api-tests

Update BDD API tests

3 years agobdd: complete coverage for API tests
Sarah Hoffmann [Sat, 16 Jan 2021 21:20:23 +0000 (22:20 +0100)]
bdd: complete coverage for API tests

Also removes some functions that are no longer used and
fixes debug output where the tests found an issue.

3 years agoremove unused output formatting functions
Sarah Hoffmann [Sat, 16 Jan 2021 16:39:49 +0000 (17:39 +0100)]
remove unused output formatting functions

3 years agobdd: remove duplicated tests
Sarah Hoffmann [Sat, 16 Jan 2021 15:57:28 +0000 (16:57 +0100)]
bdd: remove duplicated tests

3 years agobdd: clean up and extend API tests for details
Sarah Hoffmann [Sat, 16 Jan 2021 11:04:13 +0000 (12:04 +0100)]
bdd: clean up and extend API tests for details

- remove duplicates created by replacing HTML tests
  with JSON tests
- add tests for newer functions for returning geometries
  and hierarchies

3 years agocollect coverage for /website directory as well
Sarah Hoffmann [Fri, 15 Jan 2021 19:27:14 +0000 (20:27 +0100)]
collect coverage for /website directory as well

3 years agoMerge pull request #2139 from lonvia/add-pytest
Sarah Hoffmann [Fri, 15 Jan 2021 16:37:36 +0000 (17:37 +0100)]
Merge pull request #2139 from lonvia/add-pytest

Introduce unit testing for Python code

3 years agopytest may also be installed as py-test[-3]
Sarah Hoffmann [Fri, 15 Jan 2021 15:45:37 +0000 (16:45 +0100)]
pytest may also be installed as py-test[-3]

3 years agoenable pytest testing in CI
Sarah Hoffmann [Fri, 15 Jan 2021 14:22:21 +0000 (15:22 +0100)]
enable pytest testing in CI

3 years agoadd documentation for new pytest tests
Sarah Hoffmann [Fri, 15 Jan 2021 14:18:45 +0000 (15:18 +0100)]
add documentation for new pytest tests

3 years agoadd pytest as a test goal in cmake
Sarah Hoffmann [Fri, 15 Jan 2021 14:09:36 +0000 (15:09 +0100)]
add pytest as a test goal in cmake

3 years agoadd initial pytest test for Configuration
Sarah Hoffmann [Fri, 15 Jan 2021 13:42:03 +0000 (14:42 +0100)]
add initial pytest test for Configuration

3 years agoMerge pull request #2136 from lonvia/introduce-pylint
Sarah Hoffmann [Fri, 15 Jan 2021 13:39:26 +0000 (14:39 +0100)]
Merge pull request #2136 from lonvia/introduce-pylint

Introduce pylint for code style checking for Python.

3 years agopylint may be available as pylint3 or pylint
Sarah Hoffmann [Fri, 15 Jan 2021 09:52:25 +0000 (10:52 +0100)]
pylint may be available as pylint3 or pylint

3 years agouse check parameter of subprocess.run
Sarah Hoffmann [Fri, 15 Jan 2021 09:38:28 +0000 (10:38 +0100)]
use check parameter of subprocess.run

...instead of checking on our own.

Also increase required version of Python to 3.5 because of
subprocess.run().

3 years agoadd pylint to list of required linting tools
Sarah Hoffmann [Fri, 15 Jan 2021 08:04:04 +0000 (09:04 +0100)]
add pylint to list of required linting tools

With pylint being run in the CI, passing it is required now.

3 years agofix errors reported by pylint
Sarah Hoffmann [Thu, 14 Jan 2021 20:36:31 +0000 (21:36 +0100)]
fix errors reported by pylint

3 years agoMerge pull request #2135 from lonvia/python-frontend
Sarah Hoffmann [Fri, 15 Jan 2021 07:56:07 +0000 (08:56 +0100)]
Merge pull request #2135 from lonvia/python-frontend

Introduce new 'nominatim' all-in-one command-line tool

3 years agofix typo
Sarah Hoffmann [Thu, 14 Jan 2021 19:30:27 +0000 (20:30 +0100)]
fix typo

3 years agoreplace Symfony dotenv dependency with Python dotenv
Sarah Hoffmann [Thu, 14 Jan 2021 17:31:18 +0000 (18:31 +0100)]
replace Symfony dotenv dependency with Python dotenv

3 years agobdd: directly call utility scripts in lib
Sarah Hoffmann [Thu, 14 Jan 2021 17:19:22 +0000 (18:19 +0100)]
bdd: directly call utility scripts in lib

This removes the dependency on php-symfony-dotenv for the tests.

3 years agomove dotenv parsing to installed PHP scripts
Sarah Hoffmann [Thu, 14 Jan 2021 16:09:22 +0000 (17:09 +0100)]
move dotenv parsing to installed PHP scripts

This means that the php-symfony-dotenv library is now only needed
when using the legacy scripts. This includes the BDD tests which
currently still rely on the PHP utils.

3 years agouse cli tool for github CI
Sarah Hoffmann [Thu, 14 Jan 2021 13:37:52 +0000 (14:37 +0100)]
use cli tool for github CI

3 years agoforward cli tool return value as exit code
Sarah Hoffmann [Thu, 14 Jan 2021 13:36:41 +0000 (14:36 +0100)]
forward cli tool return value as exit code

3 years agoadapt documentation to new nominatim cli tool
Sarah Hoffmann [Thu, 14 Jan 2021 11:04:08 +0000 (12:04 +0100)]
adapt documentation to new nominatim cli tool

3 years agomaller command execution fixes
Sarah Hoffmann [Thu, 14 Jan 2021 11:03:49 +0000 (12:03 +0100)]
maller command execution fixes

3 years agomake sure that environment variables have highest prio
Sarah Hoffmann [Thu, 14 Jan 2021 10:12:45 +0000 (11:12 +0100)]
make sure that environment variables have highest prio

3 years agoliniting of new python code
Sarah Hoffmann [Thu, 14 Jan 2021 09:19:21 +0000 (10:19 +0100)]
liniting of new python code

3 years agoadd wrapper calls for all nominatim tool functions
Sarah Hoffmann [Thu, 14 Jan 2021 08:37:47 +0000 (09:37 +0100)]
add wrapper calls for all nominatim tool functions

3 years agoavoid accessing constants when getting data from env
Sarah Hoffmann [Thu, 14 Jan 2021 08:30:23 +0000 (09:30 +0100)]
avoid accessing constants when getting data from env

When a setting can be read from the environment variable, avoid
accessing the internal defaults. This way the scripts can be
accessed directly in \lib as long as the environment is set up
correctly with full defaults.

3 years agofix access to environment variable
Sarah Hoffmann [Thu, 14 Jan 2021 08:29:43 +0000 (09:29 +0100)]
fix access to environment variable

3 years agoimplement warming in new cli tool
Sarah Hoffmann [Wed, 13 Jan 2021 17:25:15 +0000 (18:25 +0100)]
implement warming in new cli tool

Adds infrastructure for calling the legacy PHP scripts. As the
CONST_* values cannot be set from the python script, hand the values
in via secret environment variables instead. These are all
temporary hacks for the transition phase to python code.

3 years agowarm.php needs constant setup for queries
Sarah Hoffmann [Wed, 13 Jan 2021 17:12:53 +0000 (18:12 +0100)]
warm.php needs constant setup for queries

Warming is done using the query classes and therefore the same
copy-over from dotenv settings to CONST_* parameters is needed
as for query.php.

3 years agoset CONST_LibDir directly from the source scripts
Sarah Hoffmann [Wed, 13 Jan 2021 16:00:38 +0000 (17:00 +0100)]
set CONST_LibDir directly from the source scripts

Now that the source scripts have been moved to \lib, they
can determine the position of the PHP library relative to
themselves.

3 years agomove PHP utilities into the lib directory
Sarah Hoffmann [Wed, 13 Jan 2021 09:51:44 +0000 (10:51 +0100)]
move PHP utilities into the lib directory

These are not called directly as programs but used in a library
fashion by the installed utilities. So the library directory
is a better place.

3 years agoconsolidate cli interface to single tool
Sarah Hoffmann [Wed, 13 Jan 2021 09:11:58 +0000 (10:11 +0100)]
consolidate cli interface to single tool

3 years agocreate skeleton for new CLI tools
Sarah Hoffmann [Tue, 12 Jan 2021 21:21:20 +0000 (22:21 +0100)]
create skeleton for new CLI tools

3 years agoadd skeleton for new Nominatim executables
Sarah Hoffmann [Tue, 12 Jan 2021 09:17:28 +0000 (10:17 +0100)]
add skeleton for new Nominatim executables

3 years agoMerge pull request #2132 from lonvia/reduce-api-testdb
Sarah Hoffmann [Mon, 11 Jan 2021 09:42:22 +0000 (10:42 +0100)]
Merge pull request #2132 from lonvia/reduce-api-testdb

Reduce BDD API test database to Liechtenstein

3 years agoalso run BDD API tests in CI
Sarah Hoffmann [Sat, 9 Jan 2021 16:58:06 +0000 (17:58 +0100)]
also run BDD API tests in CI

3 years agotest can be run all in one go with make
Sarah Hoffmann [Sat, 9 Jan 2021 16:57:30 +0000 (17:57 +0100)]
test can be run all in one go with make

3 years agoupdate documentation for new BDD API tests
Sarah Hoffmann [Sat, 9 Jan 2021 16:54:45 +0000 (17:54 +0100)]
update documentation for new BDD API tests

3 years agobdd: add new API test data
Sarah Hoffmann [Sat, 9 Jan 2021 16:01:33 +0000 (17:01 +0100)]
bdd: add new API test data

Make all data necessary for API tests directly available in the
repository.

3 years agobdd: convert API tests to smaller test db
Sarah Hoffmann [Sat, 9 Jan 2021 15:59:46 +0000 (16:59 +0100)]
bdd: convert API tests to smaller test db

Changes BDD API tests to restrict themselves to
Liechtenstein. One test moved to DB as no appropriate
data is available.

3 years agobdd: improve assert output for API query checks
Sarah Hoffmann [Sat, 9 Jan 2021 15:58:37 +0000 (16:58 +0100)]
bdd: improve assert output for API query checks

Adds wrapper function for checking address parts and
more explanation strings to asserts.

3 years agobdd: import API test DB as part of step setup
Sarah Hoffmann [Thu, 7 Jan 2021 10:51:38 +0000 (11:51 +0100)]
bdd: import API test DB as part of step setup

In the future, the BDD tests will simply set up the required
test database themselves. Like with the template database, it
is not reimported when it already exists unless that is explicitly
forced.

Makes most of the API tests currently fail because they still
point to old test data.

3 years agoMerge pull request #2129 from lonvia/cleanup-bdd-tests
Sarah Hoffmann [Thu, 7 Jan 2021 08:10:40 +0000 (09:10 +0100)]
Merge pull request #2129 from lonvia/cleanup-bdd-tests

Clean up Python support code for BDD tests

3 years agoadd symphony dotenv to prerequisites list
Sarah Hoffmann [Thu, 7 Jan 2021 07:56:52 +0000 (08:56 +0100)]
add symphony dotenv to prerequisites list

3 years agouse relative dir for sources for phpunit
Sarah Hoffmann [Thu, 7 Jan 2021 07:55:15 +0000 (08:55 +0100)]
use relative dir for sources for phpunit

3 years agobdd: clean up DB ops steps
Sarah Hoffmann [Wed, 6 Jan 2021 15:37:32 +0000 (16:37 +0100)]
bdd: clean up DB ops steps

Adds comments and modernizes code.

3 years agobdd: move column comparison in separate file
Sarah Hoffmann [Wed, 6 Jan 2021 11:28:09 +0000 (12:28 +0100)]
bdd: move column comparison in separate file

Introduces a new class DBRow that encapsulates the comparison
functions. This also is responsible for formatting more informative
assert messages. place and placex steps are unified.

3 years agobdd: move nominitim id reader to separate file
Sarah Hoffmann [Tue, 5 Jan 2021 15:00:48 +0000 (16:00 +0100)]
bdd: move nominitim id reader to separate file

3 years agobdd: factor out reindexing on updates
Sarah Hoffmann [Tue, 5 Jan 2021 14:17:46 +0000 (15:17 +0100)]
bdd: factor out reindexing on updates

3 years agobdd: move place table inserter into separate file
Sarah Hoffmann [Tue, 5 Jan 2021 11:12:59 +0000 (12:12 +0100)]
bdd: move place table inserter into separate file

Also simplifies usage by implementing a function that inserts
a complete table row.

3 years agobdd: move scene setup to OSM data steps
Sarah Hoffmann [Tue, 5 Jan 2021 10:26:39 +0000 (11:26 +0100)]
bdd: move scene setup to OSM data steps

The step has nothing to do with the database.

3 years agobdd: switch to auto commit mode
Sarah Hoffmann [Tue, 5 Jan 2021 10:22:43 +0000 (11:22 +0100)]
bdd: switch to auto commit mode

Put the connection to the test database into auto-commit mode
and get rid of the explicit commits. Also use cursors always in
context managers and unify the two implementations that copy
data from the place table.

3 years agobdd: remove class for lazy formatting
Sarah Hoffmann [Tue, 5 Jan 2021 09:39:44 +0000 (10:39 +0100)]
bdd: remove class for lazy formatting

assert in combination with format() does the right thing and calls
the __str__() method only when an assertion hits.

3 years agobdd: rename db_ops steps
Sarah Hoffmann [Tue, 5 Jan 2021 09:20:00 +0000 (10:20 +0100)]
bdd: rename db_ops steps

Now all files implementing steps are called steps_*.py.

3 years agobdd: move output format computation into response
Sarah Hoffmann [Tue, 5 Jan 2021 09:17:59 +0000 (10:17 +0100)]
bdd: move output format computation into response

3 years agobdd: move Response classes in own file and simplify
Sarah Hoffmann [Mon, 4 Jan 2021 21:18:01 +0000 (22:18 +0100)]
bdd: move Response classes in own file and simplify

Removes most of the duplicated parse functions, introduces
a common assert_field function with a more expressive error
message.

3 years agobdd: rename and clean up osm_data steps
Sarah Hoffmann [Mon, 4 Jan 2021 19:17:17 +0000 (20:17 +0100)]
bdd: rename and clean up osm_data steps

Move common OPL creation code into a function and remove
unused imports.

3 years agobdd: move external process execution in separate func
Sarah Hoffmann [Mon, 4 Jan 2021 18:58:59 +0000 (19:58 +0100)]
bdd: move external process execution in separate func

3 years agobdd: move NominatimEnvironment into separate file
Sarah Hoffmann [Mon, 4 Jan 2021 16:54:51 +0000 (17:54 +0100)]
bdd: move NominatimEnvironment into separate file

Also cleans up and modernizes the code and adds documentation.

3 years agobdd: move grid generation code into geometry factory
Sarah Hoffmann [Mon, 4 Jan 2021 16:04:47 +0000 (17:04 +0100)]
bdd: move grid generation code into geometry factory

3 years agobdd: move geoemtry creation into separate file
Sarah Hoffmann [Mon, 4 Jan 2021 15:29:15 +0000 (16:29 +0100)]
bdd: move geoemtry creation into separate file

Also renames the OsmDataFactory in the more appropriate
GeometryFactory and modernizes code for python3.

3 years agoremove stale code for python2
Sarah Hoffmann [Mon, 4 Jan 2021 13:14:34 +0000 (14:14 +0100)]
remove stale code for python2

3 years agoMerge pull request #2125 from lonvia/independent-project-directory
Sarah Hoffmann [Mon, 4 Jan 2021 13:10:24 +0000 (14:10 +0100)]
Merge pull request #2125 from lonvia/independent-project-directory

Allow for truely independent project directory

3 years agocreate a temporary project dir for tests
Sarah Hoffmann [Mon, 4 Jan 2021 10:39:45 +0000 (11:39 +0100)]
create a temporary project dir for tests

The project directory contains the website script as
configured through the test configuration. This means
that tests are now completely independet of any
configuration that may be contained in the build
directory.

Also removes the hack to inject additional settings via
a environment variable.

3 years agoconfigure osm2pgsql and module location via cmake
Sarah Hoffmann [Mon, 4 Jan 2021 10:37:56 +0000 (11:37 +0100)]
configure osm2pgsql and module location via cmake

The default location of osm2pgsql and the postgresql module
is decided at compile/installation time and is not necessarily
in the project directory.

With this change it is now possible to have a project directory
that is completely separate from the build directory.

3 years agoMerge pull request #2124 from lonvia/remove-nose
Sarah Hoffmann [Sun, 3 Jan 2021 20:04:59 +0000 (21:04 +0100)]
Merge pull request #2124 from lonvia/remove-nose

Remove nose dependency for tests

3 years agoremove nose requirement from documentation
Sarah Hoffmann [Sun, 3 Jan 2021 16:23:44 +0000 (17:23 +0100)]
remove nose requirement from documentation

3 years agoreplace nose assertions with simple asserts
Sarah Hoffmann [Fri, 1 Jan 2021 14:54:52 +0000 (15:54 +0100)]
replace nose assertions with simple asserts

3 years agoMerge pull request #2119 from mtmail/check-import-finished-when-tables-droped
Sarah Hoffmann [Tue, 22 Dec 2020 14:57:48 +0000 (15:57 +0100)]
Merge pull request #2119 from mtmail/check-import-finished-when-tables-droped

utils/check_import_finished: skip some checks when setup ran with --drop

3 years agoMerge pull request #2118 from mtmail/vagrant-ubuntu-dotenv
Sarah Hoffmann [Tue, 22 Dec 2020 14:54:48 +0000 (15:54 +0100)]
Merge pull request #2118 from mtmail/vagrant-ubuntu-dotenv

Vagrant ubuntu: install dotenv package

3 years agoutils/check_import_finished: skip some checks when setup ran with --drop
marc tobias [Mon, 21 Dec 2020 19:12:31 +0000 (20:12 +0100)]
utils/check_import_finished: skip some checks when setup ran with --drop

3 years agoVagrant ubuntu: install dotenv package
marc tobias [Mon, 21 Dec 2020 19:10:13 +0000 (20:10 +0100)]
Vagrant ubuntu: install dotenv package

3 years agoMerge pull request #2115 from lonvia/use-dotenv
Sarah Hoffmann [Mon, 21 Dec 2020 10:33:38 +0000 (11:33 +0100)]
Merge pull request #2115 from lonvia/use-dotenv

Switch configuration to dotenv

3 years agoswitch documentation to describing dotenv
Sarah Hoffmann [Sun, 20 Dec 2020 11:09:27 +0000 (12:09 +0100)]
switch documentation to describing dotenv

3 years agoadapt instructions for creating the test db to dotenv
Sarah Hoffmann [Sun, 20 Dec 2020 10:53:19 +0000 (11:53 +0100)]
adapt instructions for creating the test db to dotenv

3 years agomark CentOS installation instructions as broken
Sarah Hoffmann [Sun, 20 Dec 2020 10:35:29 +0000 (11:35 +0100)]
mark CentOS installation instructions as broken

Getting symfony-dotenv installed on CentOS is a major pain,
so just mark it broken instead.

Still sSwitch the config format to dotenv already.

3 years agotests: always create the config file
Sarah Hoffmann [Sat, 19 Dec 2020 16:55:46 +0000 (17:55 +0100)]
tests: always create the config file

There is also one database test that uses the API functions.

3 years agoadapt php tests to new directory constants
Sarah Hoffmann [Thu, 17 Dec 2020 15:50:52 +0000 (16:50 +0100)]
adapt php tests to new directory constants

3 years agomake phpcs happy
Sarah Hoffmann [Thu, 17 Dec 2020 15:44:52 +0000 (16:44 +0100)]
make phpcs happy

3 years agogithub actions: need dotenv
Sarah Hoffmann [Thu, 17 Dec 2020 10:53:15 +0000 (11:53 +0100)]
github actions: need dotenv

3 years agomove creation of website scripts to setup script
Sarah Hoffmann [Thu, 17 Dec 2020 13:58:09 +0000 (14:58 +0100)]
move creation of website scripts to setup script

Instead of creating the website wrapper scripts with cmake,
they are now created when --setup-website is called. The
setup of the configuration constants is directly embedded
into the scripts. This means we can get rid of the separate
settings-frontend.php. More importantly however, it means
that it is now possible to set up multiple website directories
from the same build directory.

3 years agoadapt tests to new dotenv environment
Sarah Hoffmann [Thu, 17 Dec 2020 10:46:01 +0000 (11:46 +0100)]
adapt tests to new dotenv environment

DB tests now can simply set the environment to change configuration
variables. API tests still rely on a configuration file.

Also, query.php needs to set up the CONST_* variables to work with
the query scripts. That is a tiny bit messy and duplicates code
but this part will need to be reworked later.

3 years agofix various typos
Sarah Hoffmann [Wed, 16 Dec 2020 19:41:35 +0000 (20:41 +0100)]
fix various typos