]> git.openstreetmap.org Git - nominatim.git/log
nominatim.git
3 years agoLinting fixes
Darkshredder [Mon, 8 Mar 2021 17:29:51 +0000 (22:59 +0530)]
Linting fixes

3 years agoPorted tiger-data-import to python and Added Tarball Support
Darkshredder [Mon, 8 Mar 2021 16:27:56 +0000 (21:57 +0530)]
Ported tiger-data-import to python and Added Tarball Support

3 years agoMerge pull request #2197 from lonvia/use-jinja-for-sql-preprocessing
Sarah Hoffmann [Thu, 4 Mar 2021 15:36:18 +0000 (16:36 +0100)]
Merge pull request #2197 from lonvia/use-jinja-for-sql-preprocessing

Use jinja2 for SQL preprocessing

3 years agoport index creation to python
Sarah Hoffmann [Thu, 4 Mar 2021 09:55:24 +0000 (10:55 +0100)]
port index creation to python

Also switches to jinja-based preprocessing, which allows to
simplify the SQL files. Use 'if not exists' where possible
so that the step can be rerun to fix missing indexes.

3 years agoindexer: ANALYSE must be run outside transactions
Sarah Hoffmann [Thu, 4 Mar 2021 10:05:56 +0000 (11:05 +0100)]
indexer: ANALYSE must be run outside transactions

3 years agomove table creation to jinja-based preprocessing
Sarah Hoffmann [Wed, 3 Mar 2021 21:07:51 +0000 (22:07 +0100)]
move table creation to jinja-based preprocessing

3 years agoadd new jinja2 requirement
Sarah Hoffmann [Wed, 3 Mar 2021 16:48:14 +0000 (17:48 +0100)]
add new jinja2 requirement

3 years agointroduce jinja2 for preprocessing SQL
Sarah Hoffmann [Wed, 3 Mar 2021 16:37:22 +0000 (17:37 +0100)]
introduce jinja2 for preprocessing SQL

Replaces various hand-crafted replacements of varying format with
a single Jinja2 templating mechanism. Allows full access to
configuration if necessary.

3 years agoMerge pull request #2194 from grischard/patch-1
Sarah Hoffmann [Wed, 3 Mar 2021 10:34:12 +0000 (11:34 +0100)]
Merge pull request #2194 from grischard/patch-1

Fix typo in .github/actions/build-nominatim/action.yml

3 years agoUpdate action.yml
Guillaume Rischard [Wed, 3 Mar 2021 10:20:21 +0000 (11:20 +0100)]
Update action.yml

3 years agoMerge pull request #2192 from lonvia/database-versioning
Sarah Hoffmann [Tue, 2 Mar 2021 14:57:46 +0000 (15:57 +0100)]
Merge pull request #2192 from lonvia/database-versioning

Introduce database versioning

3 years agodocument new status fields
Sarah Hoffmann [Mon, 1 Mar 2021 21:21:37 +0000 (22:21 +0100)]
document new status fields

3 years agoreturn database version with status API
Sarah Hoffmann [Mon, 1 Mar 2021 21:17:16 +0000 (22:17 +0100)]
return database version with status API

3 years agoadd database_version setting to tests
Sarah Hoffmann [Mon, 1 Mar 2021 20:49:33 +0000 (21:49 +0100)]
add database_version setting to tests

3 years agodo not depend on cmdline parameter for creating partition tables
Sarah Hoffmann [Mon, 1 Mar 2021 20:28:39 +0000 (21:28 +0100)]
do not depend on cmdline parameter for creating partition tables

The partition numbers in use only depend on the entries in search_name.

3 years agosave software version in the database
Sarah Hoffmann [Mon, 1 Mar 2021 19:35:15 +0000 (20:35 +0100)]
save software version in the database

The version represents the software version that was used to
import the data.

3 years agoreport software version of status call
Sarah Hoffmann [Mon, 1 Mar 2021 15:47:19 +0000 (16:47 +0100)]
report software version of status call

3 years agointroduce database patch level for version
Sarah Hoffmann [Mon, 1 Mar 2021 15:46:19 +0000 (16:46 +0100)]
introduce database patch level for version

This will be needed later for automatic migrations.

3 years agomake sure that calls to PHP legacy scripts are fatal on error
Sarah Hoffmann [Mon, 1 Mar 2021 15:10:45 +0000 (16:10 +0100)]
make sure that calls to PHP legacy scripts are fatal on error

3 years agointroduce table for in-database properties
Sarah Hoffmann [Mon, 1 Mar 2021 15:09:17 +0000 (16:09 +0100)]
introduce table for in-database properties

Adds a simple table where settings for the database can be
saved. This is useful for state that must not change after
import.

3 years agoremove unused import
Sarah Hoffmann [Mon, 1 Mar 2021 08:26:08 +0000 (09:26 +0100)]
remove unused import

3 years agouse tmp_path fixture in config tests
Sarah Hoffmann [Mon, 1 Mar 2021 08:24:04 +0000 (09:24 +0100)]
use tmp_path fixture in config tests

3 years agofix typo
Sarah Hoffmann [Mon, 1 Mar 2021 08:07:49 +0000 (09:07 +0100)]
fix typo

3 years agoMerge pull request #2186 from lonvia/port-import-to-python
Sarah Hoffmann [Sat, 27 Feb 2021 11:09:23 +0000 (12:09 +0100)]
Merge pull request #2186 from lonvia/port-import-to-python

Move setup procedure to Python

3 years agomake sure psql always finishes
Sarah Hoffmann [Sat, 27 Feb 2021 09:24:40 +0000 (10:24 +0100)]
make sure psql always finishes

If an execption is raised by other means, we still have to close
the stdin pipe to psql to make sure that it exits and releases its
connection to the database.

3 years agoolder versions of Postgresql need explicit return type
Sarah Hoffmann [Fri, 26 Feb 2021 17:23:24 +0000 (18:23 +0100)]
older versions of Postgresql need explicit return type

3 years agodo not truncate search_name in reverse-only mode
Sarah Hoffmann [Fri, 26 Feb 2021 17:20:53 +0000 (18:20 +0100)]
do not truncate search_name in reverse-only mode

3 years agoactions: add psutil dependency
Sarah Hoffmann [Fri, 26 Feb 2021 15:50:09 +0000 (16:50 +0100)]
actions: add psutil dependency

3 years agofix verbose flag for PHP wrapper scripts
Sarah Hoffmann [Fri, 26 Feb 2021 15:33:30 +0000 (16:33 +0100)]
fix verbose flag for PHP wrapper scripts

The flag must come after the command.

3 years agobdd: use python library where possible
Sarah Hoffmann [Fri, 26 Feb 2021 15:14:29 +0000 (16:14 +0100)]
bdd: use python library where possible

Replace calls to PHP scripts with direct calls into the
nominatim Python library where possible. This speed up
tests quite a bit.

3 years agomove setup function to python
Sarah Hoffmann [Fri, 26 Feb 2021 14:02:39 +0000 (15:02 +0100)]
move setup function to python

There are still back-calls to PHP for some of the sub-steps.
These needs some larger refactoring to be moved to Python.

3 years agoproperly close connections of indexer after use
Sarah Hoffmann [Fri, 26 Feb 2021 11:10:54 +0000 (12:10 +0100)]
properly close connections of indexer after use

3 years agoprot load-data function to python
Sarah Hoffmann [Thu, 25 Feb 2021 20:32:40 +0000 (21:32 +0100)]
prot load-data function to python

3 years agoadd a function for the intial indexing run
Sarah Hoffmann [Thu, 25 Feb 2021 10:25:01 +0000 (11:25 +0100)]
add a function for the intial indexing run

Also moves postcodes to fully parallel indexing.

3 years agoremove unused partitionfunction function
Sarah Hoffmann [Thu, 25 Feb 2021 09:22:52 +0000 (10:22 +0100)]
remove unused partitionfunction function

3 years agoport wikipedia importance functions to python
Sarah Hoffmann [Wed, 24 Feb 2021 21:02:13 +0000 (22:02 +0100)]
port wikipedia importance functions to python

3 years agomove import-data option to native python
Sarah Hoffmann [Wed, 24 Feb 2021 16:21:45 +0000 (17:21 +0100)]
move import-data option to native python

This adds a new dependecy to the Python psutil package.

3 years agointroduce custom object for cmdline arguments
Sarah Hoffmann [Wed, 24 Feb 2021 09:38:19 +0000 (10:38 +0100)]
introduce custom object for cmdline arguments

Allows to define special functions over the arguments.

Also splits CLI tests in two files as they have become too many.

3 years agoport database setup function to python
Sarah Hoffmann [Tue, 23 Feb 2021 21:50:23 +0000 (22:50 +0100)]
port database setup function to python

Hide the former PHP functions in a transition command until
they are removed.

3 years agouse psql for executing sql files
Sarah Hoffmann [Tue, 23 Feb 2021 18:05:51 +0000 (19:05 +0100)]
use psql for executing sql files

This allows to run larger files without needing to keep
them in memory.

3 years agoadd function to set up libpq environment
Sarah Hoffmann [Tue, 23 Feb 2021 13:11:11 +0000 (14:11 +0100)]
add function to set up libpq environment

Instead of parsing the DSN for each external libpq program we
are going to execute, provide a function that feeds them all
necessary parameters through the environment.

osm2pgsql is the first user.

3 years agoconvert connect() into a context manager
Sarah Hoffmann [Tue, 23 Feb 2021 09:11:21 +0000 (10:11 +0100)]
convert connect() into a context manager

3 years agoMerge pull request #2185 from lonvia/fix-deadlock-handling-for-psycopg27
Sarah Hoffmann [Thu, 25 Feb 2021 17:39:40 +0000 (18:39 +0100)]
Merge pull request #2185 from lonvia/fix-deadlock-handling-for-psycopg27

Improve deadlock detection for various versions of psycopg2

3 years agoimprove deadlock detection for various versions of psycopg2
Sarah Hoffmann [Thu, 25 Feb 2021 16:36:31 +0000 (17:36 +0100)]
improve deadlock detection for various versions of psycopg2

Psycopg2 has changed the kind of exception that is emitted on
deadlocks between versions 2.7 and 2.8. The code was already
trying to catch both kind of errors but because the
psycopg2.errors package is unknown in 2.7 and below, the
code would throw an exception on anything but a deadlock error.

This commit wraps the deadlock handling into a context manager
to avoid code duplication and uses module imports to detect if
the new error codes are available.

Also sets the required psycopg2 version to 2.7 or bigger as
versions below are difficult to test.

3 years agoMerge pull request #2182 from lonvia/change-error-for-details
Sarah Hoffmann [Tue, 23 Feb 2021 08:09:35 +0000 (09:09 +0100)]
Merge pull request #2182 from lonvia/change-error-for-details

Return 404 for details when no object is found in database

3 years agoreturn 404 for details when no bject is found in database
Sarah Hoffmann [Mon, 22 Feb 2021 15:28:29 +0000 (16:28 +0100)]
return 404 for details when no bject is found in database

Fixes #2157.

3 years agoMerge pull request #2181 from lonvia/port-more-tool-functions-to-python
Sarah Hoffmann [Mon, 22 Feb 2021 15:11:21 +0000 (16:11 +0100)]
Merge pull request #2181 from lonvia/port-more-tool-functions-to-python

Port more tool functions to python

3 years agoavoid os.environ as default valie
Sarah Hoffmann [Fri, 19 Feb 2021 18:29:57 +0000 (19:29 +0100)]
avoid os.environ as default valie

3 years agofix return code for check database run with 'not applicable'
Sarah Hoffmann [Fri, 19 Feb 2021 17:32:00 +0000 (18:32 +0100)]
fix return code for check database run with 'not applicable'

3 years agobdd tests: directly call python code for setup-website
Sarah Hoffmann [Fri, 19 Feb 2021 17:20:55 +0000 (18:20 +0100)]
bdd tests: directly call python code for setup-website

3 years agoport setup-website to python
Sarah Hoffmann [Fri, 19 Feb 2021 16:51:06 +0000 (17:51 +0100)]
port setup-website to python

3 years agoadd unit tests for new check_database code
Sarah Hoffmann [Thu, 18 Feb 2021 19:36:11 +0000 (20:36 +0100)]
add unit tests for new check_database code

3 years agoport check-database function to python
Sarah Hoffmann [Thu, 18 Feb 2021 16:32:30 +0000 (17:32 +0100)]
port check-database function to python

This change also adapts the hints to use the nominatim tool.
Slightly changed checks, so that they are just as effective on
a frozen database.

3 years agouse a frozen DB for API tests
Sarah Hoffmann [Wed, 17 Feb 2021 21:35:27 +0000 (22:35 +0100)]
use a frozen DB for API tests

This way we also test that dropping does the right thing.

3 years agoremove unused code
Sarah Hoffmann [Wed, 17 Feb 2021 21:25:23 +0000 (22:25 +0100)]
remove unused code

3 years agoport freeze function to python
Sarah Hoffmann [Wed, 17 Feb 2021 20:43:15 +0000 (21:43 +0100)]
port freeze function to python

3 years agoMerge pull request #2173 from lonvia/penality-for-housenumberless-places
Sarah Hoffmann [Wed, 17 Feb 2021 16:52:59 +0000 (17:52 +0100)]
Merge pull request #2173 from lonvia/penality-for-housenumberless-places

Increase penalty for places without housenumber

3 years agoMerge pull request #2174 from lonvia/disable-jit-for-osm2pgsql-again
Sarah Hoffmann [Tue, 16 Feb 2021 20:32:57 +0000 (21:32 +0100)]
Merge pull request #2174 from lonvia/disable-jit-for-osm2pgsql-again

Disable JIT and parallel execution for osm2pgsql updates again

3 years agodisable JIT and parallel execution for osm2pgsql updates again
Sarah Hoffmann [Tue, 16 Feb 2021 14:05:14 +0000 (15:05 +0100)]
disable JIT and parallel execution for osm2pgsql updates again

The gazetteer output doesn't disable these functions when
writing to the place table but the triggers may contain
operations that cause misplanning for the query planner.

3 years agoadd simple test for result splitting with multiple ranks
Sarah Hoffmann [Tue, 16 Feb 2021 16:59:12 +0000 (17:59 +0100)]
add simple test for result splitting with multiple ranks

3 years agoincrease penalty for places without housenumber
Sarah Hoffmann [Tue, 16 Feb 2021 16:47:06 +0000 (17:47 +0100)]
increase penalty for places without housenumber

Results where the housenumber was dropped are an unlikely result
when they refer to something other than a street. Therefore
increase their result rank so that other matches are tried first
before choosing them as a result.

Improves #2167.

3 years agoMerge pull request #2166 from mtmail/tiger-2020
Sarah Hoffmann [Tue, 16 Feb 2021 13:45:14 +0000 (14:45 +0100)]
Merge pull request #2166 from mtmail/tiger-2020

documentation: 2020 TIGER data got released

3 years agoMerge pull request #2171 from lonvia/update-vagrant-scripts-for-make-install
Sarah Hoffmann [Tue, 16 Feb 2021 13:42:38 +0000 (14:42 +0100)]
Merge pull request #2171 from lonvia/update-vagrant-scripts-for-make-install

Update vagrant scripts for make install

3 years agoswitch vagrant scripts to make install
Sarah Hoffmann [Tue, 16 Feb 2021 11:04:34 +0000 (12:04 +0100)]
switch vagrant scripts to make install

3 years agouse file copy on older cmake to install osm2pgsql
Sarah Hoffmann [Tue, 16 Feb 2021 10:06:14 +0000 (11:06 +0100)]
use file copy on older cmake to install osm2pgsql

Fixes #2170.

3 years agoignore failure to get replication date
Sarah Hoffmann [Sun, 14 Feb 2021 11:17:30 +0000 (12:17 +0100)]
ignore failure to get replication date

3 years agodocumentation: 2020 TIGER data got released
marc tobias [Wed, 3 Feb 2021 15:42:31 +0000 (16:42 +0100)]
documentation: 2020 TIGER data got released

3 years agoMerge pull request #2164 from lonvia/add-make-install
Sarah Hoffmann [Fri, 12 Feb 2021 09:32:26 +0000 (10:32 +0100)]
Merge pull request #2164 from lonvia/add-make-install

Make Nominatim installable

3 years agoalways return 0 for updates unless there is an error
Sarah Hoffmann [Thu, 11 Feb 2021 09:01:31 +0000 (10:01 +0100)]
always return 0 for updates unless there is an error

This is more in line with previous behavioru than returning
a status code when no updates are available.

3 years agodocs: adapt check-database command
Sarah Hoffmann [Wed, 10 Feb 2021 20:55:04 +0000 (21:55 +0100)]
docs: adapt check-database command

3 years agoactions: remove install directories before import
Sarah Hoffmann [Wed, 10 Feb 2021 15:35:50 +0000 (16:35 +0100)]
actions: remove install directories before import

This ensures that any dangling references to the build
or source directory are caught by the CI.

3 years agoneed test database for analysing cli test
Sarah Hoffmann [Wed, 10 Feb 2021 15:19:51 +0000 (16:19 +0100)]
need test database for analysing cli test

3 years agoactions: test import with installed version of Nominatim
Sarah Hoffmann [Wed, 10 Feb 2021 15:02:14 +0000 (16:02 +0100)]
actions: test import with installed version of Nominatim

3 years agoadd 'make install' to installation instructions
Sarah Hoffmann [Wed, 10 Feb 2021 10:15:21 +0000 (11:15 +0100)]
add 'make install' to installation instructions

3 years agotest for existance of country grid in cmake already
Sarah Hoffmann [Wed, 10 Feb 2021 09:40:36 +0000 (10:40 +0100)]
test for existance of country grid in cmake already

Given that the file potentially gets installed, it needs to be
present during build time already. Checking during the import
is too late.

3 years agomake installation targets conditional to what is built
Sarah Hoffmann [Wed, 10 Feb 2021 09:04:07 +0000 (10:04 +0100)]
make installation targets conditional to what is built

3 years agoadd make install target
Sarah Hoffmann [Tue, 9 Feb 2021 20:04:42 +0000 (21:04 +0100)]
add make install target

Installation includes PHP andPython libraries, settings, the basic
country data, the postgresql module and our custom version of
osm2pgsql. The latter is installed in our private library directory
so that it does not get in the way of a potentially installed
osm2pgsql from the distribution.

3 years agoadapt unit tests to new directory structure
Sarah Hoffmann [Tue, 9 Feb 2021 19:13:00 +0000 (20:13 +0100)]
adapt unit tests to new directory structure

3 years agouse DataDir constant for data only
Sarah Hoffmann [Tue, 9 Feb 2021 19:04:08 +0000 (20:04 +0100)]
use DataDir constant for data only

So far the data directory constant has pointed to the source
directory to be usable with different subdirectories. Now only
the data subdirectory itself is being used with the constant,
so point to the directory directly.

3 years agoforgot to replace one occurence of sql_dir
Sarah Hoffmann [Tue, 9 Feb 2021 18:32:05 +0000 (19:32 +0100)]
forgot to replace one occurence of sql_dir

3 years agoremove unused BINDIR constant
Sarah Hoffmann [Tue, 9 Feb 2021 18:30:31 +0000 (19:30 +0100)]
remove unused BINDIR constant

3 years agointroduce constant for configuration directory
Sarah Hoffmann [Tue, 9 Feb 2021 17:45:45 +0000 (18:45 +0100)]
introduce constant for configuration directory

This replaces {data_dir}/settings throughout the code, so that
the configuration may be placed somewhere else in the directory
structure (e.g. in /etc).

3 years agorename sql directory to lib-sql
Sarah Hoffmann [Tue, 9 Feb 2021 14:26:56 +0000 (15:26 +0100)]
rename sql directory to lib-sql

Also introduces a separate constant for the sql directory, so that
it can be put separately from the rest of the data if required.

3 years agorename lib to lib-php
Sarah Hoffmann [Tue, 9 Feb 2021 10:52:07 +0000 (11:52 +0100)]
rename lib to lib-php

3 years agoupdate osm2pgsql (disable install rule)
Sarah Hoffmann [Tue, 9 Feb 2021 08:48:50 +0000 (09:48 +0100)]
update osm2pgsql (disable install rule)

3 years agointegrate analyse of indexing into nominatim tool
Sarah Hoffmann [Mon, 8 Feb 2021 21:21:57 +0000 (22:21 +0100)]
integrate analyse of indexing into nominatim tool

3 years agoconsolidate warm and db-check into single admin command
Sarah Hoffmann [Mon, 8 Feb 2021 20:05:06 +0000 (21:05 +0100)]
consolidate warm and db-check into single admin command

3 years agosplit cli.py by subcommands
Sarah Hoffmann [Mon, 8 Feb 2021 16:23:05 +0000 (17:23 +0100)]
split cli.py by subcommands

Reduces file size below 1000 lines.

3 years agomove website into php library directory
Sarah Hoffmann [Mon, 8 Feb 2021 11:00:34 +0000 (12:00 +0100)]
move website into php library directory

3 years agomove postcode table setup to sql/
Sarah Hoffmann [Mon, 8 Feb 2021 10:48:45 +0000 (11:48 +0100)]
move postcode table setup to sql/

Also moves the call to the setup from the setup-db
step to the calculate-postcodes step. The tables also need
no longer be accessible by the webservice.

3 years agoremove unused SQL script
Sarah Hoffmann [Mon, 8 Feb 2021 10:28:24 +0000 (11:28 +0100)]
remove unused SQL script

3 years agofix off-by-one error in replication download
Sarah Hoffmann [Thu, 4 Feb 2021 16:04:04 +0000 (17:04 +0100)]
fix off-by-one error in replication download

3 years agoMerge pull request #2161 from lonvia/timeout-for-replication
Sarah Hoffmann [Thu, 4 Feb 2021 15:52:24 +0000 (16:52 +0100)]
Merge pull request #2161 from lonvia/timeout-for-replication

Reintroduce timeout for replication file download

3 years agoreintroduce timeout for replication file download
Sarah Hoffmann [Thu, 4 Feb 2021 10:47:11 +0000 (11:47 +0100)]
reintroduce timeout for replication file download

This ports the --socket-timeout parameter from
pyosmium-get-changes which ensures that the update
process eventually times out on hanging network connections.

3 years agoMerge pull request #2160 from lonvia/introduce-project-dir
Sarah Hoffmann [Thu, 4 Feb 2021 08:52:59 +0000 (09:52 +0100)]
Merge pull request #2160 from lonvia/introduce-project-dir

Officially introduce and recommend use of a project directory

3 years agoreplace make serve with nominatim serve command
Sarah Hoffmann [Wed, 3 Feb 2021 15:17:46 +0000 (16:17 +0100)]
replace make serve with nominatim serve command

With the website directory now tied to the project directory instead
of the build directory, it is no longer possible to use make for
running the web server.

3 years agoadd migration info for status table
Sarah Hoffmann [Wed, 3 Feb 2021 13:13:09 +0000 (14:13 +0100)]
add migration info for status table

3 years agoRevert "increase splitting for large geometries"
Sarah Hoffmann [Wed, 3 Feb 2021 09:23:38 +0000 (10:23 +0100)]
Revert "increase splitting for large geometries"

This reverts commit 559fe513fa8ffa3a5bc3a7d07c252a872614c073.

Increasing the splitting results in geometries where with rounding
issues at the split points, so that contain operations do not
work as expected anymore.

Fixes #2137.

3 years agoadapt quick start instructions in README to project dir
Sarah Hoffmann [Wed, 3 Feb 2021 09:17:22 +0000 (10:17 +0100)]
adapt quick start instructions in README to project dir

3 years agodo not overwrite custom set module paths
Sarah Hoffmann [Tue, 2 Feb 2021 16:33:41 +0000 (17:33 +0100)]
do not overwrite custom set module paths

Given that the module is now copied to the project directory
when no module path is set, we need the information that the
module path is empty. Therefore hand in the default module path
in a separate variable.