]> git.openstreetmap.org Git - nominatim.git/log
nominatim.git
3 years agodocs: unpacking tiger data is no longer necessary
Sarah Hoffmann [Tue, 6 Apr 2021 13:56:08 +0000 (15:56 +0200)]
docs: unpacking tiger data is no longer necessary

3 years agoMerge pull request #2250 from lonvia/save-transliterated-housenumbers
Sarah Hoffmann [Mon, 5 Apr 2021 13:48:22 +0000 (15:48 +0200)]
Merge pull request #2250 from lonvia/save-transliterated-housenumbers

Switch to saving transliterated housenumbers in placex

3 years agoadd migration for transliterated housenumbers
Sarah Hoffmann [Fri, 2 Apr 2021 15:28:52 +0000 (17:28 +0200)]
add migration for transliterated housenumbers

3 years agouse new transliteration in initial housenumber word computation
Sarah Hoffmann [Thu, 1 Apr 2021 16:52:06 +0000 (18:52 +0200)]
use new transliteration in initial housenumber word computation

The new create_housenumber_id() function splits housenumber
lists correctly. Otherwise there is no difference.

3 years agocorrectly handle housenumber lists
Sarah Hoffmann [Mon, 29 Mar 2021 15:36:44 +0000 (17:36 +0200)]
correctly handle housenumber lists

Lists are now standardised to use a semicolon separator.

3 years agomove transliteration of housenumbers into indexing
Sarah Hoffmann [Mon, 29 Mar 2021 14:45:09 +0000 (16:45 +0200)]
move transliteration of housenumbers into indexing

Housenumbers are now saved in transliterated form in the housenumber
column. This saves the transliteration step during lookup.

3 years agotests for finding non-ascii housenumbers
Sarah Hoffmann [Mon, 29 Mar 2021 14:36:45 +0000 (16:36 +0200)]
tests for finding non-ascii housenumbers

3 years agoreturn housenumbers always from address field
Sarah Hoffmann [Mon, 29 Mar 2021 07:36:56 +0000 (09:36 +0200)]
return housenumbers always from address field

This means that we can use normalized versions of the
housenumber in the housenumber field as it is no longer
a user visible field.

3 years agoMerge pull request #2248 from darkshredder/special-term-test
Sarah Hoffmann [Sat, 3 Apr 2021 16:31:01 +0000 (18:31 +0200)]
Merge pull request #2248 from darkshredder/special-term-test

Added Test for TokenSpecialTerm

3 years agoAdded Test for TokenSpecialTerm
Darkshredder [Thu, 1 Apr 2021 23:19:05 +0000 (04:49 +0530)]
Added Test for TokenSpecialTerm

3 years agoMerge pull request #2247 from lonvia/index-for-housenumber-lookup
Sarah Hoffmann [Thu, 1 Apr 2021 16:35:00 +0000 (18:35 +0200)]
Merge pull request #2247 from lonvia/index-for-housenumber-lookup

Index for housenumber lookup

3 years agoadd recommendation for Postgresql 11+
Sarah Hoffmann [Thu, 1 Apr 2021 12:37:20 +0000 (14:37 +0200)]
add recommendation for Postgresql 11+

3 years agouse non-key index to speed up housenumber search
Sarah Hoffmann [Thu, 1 Apr 2021 12:29:34 +0000 (14:29 +0200)]
use non-key index to speed up housenumber search

On Postgresql versions 11+ add an index to speed up the lookup
of housenumbers for terms found in search_name. This is really
just a band-aid around the query planer's interpretation of the
query.

3 years agoMerge pull request #2243 from darkshredder/XML-format-fix
Sarah Hoffmann [Tue, 30 Mar 2021 07:19:01 +0000 (09:19 +0200)]
Merge pull request #2243 from darkshredder/XML-format-fix

Fixed: XML format: more_url points to localhost, not base URL

3 years agoMerge pull request #2244 from AntoJvlt/import-special-phrases-tests-cleaning
Sarah Hoffmann [Tue, 30 Mar 2021 07:17:27 +0000 (09:17 +0200)]
Merge pull request #2244 from AntoJvlt/import-special-phrases-tests-cleaning

Cleaned tests for special phrases.

3 years agoAdded HTTP_HOST to if statement
Darkshredder [Mon, 29 Mar 2021 21:32:55 +0000 (03:02 +0530)]
Added HTTP_HOST to if statement

3 years agoCleaned tests of exceptions and fix phrase_settings.json test file name.
AntoJvlt [Mon, 29 Mar 2021 20:07:29 +0000 (22:07 +0200)]
Cleaned tests of exceptions and fix phrase_settings.json test file name.

3 years agofixed: XML format: more_url points to localhost, not base URL
Darkshredder [Mon, 29 Mar 2021 19:32:43 +0000 (01:02 +0530)]
fixed: XML format: more_url points to localhost, not base URL

3 years agoMerge pull request #2234 from darkshredder/add-man-page
Sarah Hoffmann [Mon, 29 Mar 2021 12:25:10 +0000 (14:25 +0200)]
Merge pull request #2234 from darkshredder/add-man-page

Added Manual page for Nominatim tool

3 years agoallow sorting by housenumbers for rare street names
Sarah Hoffmann [Mon, 29 Mar 2021 10:06:51 +0000 (12:06 +0200)]
allow sorting by housenumbers for rare street names

Usually we don't narrow down search results by house number when
only a street name is given because there may be a lot of rows
to cross check when the street name is very frequent. However,
when it is known to be rare, the housenumber check may be done
anyway.

Fixes #2238.

3 years agoUpdate manpage after rebase
Darkshredder [Mon, 29 Mar 2021 08:57:06 +0000 (14:27 +0530)]
Update manpage after rebase

3 years agoNominatim/cli.py rebase fixes
Darkshredder [Mon, 29 Mar 2021 08:46:41 +0000 (14:16 +0530)]
Nominatim/cli.py rebase fixes

3 years agoRebase with master
Darkshredder [Mon, 29 Mar 2021 08:30:45 +0000 (14:00 +0530)]
Rebase with master

3 years agofixed indentation and used sed to remove AUTHORS section
Darkshredder [Wed, 24 Mar 2021 15:08:14 +0000 (20:38 +0530)]
fixed indentation and used sed to remove AUTHORS section

3 years agoAdded Manual page and fixed documentation
Darkshredder [Tue, 23 Mar 2021 02:40:53 +0000 (08:10 +0530)]
Added Manual page and fixed documentation

3 years agoMerge pull request #2228 from AntoJvlt/import-special-phrases-porting-python
Sarah Hoffmann [Mon, 29 Mar 2021 07:49:35 +0000 (09:49 +0200)]
Merge pull request #2228 from AntoJvlt/import-special-phrases-porting-python

Import special phrases porting python

3 years agoChange command 'import-special-phrases --from-wiki' to 'special-phrases --import...
AntoJvlt [Fri, 26 Mar 2021 00:57:51 +0000 (01:57 +0100)]
Change command 'import-special-phrases --from-wiki' to 'special-phrases --import-from-wiki'.

3 years agoErrors fixes, Cleaning code, Improvement and addition of tests
AntoJvlt [Fri, 26 Mar 2021 00:53:33 +0000 (01:53 +0100)]
Errors fixes, Cleaning code, Improvement and addition of tests

3 years agoEncapsulation of tools/special_phrases.py into SpecialPhrasesImporter class and add...
AntoJvlt [Wed, 24 Mar 2021 13:26:52 +0000 (14:26 +0100)]
Encapsulation of tools/special_phrases.py into SpecialPhrasesImporter class and add new tests.

3 years agoCode cleaning, tests simplification and use of python3-icu package
AntoJvlt [Mon, 22 Mar 2021 22:56:24 +0000 (23:56 +0100)]
Code cleaning, tests simplification and use of python3-icu package

3 years agoUpdated documentation for PyICU support
AntoJvlt [Mon, 22 Mar 2021 00:46:46 +0000 (01:46 +0100)]
Updated documentation for PyICU support

3 years agoIntroduction of PyICU for transliteration in python. Reversed changes in normalizatio...
AntoJvlt [Sun, 21 Mar 2021 23:58:49 +0000 (00:58 +0100)]
Introduction of PyICU for transliteration in python. Reversed changes in normalization.sql.

3 years agoAdded wrapper in specialphrases.php to call corresponding nominatim command.
AntoJvlt [Sun, 21 Mar 2021 23:16:39 +0000 (00:16 +0100)]
Added wrapper in specialphrases.php to call corresponding nominatim command.

3 years agoChanged phrase_settings.py to phrase-settings.json and added migration function for...
AntoJvlt [Sun, 21 Mar 2021 23:07:55 +0000 (00:07 +0100)]
Changed phrase_settings.py to phrase-settings.json and added migration function for old php settings file.

3 years agoChanged phrase_settings.py to phrase-settings.json and added migration function for...
AntoJvlt [Sun, 21 Mar 2021 23:07:55 +0000 (00:07 +0100)]
Changed phrase_settings.py to phrase-settings.json and added migration function for old php settings file.

3 years agoMerge pull request #2233 from lonvia/index-for-postcode-ids
Sarah Hoffmann [Tue, 23 Mar 2021 08:18:10 +0000 (09:18 +0100)]
Merge pull request #2233 from lonvia/index-for-postcode-ids

Create postcode id index earlier

3 years agocreate postcode id index earlier
Sarah Hoffmann [Mon, 22 Mar 2021 21:24:56 +0000 (22:24 +0100)]
create postcode id index earlier

Now that the indexer takes care of indexing the postcode tables,
the id index is needed to find the rows to index.

3 years agoMerge pull request #2231 from mtmail/correct-cli-help-page
Sarah Hoffmann [Sun, 21 Mar 2021 15:52:20 +0000 (16:52 +0100)]
Merge pull request #2231 from mtmail/correct-cli-help-page

nominatim -h was printing wrong text for lookup and details

3 years agoavoid division by zero in progress meter
Sarah Hoffmann [Sun, 21 Mar 2021 15:47:22 +0000 (16:47 +0100)]
avoid division by zero in progress meter

On Windows systems the timer may not be accurate enough to measure
the time between init() and done(). Avoid computing statistics with
a diff time of 0 in such cases.

Fixes #2230.

3 years agonominatim -h was priting wrong text for lookup and details
marc tobias [Sun, 21 Mar 2021 15:06:41 +0000 (16:06 +0100)]
nominatim -h was priting wrong text for lookup and details

3 years agoDeleted specialphrases.php and phrase_settings.php
AntoJvlt [Sat, 20 Mar 2021 18:48:05 +0000 (19:48 +0100)]
Deleted specialphrases.php and phrase_settings.php

3 years agoUpdated specialphrases_testdb.sql
AntoJvlt [Sat, 20 Mar 2021 18:17:03 +0000 (19:17 +0100)]
Updated specialphrases_testdb.sql

3 years agoPorted functions for the import of special phrases from php to python.
AntoJvlt [Sat, 20 Mar 2021 17:55:08 +0000 (18:55 +0100)]
Ported functions for the import of special phrases from php to python.

- the command is now --import-special-phrases
- the output is not an sql file anymore, data are directly imported to the database.
- the little part on the documentation (section data import) has been modified.

3 years agobdd tests: make indexing less verbose
Sarah Hoffmann [Sat, 20 Mar 2021 09:39:29 +0000 (10:39 +0100)]
bdd tests: make indexing less verbose

Do not print progress info for indexing when there is an error
in the BDD tests.

3 years agoMerge pull request #2219 from lonvia/bdd-test-remove-php
Sarah Hoffmann [Wed, 17 Mar 2021 10:40:34 +0000 (11:40 +0100)]
Merge pull request #2219 from lonvia/bdd-test-remove-php

BDD tests: run all setup via nominatim Python library

3 years agobdd: run all setup via nominatim Python library
Sarah Hoffmann [Tue, 16 Mar 2021 21:13:33 +0000 (22:13 +0100)]
bdd: run all setup via nominatim Python library

Drops all calls to PHP utility functions. nominatim cli functions
are used where possible, to stay as close to the final code as
possible with the tests.

By removing the PHP calls, the test code now only uses osm2pgsql and
the database module from the build directory.

3 years agoMerge pull request #2216 from lonvia/fix-reverse-interpolation
Sarah Hoffmann [Mon, 15 Mar 2021 13:08:54 +0000 (14:08 +0100)]
Merge pull request #2216 from lonvia/fix-reverse-interpolation

Reverse: do not prefer interpolations over closer housenumbers

3 years agoreverse: do not prefer interpolations over closer housenumbers
Sarah Hoffmann [Mon, 15 Mar 2021 09:50:04 +0000 (10:50 +0100)]
reverse: do not prefer interpolations over closer housenumbers

Always look up the closest housenumber before looking up
interpolations. This ensures that closer housenumbers are
preferred over interpolations.

Fixes #2214.

3 years agoMerge pull request #2212 from darkshredder/country-name
Sarah Hoffmann [Mon, 15 Mar 2021 08:36:06 +0000 (09:36 +0100)]
Merge pull request #2212 from darkshredder/country-name

Ported createCountryNames() to python and Added tests

3 years agoAdd setup.php
Darkshredder [Sun, 14 Mar 2021 09:32:30 +0000 (15:02 +0530)]
Add setup.php

3 years agofix template variable name
Sarah Hoffmann [Sat, 13 Mar 2021 11:05:53 +0000 (12:05 +0100)]
fix template variable name

3 years agoMerge pull request #2213 from lonvia/tweak-search-weights
Sarah Hoffmann [Fri, 12 Mar 2021 14:47:36 +0000 (15:47 +0100)]
Merge pull request #2213 from lonvia/tweak-search-weights

Some more tweaking of the ranking of search interpretations

3 years agoLinting fix
Darkshredder [Fri, 12 Mar 2021 12:58:47 +0000 (18:28 +0530)]
Linting fix

3 years agorefactored tests and made changes to code for easy readibility
Darkshredder [Fri, 12 Mar 2021 12:53:20 +0000 (18:23 +0530)]
refactored tests and made changes to code for easy readibility

3 years agoPorted createCountryNames() to python and added tests
Darkshredder [Fri, 12 Mar 2021 04:58:41 +0000 (10:28 +0530)]
Ported createCountryNames() to python and added tests

3 years agoMerge pull request #2204 from darkshredder/tiger-data
Sarah Hoffmann [Thu, 11 Mar 2021 21:48:38 +0000 (22:48 +0100)]
Merge pull request #2204 from darkshredder/tiger-data

Ported tiger-data-import to Python and Added Tarball Support

3 years agodo not mix partial names with other words
Sarah Hoffmann [Thu, 11 Mar 2021 21:44:49 +0000 (22:44 +0100)]
do not mix partial names with other words

As soon as a housenumber, postcode, etc. appear, the name term
must obviously be closed and no further partial terms can be
appended.

3 years agomake linter happy
Sarah Hoffmann [Thu, 11 Mar 2021 20:14:23 +0000 (21:14 +0100)]
make linter happy

3 years agoavoid multi-term partials in names
Sarah Hoffmann [Thu, 11 Mar 2021 19:42:37 +0000 (20:42 +0100)]
avoid multi-term partials in names

Names are either full words or single-word partial names.
Searching for multi-word partials yields exactly the same
result as with full words.

3 years agohigher penalty for special searches
Sarah Hoffmann [Thu, 11 Mar 2021 19:34:21 +0000 (20:34 +0100)]
higher penalty for special searches

Adds a general higher penalty for special search term and an
additional one if the term is anywhere but the beginning or the
end. Also housenumbers and special searches together are less
likely.

3 years agodo not mix partial and full name terms
Sarah Hoffmann [Thu, 11 Mar 2021 19:22:54 +0000 (20:22 +0100)]
do not mix partial and full name terms

If NameNonSearch already contains a partial term, then a
full term must not be added to the Name list anymore.

3 years agoadd one-rank penalty for using partial search
Sarah Hoffmann [Thu, 11 Mar 2021 16:34:23 +0000 (17:34 +0100)]
add one-rank penalty for using partial search

Ensures that full matches are preferred over partial ones even when
the full word consists of only one term.

3 years agofix result splitting for last search group
Sarah Hoffmann [Thu, 11 Mar 2021 16:14:46 +0000 (17:14 +0100)]
fix result splitting for last search group

When we are in the final iteration of the search groups, it is not
possible to further delay the results. Unconditionally use the
results with the best rank instead.

3 years agogive preference to full words in address, too
Sarah Hoffmann [Thu, 11 Mar 2021 14:03:36 +0000 (15:03 +0100)]
give preference to full words in address, too

Full word terms are already preferred for the name part. Adding
only one-word partials to the address, makes it impossible to
give a similar preference for the address part. Each term adds
a rank penalty. The problem here is that we interpret the query
forwards and backwards. Having different penalty systems for
name and address means that the same term ends up with different
penalties and that often leads to interpretations of the wrong
direction being in the way.

3 years agoAdded fixture for sql_preprocessor and fixed some issues
Darkshredder [Thu, 11 Mar 2021 10:09:17 +0000 (15:39 +0530)]
Added fixture for sql_preprocessor and fixed some issues

3 years agoAdded test for tarfile
Darkshredder [Wed, 10 Mar 2021 12:44:17 +0000 (18:14 +0530)]
Added test for tarfile

3 years agoAdded test and removed runlegacyscript
Darkshredder [Wed, 10 Mar 2021 11:48:12 +0000 (17:18 +0530)]
Added test and removed runlegacyscript

3 years agofixed linting, refactored threaded sql handling and removed importTigerData() function
Darkshredder [Wed, 10 Mar 2021 07:58:29 +0000 (13:28 +0530)]
fixed linting, refactored threaded sql handling and removed importTigerData() function

3 years agoTest fixes
Darkshredder [Mon, 8 Mar 2021 19:30:56 +0000 (01:00 +0530)]
Test fixes

3 years agoLinting fixes
Darkshredder [Mon, 8 Mar 2021 17:40:49 +0000 (23:10 +0530)]
Linting fixes

3 years agoLinting fixes
Darkshredder [Mon, 8 Mar 2021 17:29:51 +0000 (22:59 +0530)]
Linting fixes

3 years agoPorted tiger-data-import to python and Added Tarball Support
Darkshredder [Mon, 8 Mar 2021 16:27:56 +0000 (21:57 +0530)]
Ported tiger-data-import to python and Added Tarball Support

3 years agoMerge pull request #2200 from lonvia/migrations-for-current-version
Sarah Hoffmann [Mon, 8 Mar 2021 09:14:03 +0000 (10:14 +0100)]
Merge pull request #2200 from lonvia/migrations-for-current-version

Introduce a command for database migration

3 years agodocumentation for new migration command
Sarah Hoffmann [Sat, 6 Mar 2021 15:38:37 +0000 (16:38 +0100)]
documentation for new migration command

3 years agoautomatic migration from 3.6 release
Sarah Hoffmann [Tue, 2 Mar 2021 20:26:13 +0000 (21:26 +0100)]
automatic migration from 3.6 release

Adds a 'admin --migrate' command that checks for the current
database version and runs any necessary migrations. Also
has migrations going back to 3.6.

3 years agoMerge pull request #2197 from lonvia/use-jinja-for-sql-preprocessing
Sarah Hoffmann [Thu, 4 Mar 2021 15:36:18 +0000 (16:36 +0100)]
Merge pull request #2197 from lonvia/use-jinja-for-sql-preprocessing

Use jinja2 for SQL preprocessing

3 years agoport index creation to python
Sarah Hoffmann [Thu, 4 Mar 2021 09:55:24 +0000 (10:55 +0100)]
port index creation to python

Also switches to jinja-based preprocessing, which allows to
simplify the SQL files. Use 'if not exists' where possible
so that the step can be rerun to fix missing indexes.

3 years agoindexer: ANALYSE must be run outside transactions
Sarah Hoffmann [Thu, 4 Mar 2021 10:05:56 +0000 (11:05 +0100)]
indexer: ANALYSE must be run outside transactions

3 years agomove table creation to jinja-based preprocessing
Sarah Hoffmann [Wed, 3 Mar 2021 21:07:51 +0000 (22:07 +0100)]
move table creation to jinja-based preprocessing

3 years agoadd new jinja2 requirement
Sarah Hoffmann [Wed, 3 Mar 2021 16:48:14 +0000 (17:48 +0100)]
add new jinja2 requirement

3 years agointroduce jinja2 for preprocessing SQL
Sarah Hoffmann [Wed, 3 Mar 2021 16:37:22 +0000 (17:37 +0100)]
introduce jinja2 for preprocessing SQL

Replaces various hand-crafted replacements of varying format with
a single Jinja2 templating mechanism. Allows full access to
configuration if necessary.

3 years agoMerge pull request #2194 from grischard/patch-1
Sarah Hoffmann [Wed, 3 Mar 2021 10:34:12 +0000 (11:34 +0100)]
Merge pull request #2194 from grischard/patch-1

Fix typo in .github/actions/build-nominatim/action.yml

3 years agoUpdate action.yml
Guillaume Rischard [Wed, 3 Mar 2021 10:20:21 +0000 (11:20 +0100)]
Update action.yml

3 years agoMerge pull request #2192 from lonvia/database-versioning
Sarah Hoffmann [Tue, 2 Mar 2021 14:57:46 +0000 (15:57 +0100)]
Merge pull request #2192 from lonvia/database-versioning

Introduce database versioning

3 years agodocument new status fields
Sarah Hoffmann [Mon, 1 Mar 2021 21:21:37 +0000 (22:21 +0100)]
document new status fields

3 years agoreturn database version with status API
Sarah Hoffmann [Mon, 1 Mar 2021 21:17:16 +0000 (22:17 +0100)]
return database version with status API

3 years agoadd database_version setting to tests
Sarah Hoffmann [Mon, 1 Mar 2021 20:49:33 +0000 (21:49 +0100)]
add database_version setting to tests

3 years agodo not depend on cmdline parameter for creating partition tables
Sarah Hoffmann [Mon, 1 Mar 2021 20:28:39 +0000 (21:28 +0100)]
do not depend on cmdline parameter for creating partition tables

The partition numbers in use only depend on the entries in search_name.

3 years agosave software version in the database
Sarah Hoffmann [Mon, 1 Mar 2021 19:35:15 +0000 (20:35 +0100)]
save software version in the database

The version represents the software version that was used to
import the data.

3 years agoreport software version of status call
Sarah Hoffmann [Mon, 1 Mar 2021 15:47:19 +0000 (16:47 +0100)]
report software version of status call

3 years agointroduce database patch level for version
Sarah Hoffmann [Mon, 1 Mar 2021 15:46:19 +0000 (16:46 +0100)]
introduce database patch level for version

This will be needed later for automatic migrations.

3 years agomake sure that calls to PHP legacy scripts are fatal on error
Sarah Hoffmann [Mon, 1 Mar 2021 15:10:45 +0000 (16:10 +0100)]
make sure that calls to PHP legacy scripts are fatal on error

3 years agointroduce table for in-database properties
Sarah Hoffmann [Mon, 1 Mar 2021 15:09:17 +0000 (16:09 +0100)]
introduce table for in-database properties

Adds a simple table where settings for the database can be
saved. This is useful for state that must not change after
import.

3 years agoremove unused import
Sarah Hoffmann [Mon, 1 Mar 2021 08:26:08 +0000 (09:26 +0100)]
remove unused import

3 years agouse tmp_path fixture in config tests
Sarah Hoffmann [Mon, 1 Mar 2021 08:24:04 +0000 (09:24 +0100)]
use tmp_path fixture in config tests

3 years agofix typo
Sarah Hoffmann [Mon, 1 Mar 2021 08:07:49 +0000 (09:07 +0100)]
fix typo

3 years agoMerge pull request #2186 from lonvia/port-import-to-python
Sarah Hoffmann [Sat, 27 Feb 2021 11:09:23 +0000 (12:09 +0100)]
Merge pull request #2186 from lonvia/port-import-to-python

Move setup procedure to Python

3 years agomake sure psql always finishes
Sarah Hoffmann [Sat, 27 Feb 2021 09:24:40 +0000 (10:24 +0100)]
make sure psql always finishes

If an execption is raised by other means, we still have to close
the stdin pipe to psql to make sure that it exits and releases its
connection to the database.

3 years agoolder versions of Postgresql need explicit return type
Sarah Hoffmann [Fri, 26 Feb 2021 17:23:24 +0000 (18:23 +0100)]
older versions of Postgresql need explicit return type

3 years agodo not truncate search_name in reverse-only mode
Sarah Hoffmann [Fri, 26 Feb 2021 17:20:53 +0000 (18:20 +0100)]
do not truncate search_name in reverse-only mode