]> git.openstreetmap.org Git - nominatim.git/commit
reorganize keyword creation for legacy tokenizer
authorSarah Hoffmann <lonvia@denofr.de>
Sun, 23 May 2021 21:58:58 +0000 (23:58 +0200)
committerSarah Hoffmann <lonvia@denofr.de>
Mon, 24 May 2021 08:41:42 +0000 (10:41 +0200)
commit4f4d15c28a8743c2f3dfb6d3e5b787b94ef66fc5
treec62f98ee68cf2de4035832b0367a2790c98fb720
parentfa3e48c59f7456e24a551171495edee063ca8ff5
reorganize keyword creation for legacy tokenizer

- only save partial words without internal spaces
- consider comma and semicolon a separator of full words
- consider parts before an opening bracket a full word
  (but not the part after the bracket)

Fixes #244.
lib-sql/tokenizer/legacy_tokenizer.sql
nominatim/tokenizer/legacy_icu_tokenizer.py
test/bdd/db/import/search_name.feature
test/python/test_tokenizer_legacy_icu.py