From f79bb5a2937cf8735cc4dc511adbe4a15a305d8d Mon Sep 17 00:00:00 2001 From: Sarah Hoffmann Date: Fri, 29 May 2026 15:50:56 +0200 Subject: [PATCH] move sanitizer documentation out of the tokenizer section --- docs/customize/Sanitizers.md | 112 +++++++++++++++++++ docs/customize/Tokenizers.md | 118 +------------------- docs/develop/ICU-Tokenizer-Modules.md | 121 +------------------- docs/develop/Sanitizer-Modules.md | 152 ++++++++++++++++++++++++++ mkdocs.yml | 2 + 5 files changed, 276 insertions(+), 229 deletions(-) create mode 100644 docs/customize/Sanitizers.md create mode 100644 docs/develop/Sanitizer-Modules.md diff --git a/docs/customize/Sanitizers.md b/docs/customize/Sanitizers.md new file mode 100644 index 00000000..41c40556 --- /dev/null +++ b/docs/customize/Sanitizers.md @@ -0,0 +1,112 @@ +# Sanitizers + +_Sanitizing_ is the process of cleaning up and otherwise preprocessing names +before adding them to the search index during the import process. This allows +to clean up tagging, normalise different spellings and mark names with extra +attributes for further processing. + +!!! hint + Sanitizers only have an effect on how the search index is built. They + do not change the information about each place that is saved in the + database. In particular, they have no influence on how the results are + displayed. The returned results always show the original information as + stored in the OpenStreetMap database. + + +## Configuration + +The sanitizing process is defined in the 'sanitizers.yaml' configuration +file. The file must contain a list of steps. Each step has a mandatory +parameter `step` which defines the type of sanitizer. Additional step +configuration may then be set with additional parameters. + +The steps are executed in the order that they are defined in the configuration +file. Order matters here: each sanitizer works with the output of the previous +step. + +## Pre-defined sanitizers + +The following is a list of sanitizers that are shipped with Nominatim. +To learn about how to add your own custom sanitizer, see the section on +[custom sanitizer modules](../develop/Sanitizer-Modules.md). + +### affix-expansion + +::: nominatim_db.tokenizer.sanitizers.affix_expansion + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### clean-housenumbers + +::: nominatim_db.tokenizer.sanitizers.clean_housenumbers + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### clean-postcodes + +::: nominatim_db.tokenizer.sanitizers.clean_postcodes + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### clean-tiger-tags + +::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### delete-names + +::: nominatim_db.tokenizer.sanitizers.delete_names + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### derive-names + +::: nominatim_db.tokenizer.sanitizers.derive_names + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### split-name-list + +::: nominatim_db.tokenizer.sanitizers.split_name_list + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### strip-brace-terms + +::: nominatim_db.tokenizer.sanitizers.strip_brace_terms + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### tag-analyzer-by-language + +::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +### tag-japanese + +::: nominatim_db.tokenizer.sanitizers.tag_japanese + options: + members: False + heading_level: 6 + docstring_section_style: spacy + diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md index 7d14f620..ca55c8c1 100644 --- a/docs/customize/Tokenizers.md +++ b/docs/customize/Tokenizers.md @@ -29,15 +29,12 @@ NOMINATIM_TOKENIZER=icu ### How it works -On import the tokenizer processes names in the following three stages: +On import the tokenizer receives the sanaitzed list of names and +processes it in the following two stages: -1. During the **Sanitizer step** incoming names are cleaned up and converted to - **full names**. This step can be used to regularize spelling, split multi-name - tags into their parts and tag names with additional attributes. See the - [Sanitizers section](#sanitizers) below for available cleaning routines. -2. The **Normalization** part removes all information from the full names +1. The **normalization** part removes all information from the full names that are not relevant for search. -3. The **Token analysis** step takes the normalized full names and creates +2. The **token analysis** step takes the normalized full names and creates all transliterated variants under which the name should be searchable. See the [Token analysis](#token-analysis) section below for more information. @@ -80,8 +77,6 @@ normalization: transliteration: - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml - ":: Ascii ()" -sanitizers: - - step: split-name-list token-analysis: - analyzer: generic variants: @@ -153,109 +148,6 @@ and may again include other files. YAML syntax. You should therefore always enclose the ICU rules in double-quotes. -#### Sanitizers - -The sanitizers section defines an ordered list of functions that are applied -to the name and address tags before they are further processed by the tokenizer. -They allows to clean up the tagging and bring it to a standardized form more -suitable for building the search index. - -!!! hint - Sanitizers only have an effect on how the search index is built. They - do not change the information about each place that is saved in the - database. In particular, they have no influence on how the results are - displayed. The returned results always show the original information as - stored in the OpenStreetMap database. - -Each entry contains information of a sanitizer to be applied. It has a -mandatory parameter `step` which gives the name of the sanitizer. Depending -on the type, it may have additional parameters to configure its operation. - -The order of the list matters. The sanitizers are applied exactly in the order -that is configured. Each sanitizer works on the results of the previous one. - -The following is a list of sanitizers that are shipped with Nominatim. - -##### split-name-list - -::: nominatim_db.tokenizer.sanitizers.split_name_list - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### strip-brace-terms - -::: nominatim_db.tokenizer.sanitizers.strip_brace_terms - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### tag-analyzer-by-language - -::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### clean-housenumbers - -::: nominatim_db.tokenizer.sanitizers.clean_housenumbers - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### clean-postcodes - -::: nominatim_db.tokenizer.sanitizers.clean_postcodes - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### clean-tiger-tags - -::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### delete-names - -::: nominatim_db.tokenizer.sanitizers.delete_names - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### derive-names - -::: nominatim_db.tokenizer.sanitizers.derive_names - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### tag-japanese - -::: nominatim_db.tokenizer.sanitizers.tag_japanese - options: - members: False - heading_level: 6 - docstring_section_style: spacy - -##### affix-expansion - -::: nominatim_db.tokenizer.sanitizers.affix_expansion - options: - members: False - heading_level: 6 - docstring_section_style: spacy - #### Token Analysis Token analyzers take a full name and transform it into one or more normalized @@ -267,7 +159,7 @@ decomposition and abbreviation. The ICU tokenizer may use different analyzers for different names. To select the analyzer to be used, the name must be tagged with the `analyzer` attribute by a sanitizer (see for example the -[tag-analyzer-by-language sanitizer](#tag-analyzer-by-language)). +[tag-analyzer-by-language sanitizer](Sanitizers.md#tag-analyzer-by-language)). The token-analysis section contains the list of configured analyzers. Each analyzer must have an `id` parameter that uniquely identifies the analyzer. diff --git a/docs/develop/ICU-Tokenizer-Modules.md b/docs/develop/ICU-Tokenizer-Modules.md index 30b85ac7..e353c310 100644 --- a/docs/develop/ICU-Tokenizer-Modules.md +++ b/docs/develop/ICU-Tokenizer-Modules.md @@ -1,22 +1,22 @@ -# Writing custom sanitizer and token analysis modules for the ICU tokenizer +# Writing custom token analysis modules for the ICU tokenizer The [ICU tokenizer](../customize/Tokenizers.md#icu-tokenizer) provides a highly customizable method to pre-process and normalize the name information of the input data before it is added to the search index. It comes with a -selection of sanitizers and token analyzers which you can use to adapt your +selection of token analyzers which you can use to adapt your installation to your needs. If the provided modules are not enough, you can also provide your own implementations. This section describes the API -of sanitizers and token analysis. +for token analysis. !!! warning This API is currently in early alpha status. While this API is meant to - be a public API on which other sanitizers and token analyzers may be + be a public API on which other token analyzers may be implemented, it is not guaranteed to be stable at the moment. ## Using non-standard modules -Sanitizer names (in the `step` property), token analysis names (in the +Token analysis names (in the `analyzer`) and query preprocessor names (in the `step` property) may refer to externally supplied modules. There are two ways to include external modules: through a library or from the project directory. @@ -75,117 +75,6 @@ are interpreted. The following phrase types are known: | PHRASE_COUNTRY | Contains the country name or code | -## Custom sanitizer modules - -A sanitizer module must export a single factory function `create` with the -following signature: - -``` python -def create(config: SanitizerConfig) -> Callable[[ProcessInfo], None] -``` - -The function receives the custom configuration for the sanitizer and must -return a callable (function or class) that transforms the name and address -terms of a place. When a place is processed, then a `ProcessInfo` object -is created from the information that was queried from the database. This -object is sequentially handed to each configured sanitizer, so that each -sanitizer receives the result of processing from the previous sanitizer. -After the last sanitizer is finished, the resulting name and address lists -are forwarded to the token analysis module. - -Sanitizer functions are instantiated once and then called for each place -that is imported or updated. They don't need to be thread-safe. -If multi-threading is used, each thread creates their own instance of -the function. - -### Sanitizer configuration - -::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig - options: - heading_level: 6 - -### The main filter function of the sanitizer - -The filter function receives a single object of type `ProcessInfo` -which has with three members: - - * `place: PlaceInfo`: read-only information about the place being processed. - See PlaceInfo below. - * `names: List[PlaceName]`: The current list of names for the place. - * `address: List[PlaceName]`: The current list of address names for the place. - -While the `place` member is provided for information only, the `names` and -`address` lists are meant to be manipulated by the sanitizer. It may add and -remove entries, change information within a single entry (for example by -adding extra attributes) or completely replace the list with a different one. - -#### PlaceInfo - information about the place - -::: nominatim_db.data.place_info.PlaceInfo - options: - heading_level: 6 - - -#### PlaceName - extended naming information - -::: nominatim_db.data.place_name.PlaceName - options: - heading_level: 6 - - -### Example: Filter for US street prefixes - -The following sanitizer removes the directional prefixes from street names -in the US: - -!!! example - ``` python - import re - - def _filter_function(obj): - if obj.place.country_code == 'us' \ - and obj.place.rank_address >= 26 and obj.place.rank_address <= 27: - for name in obj.names: - name.name = re.sub(r'^(north|south|west|east) ', - '', - name.name, - flags=re.IGNORECASE) - - def create(config): - return _filter_function - ``` - -This is the most simple form of a sanitizer module. If defines a single -filter function and implements the required `create()` function by returning -the filter. - -The filter function first checks if the object is interesting for the -sanitizer. Namely it checks if the place is in the US (through `country_code`) -and it the place is a street (a `rank_address` of 26 or 27). If the -conditions are met, then it goes through all available names and -removes any leading directional prefix using a simple regular expression. - -Save the source code in a file in your project directory, for example as -`us_streets.py`. Then you can use the sanitizer in your `icu_tokenizer.yaml`: - -``` yaml -... -sanitizers: - - step: us_streets.py -... -``` - -!!! warning - This example is just a simplified show case on how to create a sanitizer. - It is not really meant for real-world use: while the sanitizer would - correctly transform `West 5th Street` into `5th Street`. it would also - shorten a simple `North Street` to `Street`. - -For more sanitizer examples, have a look at the sanitizers provided by Nominatim. -They can be found in the directory -[`src/nominatim_db/tokenizer/sanitizers`](https://github.com/osm-search/Nominatim/tree/master/src/nominatim_db/tokenizer/sanitizers). - - ## Custom token analysis module ::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule diff --git a/docs/develop/Sanitizer-Modules.md b/docs/develop/Sanitizer-Modules.md new file mode 100644 index 00000000..05a076ea --- /dev/null +++ b/docs/develop/Sanitizer-Modules.md @@ -0,0 +1,152 @@ +# Writing custom sanitizer modules + +Sanitizers are used for preprocessing name and address information +from the OpenStreetMap input data for the search index. Read more about +them in the [Customizing sanitizers](../customize/Sanitizers.md) section. + +This section explains how to write your own sanitizer step function. + +## Using custom sanitizer modules + +To use a custom made sanitizer step, simply refer to the sanitizer module +in the `step` property. There are two ways +to include external modules: through a library or from the project directory. + +To include a module from a library, use the absolute import path as name and +make sure the library can be found in your PYTHONPATH. + +!!! Example + You have your sanitizer steps in a Python package my_sanitizer and + want to refer to the step implemented in module `translate_street.py`. + + ``` + - step: my_sanitizer.translate_street + config: some other config info for the step + ``` + +To use a custom module without creating a library, you can put the module +somewhere in your project directory and then use the relative path to the +file. Include the whole name of the file including the `.py` ending. + +!!! Example + You have put your module `translate_street.py` directly into the project + directory. + + ``` + - step: translate_street.py + config: some other config info for the step + ``` + +## Basic sanitizer module setup + +A sanitizer module must export a single factory function `create` with the +following signature: + +``` python +def create(config: SanitizerConfig) -> Callable[[ProcessInfo], None] +``` + +The function receives the custom configuration for the sanitizer and must +return a callable (function or class) that transforms the name and address +terms of a place. When a place is processed, then a `ProcessInfo` object +is created from the information that was queried from the database. This +object is sequentially handed to each configured sanitizer, so that each +sanitizer receives the result of processing from the previous sanitizer. +After the last sanitizer is finished, the resulting name and address lists +are forwarded to the token analysis module. + +Sanitizer functions are instantiated once and then called for each place +that is imported or updated. They don't need to be thread-safe. +If multi-threading is used, each thread creates their own instance of +the function. + +### Sanitizer configuration + +::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig + options: + heading_level: 6 + +### The main filter function of the sanitizer + +The filter function receives a single object of type `ProcessInfo` +which has with three members: + + * `place: PlaceInfo`: read-only information about the place being processed. + See PlaceInfo below. + * `names: List[PlaceName]`: The current list of names for the place. + * `address: List[PlaceName]`: The current list of address names for the place. + +While the `place` member is provided for information only, the `names` and +`address` lists are meant to be manipulated by the sanitizer. It may add and +remove entries, change information within a single entry (for example by +adding extra attributes) or completely replace the list with a different one. + +#### PlaceInfo - information about the place + +::: nominatim_db.data.place_info.PlaceInfo + options: + heading_level: 6 + + +#### PlaceName - extended naming information + +::: nominatim_db.data.place_name.PlaceName + options: + heading_level: 6 + + +### Example: Filter for US street prefixes + +The following sanitizer removes the directional prefixes from street names +in the US: + +!!! example + ``` python + import re + + def _filter_function(obj): + if obj.place.country_code == 'us' \ + and obj.place.rank_address >= 26 and obj.place.rank_address <= 27: + for name in obj.names: + name.name = re.sub(r'^(north|south|west|east) ', + '', + name.name, + flags=re.IGNORECASE) + + def create(config): + return _filter_function + ``` + +This is the most simple form of a sanitizer module. If defines a single +filter function and implements the required `create()` function by returning +the filter. + +The filter function first checks if the object is interesting for the +sanitizer. Namely it checks if the place is in the US (through `country_code`) +and it the place is a street (a `rank_address` of 26 or 27). If the +conditions are met, then it goes through all available names and +removes any leading directional prefix using a simple regular expression. + +Save the source code in a file in your project directory, for example as +`us_streets.py`. Then you can use the sanitizer in your `icu_tokenizer.yaml`: + +``` yaml +... +sanitizers: + - step: us_streets.py +... +``` + +!!! warning + This example is just a simplified show case on how to create a sanitizer. + It is not really meant for real-world use: while the sanitizer would + correctly transform `West 5th Street` into `5th Street`. it would also + shorten a simple `North Street` to `Street`. + +For more sanitizer examples, have a look at the sanitizers provided by Nominatim. +They can be found in the directory +[`src/nominatim_db/tokenizer/sanitizers`](https://github.com/osm-search/Nominatim/tree/master/src/nominatim_db/tokenizer/sanitizers). + + + + diff --git a/mkdocs.yml b/mkdocs.yml index 3fd9cd5a..b0d91311 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -42,6 +42,7 @@ nav: - 'Per-Country Data': 'customize/Country-Settings.md' - 'Place Ranking' : 'customize/Ranking.md' - 'Importance' : 'customize/Importance.md' + - 'Sanitizers' : 'customize/Sanitizers.md' - 'Tokenizers' : 'customize/Tokenizers.md' - 'Special Phrases': 'customize/Special-Phrases.md' - 'External data: US housenumbers from TIGER': 'customize/Tiger.md' @@ -58,6 +59,7 @@ nav: - 'Architecture Overview' : 'develop/overview.md' - 'Database Layout' : 'develop/Database-Layout.md' - 'Indexing' : 'develop/Indexing.md' + - 'Sanitizers' : 'develop/Sanitizer-Modules.md' - 'Tokenizers' : 'develop/Tokenizers.md' - 'Custom modules for ICU tokenizer': 'develop/ICU-Tokenizer-Modules.md' - 'Setup for Development' : 'develop/Development-Environment.md' -- 2.47.3