overhaul the token analysis interface

[nominatim.git] / docs / develop / ICU-Tokenizer-Modules.md

diff --git a/docs/develop/ICU-Tokenizer-Modules.md b/docs/develop/ICU-Tokenizer-Modules.md
index 0578026c869d1676759bfd85f37ca88ea4ab41e7..3484cf4edb000fa3dd45507a5c073bea651a4a40 100644 (file)
--- a/docs/develop/ICU-Tokenizer-Modules.md
+++ b/docs/develop/ICU-Tokenizer-Modules.md
@@ -7,6 +7,12 @@ selection of sanitizers and token analyzers which you can use to adapt your
 installation to your needs. If the provided modules are not enough, you can
 also provide your own implementations. This section describes how to do that.
 
+!!! warning
+    This API is currently in early alpha status. While this API is meant to
+    be a public API on which other sanitizers and token analyzers may be
+    implemented, it is not guaranteed to be stable at the moment.
+
+
 ## Using non-standard sanitizers and token analyzers
 
 Sanitizer names (in the `step` property) and token analysis names (in the
@@ -52,7 +58,8 @@ the function.
 
 ### The sanitation function
 
-The sanitation function receives a single object with three members:
+The sanitation function receives a single object of type `ProcessInfo`
+which has with three members:
 
  * `place`: read-only information about the place being processed.
    See PlaceInfo below.
@@ -62,7 +69,7 @@ The sanitation function receives a single object with three members:
    is a PlaceName object.
 
 While the `place` member is provided for information only, the `names` and
-`address` lists are meant to be manipulated by the sanitizer. If may add and
+`address` lists are meant to be manipulated by the sanitizer. It may add and
 remove entries, change information within a single entry (for example by
 adding extra attributes) or completely replace the list with a different one.
 
@@ -76,7 +83,24 @@ adding extra attributes) or completely replace the list with a different one.
 
 #### PlaceName - extended naming information
 
-::: nominatim.tokenizer.sanitizers.base.PlaceName
+::: nominatim.data.place_name.PlaceName
+    rendering:
+        show_source: no
+        heading_level: 6
+
+## Custom token analysis module
+
+Setup of a token analyser is split into two parts: configuration and
+analyser factory. A token analysis module must therefore implement two
+functions:
+
+::: nominatim.tokenizer.token_analysis.base.AnalysisModule
+    rendering:
+        show_source: no
+        heading_level: 6
+
+
+::: nominatim.tokenizer.token_analysis.base.Analyzer
     rendering:
         show_source: no
         heading_level: 6