If you have configured a flatnode file for the import, then you
need to keep this flatnode file around for updates.
-#### Installing the newest version of Pyosmium
+### Installing the newest version of Pyosmium
-It is recommended to install Pyosmium via pip. Make sure to use python3.
+The replication process uses
+[Pyosmium](https://docs.osmcode.org/pyosmium/latest/updating_osm_data.html)
+to download update data from the server.
+It is recommended to install Pyosmium via pip.
Run (as the same user who will later run the updates):
```sh
pip3 install --user osmium
```
-#### Setting up the update process
+### Setting up the update process
-Next the update needs to be initialised. By default Nominatim is configured
+Next the update process needs to be initialised. By default Nominatim is configured
to update using the global minutely diffs.
If you want a different update source you will need to add some settings
The `replication --init` command needs to be rerun whenever the replication
service is changed.
-#### Updating Nominatim
+### Updating Nominatim
-The following command will keep your database constantly up to date:
+Nominatim supports different modes how to retrieve the update data from the
+server. Which one you want to use depends on your exact setup and how often you
+want to retrieve updates.
+
+These instructions are for using a single source of updates. If you have
+imported multiple country extracts and want to keep them
+up-to-date, [Advanced installations section](Advanced-Installations.md)
+contains instructions to set up and update multiple country extracts.
+
+#### Continuous updates
+
+This is the easiest mode. Simply run the replication command without any
+parameters:
nominatim replication
-If you have imported multiple country extracts and want to keep them
-up-to-date, [Advanced installations section](Advanced-Installations.md) contains instructions
-to set up and update multiple country extracts.
+The update application keeps running forever and retrieves and applies
+new updates from the server as they are published.
+
+You can run this command as a simple systemd service. Create a service
+description like that in `/etc/systemd/system/nominatim-update.service`:
+
+```
+[Unit]
+Description=Continuous updates of Nominatim
+
+[Service]
+WorkingDirectory=/srv/nominatim
+ExecStart=nominatim replication
+StandardOutput=append:/var/log/nominatim-updates.log
+StandardError=append:/var/log/nominatim-updates.error.log
+User=nominatim
+Group=nominatim
+Type=simple
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Replace the `WorkingDirectory` with your project directory. Also adapt user
+and group names as required.
+
+Now activate the service and start the updates:
+
+```
+sudo systemctl daemon-reload
+sudo systemctl enable nominatim-updates
+sudo systemctl start nominatim-updates
+```
+
+#### One-time mode
+
+When the `--once` parameter is given, then Nominatim will download exactly one
+batch of updates and then exit. This one-time mode still respects the
+`NOMINATIM_REPLICATION_UPDATE_INTERVAL` that you have set. If according to
+the update interval no new data has been published yet, it will go to sleep
+until the next expected update and only then attempt to download the next batch.
+
+The one-time mode is particularly useful if you want to run updates continuously
+but need to schedule other work in between updates. For example, the main
+service at osm.org uses it, to regularly recompute postcodes -- a process that
+must not be run while updates are in progress. Its update script
+looks like this:
+
+```sh
+#!/bin/bash
+
+# Switch to your project directory.
+cd /srv/nominatim
+
+while true; do
+ nominatim replication --once
+ if [ -f "/srv/nominatim/schedule-mainenance" ]; then
+ rm /srv/nominatim/schedule-mainenance
+ nominatim refresh --postcodes
+ fi
+done
+```
+
+A cron job then creates the file `/srv/nominatim/need-mainenance` once per night.
+
+
+#### Catch-up mode
+
+With the `--catch-up` parameter, Nominatim will immediately try to download
+all changes from the server until the database is up-to-date. The catch-up mode
+still respects the parameter `NOMINATIM_REPLICATION_MAX_DIFF`. It downloads and
+applies the changes in appropriate batches until all is done.
+
+The catch-up mode is foremost useful to bring the database up to speed after the
+initial import. Give that the service usually is not in production at this
+point, you can temporarily be a bit more generous with the batch size and
+number of threads you use for the updates by running catch-up like this:
+
+```
+cd /srv/nominatim
+NOMINATIM_REPLICATION_MAX_DIFF=5000 nominatim replication --catch-up --threads 15
+```
+
+The catch-up mode is also useful when you want to apply updates at a lower
+frequency than what the source publishes. You can set up a cron job to run
+replication catch-up at whatever interval you desire.
+
+!!! hint
+ When running scheduled updates with catch-up, it is a good idea to choose
+ a replication source with an update frequency that is an order of magnitude
+ lower. For example, if you want to update once a day, use an hourly updated
+ source. This makes sure that you don't miss an entire day of updates when
+ the source is unexpectely late to publish its update.
+
+ If you want to use the source with the same update frequency (e.g. a daily
+ updated source with daily updates), use the
+ continuous update mode. It ensures to re-request the newest update until it
+ is published.
help='Initialise the update process')
group.add_argument('--no-update-functions', dest='update_functions',
action='store_false',
- help=("Do not update the trigger function to "
- "support differential updates."))
+ help="Do not update the trigger function to "
+ "support differential updates (EXPERT)")
group = parser.add_argument_group('Arguments for updates')
group.add_argument('--check-for-updates', action='store_true',
help='Check if new updates are available and exit')
group.add_argument('--once', action='store_true',
- help=("Download and apply updates only once. When "
- "not set, updates are continuously applied"))
+ help="Download and apply updates only once. When "
+ "not set, updates are continuously applied")
+ group.add_argument('--catch-up', action='store_true',
+ help="Download and apply updates until no new "
+ "data is available on the server")
group.add_argument('--no-index', action='store_false', dest='do_index',
help=("Do not index the new data. Only usable "
"together with --once"))
round_time(end - start_import),
round_time(end - batchdate))
+
+ @staticmethod
+ def _compute_update_interval(args):
+ if args.catch_up:
+ return 0
+
+ update_interval = args.config.get_int('REPLICATION_UPDATE_INTERVAL')
+ # Sanity check to not overwhelm the Geofabrik servers.
+ if 'download.geofabrik.de' in args.config.REPLICATION_URL\
+ and update_interval < 86400:
+ LOG.fatal("Update interval too low for download.geofabrik.de.\n"
+ "Please check install documentation "
+ "(https://nominatim.org/release-docs/latest/admin/Import-and-Update#"
+ "setting-up-the-update-process).")
+ raise UsageError("Invalid replication update interval setting.")
+
+ return update_interval
+
+
@staticmethod
def _update(args):
from ..tools import replication
from ..indexer.indexer import Indexer
from ..tokenizer import factory as tokenizer_factory
+ update_interval = UpdateReplication._compute_update_interval(args)
+
params = args.osm2pgsql_options(default_cache=2000, default_threads=1)
params.update(base_url=args.config.REPLICATION_URL,
- update_interval=args.config.get_int('REPLICATION_UPDATE_INTERVAL'),
+ update_interval=update_interval,
import_file=args.project_dir / 'osmosischange.osc',
max_diff_size=args.config.get_int('REPLICATION_MAX_DIFF'),
indexed_only=not args.once)
- # Sanity check to not overwhelm the Geofabrik servers.
- if 'download.geofabrik.de' in params['base_url']\
- and params['update_interval'] < 86400:
- LOG.fatal("Update interval too low for download.geofabrik.de.\n"
- "Please check install documentation "
- "(https://nominatim.org/release-docs/latest/admin/Import-and-Update#"
- "setting-up-the-update-process).")
- raise UsageError("Invalid replication update interval setting.")
-
if not args.once:
if not args.do_index:
LOG.fatal("Indexing cannot be disabled when running updates continuously.")
index_start = dt.datetime.now(dt.timezone.utc)
indexer = Indexer(args.config.get_libpq_dsn(), tokenizer,
args.threads or 1)
- indexer.index_boundaries(0, 30)
- indexer.index_by_rank(0, 30)
+ indexer.index_full(analyse=False)
with connect(args.config.get_libpq_dsn()) as conn:
status.set_indexed(conn, True)
else:
index_start = None
+ if state is replication.UpdateState.NO_CHANGES and \
+ args.catch_up or update_interval > 40*60:
+ while indexer.has_pending():
+ indexer.index_full(analyse=False)
+
if LOG.isEnabledFor(logging.WARNING):
UpdateReplication._report_update(batchdate, start, index_start)
- if args.once:
+ if args.once or (args.catch_up and state is replication.UpdateState.NO_CHANGES):
break
if state is replication.UpdateState.NO_CHANGES:
self.num_threads = num_threads
+ def has_pending(self):
+ """ Check if any data still needs indexing.
+ This function must only be used after the import has finished.
+ Otherwise it will be very expensive.
+ """
+ with connect(self.dsn) as conn:
+ with conn.cursor() as cur:
+ cur.execute("SELECT 'a' FROM placex WHERE indexed_status > 0 LIMIT 1")
+ return cur.rowcount > 0
+
+
def index_full(self, analyse=True):
""" Index the complete database. This will first index boundaries
followed by all other objects. When `analyse` is True, then the
@pytest.fixture
def index_mock(monkeypatch, tokenizer_mock, init_status):
mock = MockParamCapture()
- monkeypatch.setattr(nominatim.indexer.indexer.Indexer, 'index_boundaries', mock)
- monkeypatch.setattr(nominatim.indexer.indexer.Indexer, 'index_by_rank', mock)
+ monkeypatch.setattr(nominatim.indexer.indexer.Indexer, 'index_full', mock)
return mock
with pytest.raises(IndexError):
self.call_nominatim()
- assert index_mock.called == 4
+ assert index_mock.called == 2
def test_replication_update_continuous_no_change(self, monkeypatch, index_mock):
with pytest.raises(IndexError):
self.call_nominatim()
- assert index_mock.called == 2
+ assert index_mock.called == 1
assert sleep_mock.called == 1
assert sleep_mock.last_args[0] == 60
projects / nominatim.git / commitdiff