optimization after first data download

[stefanocudini]

hi,

I was trying to optimize my pelias installation by the clone this repo.
I wanted to try to figure out which data in the directory /data are cached or are data reused in future imports, I need to understand how to optimize disk space

For example, I understand that whosonfirst data is a files geojson that is subsequently loaded into the sqlite database, so after executing the command
pelias download wof
I can free up the space used by the directory /data/whosonfirst/data/*
then after imports of other datasource.. pelias will use the sqlite databases for adminLookup right?

[orangejulius]

Hi @stefanocudini,
I think it works the other way around. In the case where you're specifying importPlace to restrict to a subset of WOF data, the sqlite database is used to create a partial extract on the filesystem in the /data/whosonfirst/data directory.

In the future, we're probably going to move away from using that filesystem layout and go directly to sqlite, which you can try out today by setting "sqlite": true in pelias.json. That will save some disk space. We've also found that the high number of files and directories created by the WOF disk structure can itself create performance issues.

[stefanocudini]

@orangejulius I updated the wos module to the latest version and it seems to me that now only sqlite files are present, can you confirm that sqlite is now the default: true?

good job!

[Joxit]

Hi @stefanocudini, yes since pelias/whosonfirst#487 only sqlite is supported by pelias 😄

[stefanocudini]

@Joxit my reference data before updating this plugin was more than 1 year ago