pelias dashboard shows different stats, planet build is not completed.

[imran-5]

Hey team!

I have built two times the planet build with Pelias/docker without any error.

---

My planet build dashboard stats results.

---

Geocoder.earth Pelias build dashboard stats results.

https://pelias-dashboard.geocode.earth/pelias

there is a huge difference in the database size and other values.
I tried acceptance-test, fuzzy-test, fuzzy-tester some of the results missing.

how I can improve the results to make the perfect database??

[missinglink]

Can you please provide more detailed information about the test failure you're seeing from the acceptance-tests?

[w0pr]

Can you please provide more detailed information about the test failure you're seeing from the acceptance-tests?

I had similar results from a planet build started last week, see this and following posts on gitter:

Aggregate test results
Pass: 383
Improvements: 18
Fail: 90
Placeholders: 0
Regressions: 218
Took 26960ms
Test success rate 68.45%

FATAL ERROR: 218 regression(s) detected.

I omitted the prepare polylines stage and used the download from geocode.earth:
https://data.geocode.earth/osm/2019-13/planet-latest-valhalla.polylines.0sv.gz

[orangejulius]

Hi @w0pr,
I believe this is at least partially due to an issue in the Who's on First bundle data. As you can see here the "latest" bundles are sometimes smaller than older releases.

The Who's on First project is in the process of switching to distributing data via SQLite databases, and the Pelias project is also going to move to support only the SQLite distributions from WOF fairly soon (see pelias/whosonfirst#460 for updates on that task). In the meantime, I think you'll get better results if you set imports.whosonfirst.sqlite to true in your pelias.json file.

[w0pr]

The Who's on First project is in the process of switching to distributing data via SQLite databases, and the Pelias project is also going to move to support only the SQLite distributions from WOF fairly soon (see pelias/whosonfirst#460 for updates on that task). In the meantime, I think you'll get better results if you set imports.whosonfirst.sqlite to true in your pelias.json file.

Yes, I was planning to do this anyway, as soon as I read about it in another issue, since my very first import had a few downloads missing, which I did not notice.

Will this lead to the correct sqlite DBs to be downloaded in "pelias download wof" stage or do I have to do this myself?

I also extracted polylines with valhalla myself, since the geocode.earth download is from March this year.

I guess I don't need to re-run prepare interpolation as well with latest data? Took several days last time, but I really want to get the best results out of this.

[w0pr]

In the meantime, I think you'll get better results if you set imports.whosonfirst.sqlite to true in your pelias.json file.

Will this lead to the correct sqlite DBs to be downloaded in "pelias download wof" stage or do I have to do this myself?

Answering my own question here: Yes, setting sqlite to true in the imports section of pelias.json will download sqlite DBs on "pelias download wof" command.

[missinglink]

the interpolation build does not use whosonfirst data, so there is no need to re-run it.

[imran-5]

I have made a little progress after tweaking.

"importPostalcodes": true,
"importVenues": true,
"sqlite": true

---

#Geocoder.earth Pelias build.

##My planet build

[missinglink]

FYI:

• elastic: add two new elasticsearch commands #145
• Do not import both whosonfirst-data-latest.db and whosonfirst-data-admin-$$-latest.db whosonfirst#469

are you able to confirm if your build is affected by these issues?

• admin-us and admin-rs SQLite distributions are not being updated whosonfirst/go-whosonfirst-dist#17 (USA has 50 States + DC (placetype: region x51))
• Some localities are missing whosonfirst-data-locality-1542073707 extract whosonfirst/go-whosonfirst-dist#9 (Missing localities)

[missinglink]

We are not importing WOF venues for Geocode Earth, which accounts for the discrepancy there.
There have been fluctuations in the OA file size which likely accounts for the differences in the address count.

[w0pr]

This is my latest planet build with own polylines extract using valhalla.
Any shortcomings seem stem from missing source/data from WOF or OA downloads; hopefully they get sorted out soon:

 "imports": {
    "adminLookup": {
      "enabled": true
    },
    "geonames": {
      "datapath": "/data/geonames",
      "countryCode": "ALL"
    },
    "openstreetmap": {
      "download": [
        { "sourceURL":
"https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf" }
      ],
      "leveldbpath": "/tmp",
      "datapath": "/data/openstreetmap",
      "import": [{
        "filename": "planet-latest.osm.pbf"
      }]
    },
    "openaddresses": {
      "datapath": "/data/openaddresses",
      "files": [
      ]
    },
    "polyline": {
      "datapath": "/data/polylines",
      "files": [ "my_extract_valhalla.0sv" ]
    },
    "whosonfirst": {
      "sqlite": true,
      "datapath": "/data/whosonfirst",
      "importVenues": false,
      "importPostalcodes": true
    },
    "interpolation": {
      "download": {
        "tiger": {
          "datapath": "/data/tiger"
        }
      }
    }