Serving bundled tiles.

[FunMiles]

I have written some code to extract tile files from a planet.mbtiles into a directory structure z/y/x.pbf.
The resulting directory can be used to serve files directly instead of obtaining the data from the planet.mbtiles database.
I also have used a browser-side database to cache the tiles, making it much faster to retrieve them after the initial load and reducing the network traffic. Note that MapLibre allows to set a cache size so that caching is handled directly by the MapLibre library. Doing it on the app side however does give us more control on what to keep and allows this:

The next step could be to serve some zip file of tiles that can be stored in the browser-side database. Such a feature can be very useful in several situations. E.g. for an app that wants to cover just one country or region, the zip file could contain all the tiles just for that region. I believe @BenOnTrack could make use of this?

[BenOnTrack]

Hey, i assume you're using sth similar to mbutil to extract the tiles prior to query them. It might ease and accelerate the process but how do you do if you want to update the initial mbtiles? I believe mbtiles have been created to avoid the need to extract them, but I may be wrong as I'm still learning too.

As of today tileserver can serve multiple mbtiles source at once, I believe this is what you're trying to achieve with the .zip structure. I like the idea.

Today I'm still looking at other solution since I can't manage to run your project on Android (with capacitor and adapter-node ?)

But I'll keep an eye to your Dev and keep me posted if you're trying to achieve a offline mobile app as well, we could mutually help each others

[FunMiles]

Hey, i assume you're using sth similar to mbutil to extract the tiles prior to query them.

Yes, essentially, it is a loop over the tiles of a given level. You could write a script around mbutil to do the same. I think I will slightly change the way I am doing it so it can be used on any mbtiles file rather than a full planet file. It will work better also for your use case. Just put your file as planet.mbtiles and extract zip files from it. Depending on your data size, slicing the data into multiple zip files may make it easier for the download.

It might ease and accelerate the process but how do you do if you want to update the initial mbtiles?

I don't have yet a full answer to that. I would first need to look at planetiler to see if it can create just tile updates, given the openstreetmap updates. If that is the case, just creating update zip files with the updated tiles should do the job.

I believe mbtiles have been created to avoid the need to extract them, but I may be wrong as I'm still learning too.

mbtiles files indeed can be accessed directly with sqlite3. Unfortunately, sqlite3 requires fs which I don't believe is an option on the browser. For most browser the local storage of file would be quickly exceeded by the size of the mbtiles file anyways.
However, the good news is that all modern browsers support indexedDB, which, like sqlite3, is a database. So we can stores the tiles in an indexedDB table, using [z,x,y] as a key, and the file body as a data column.

Today I'm still looking at other solution since I can't manage to run your project on Android (with capacitor and adapter-node ?)

But I'll keep an eye to your Dev and keep me posted if you're trying to achieve a offline mobile app as well, we could mutually help each others

I have not had time to make the PWA aspect of the app, as I was occupied with some authentication issues in my app. I currently have a working prototype for the zip file download system. I will work on adding the PWA aspect to it and will load a prototype on cloudflare and publish the source code in a branch of this project.

[FunMiles]

@BenOnTrack There is a new branch to the project dev/pwa that is a complete PWA with a zip file of the data for levels 0 to 2, which does not allow to zoom much, but does demonstrate how to achieve the goal of creating an installable Progressive Web App.
You should be able to install it on your Android device from Chrome's menu:
https://funmiles.github.io/Planetiler_MapLibre_Sveltekit/
Once installed and you've run it at least once to download the zip file, it should be able to run offline.

None of it would have been possible without the help of @userquin who knows quite a lot more than I do about the VitePWA plugin.
Give it a try and let us know if that goes in the direction of what you want to achieve.

[BenOnTrack]

Let me have a look this week and I'll sure come back to you! Thanks for the heads up!

[BenOnTrack]

I took some time to review the dev/pwa branch, it looks really good!
I may be wrong, but since PWA allows you to build a unique app that works on both the web and your phone, I'm wondering if that would make more sense for:

• the web version, to use tiles served by a server (your own, or a solution such as maptiler)
• the phone version, assuming it would be 100% offline, to use a folder located on the phone to store the .mbtiles or .zip ? to reach something similar to OrganicMaps, where the User can download Gb of data without any worry.

AS I have 0 background with PWA, I'm not sure if that's possible to use a folder located outside of the app itself to store the input data (for instance, you could manually push you own .mbtiles / .zip to this folder) hence removing the constraint on the size of the app itself.

On my end I'm still looking on solution that works directly with Sveltekilt and plugin such as Capacitor that does the job for you (to convert into .apk for both android and ios) but unsuccessful so far.

Happy to hear your thoughts ?

[BenOnTrack]

Hi @FunMiles , did you get any chance to look into this ^ ?

[FunMiles]

PS: The same branch has a script to extract all the tiles for a given range of levels. I have summarily documented the steps to generate another tile file. If you extract data from a limited area, you might be able to pack more than 2 levels in a reasonably sized zip file.
Beware that excessive zip file sizes are likely to cause issues.

[FunMiles]

Hi @FunMiles , did you get any chance to look into this ^ ?

I wrote a reply half asleep, and my reply did not really address your questions. So here is an edited version that better answers your questions:

• the web version, to use tiles served by a server (your own, or a solution such as maptiler)

The good news is that I believe it is possible to test if the PWA is installed or not. I will rephrase your bullet point as being an non-installed version. Yes, you could do that, have the non-installed version work from server-based data.

• the phone version, assuming it would be 100% offline, to use a folder located on the phone to store the .mbtiles or .zip ? to reach something similar to OrganicMaps, where the User can download Gb of data without any worry.

There is no portable file-system accessible by a browser app. One can store data in localStorage but as you'll see in the docs it is not standardized. So the alternative that has almost no limitation is the indexedDb. I have implemented the indexedDb in the app I linked above.

However, since I wrote that, I also went back to planetiler's page and saw the mention of a different file format called pmtiles.
It seems you can serve such files from a service like AWS S3, or, cheaper, Cloudflare R2 (no egress fees) directly to maplibre_gl.
maplibre_gl itself can be told it can cache tiles, but I don't know if you can force it to cache all the tiles for a given bounding box or al the tiles in the pmtiles file.
They do have docs and code on how to serve the content of the file from AWS and Cloudflare. I will be looking into it.

One more thing I need to mention is an underlying technology used in the PWA: the service-worker.
The service worker is a layer between the browser and the Internet. It downloads all files needed to run the app offline and answer the requests for those files without going to the network. It can also cache files that are dynamically accessed. I am wondering if it could be told to download all the tiles you want and thus make it even easier to have a unified approach between installed and uninstalled PWA. The uninstalled would still go to the network and the installed one would precache all the tiles and answer from the service-worker's side.

I will be investigating this approach. The trouble I see is that if the server is AWS S3 (or cloudlfare without the serverless worker), then the browser must be requesting sub-parts of the pmtiles file. I do not know if workbox, the Google tool used in the service-worker can handle that.

[BenOnTrack]

Thanks for the feedback - I guess (you can tell) I'm new to PWA and probably find it "safer" to use native libraries, especially if you're looking to share the app with other people. But to be honest I agree and believe that it's probably the best way to go, if Capacitor (or other similar alternatives) are not yet ready. I'm not interested to learn how to code native apps and like the idea of having a single code for both web and ios/android.

[userquin]

Creating a native integration requires minimal knowledge: for Android Java/Kotlin, for iOS/macOS Swift/Objetive-C. About Tauri (Rust), I have no idea.

[FunMiles]

I've had an app on Google store with 7 million downloads. To be honest, dealing with native app stores was for me a big pain point. They keep having changes you have to adapt to. Now they emailed me I need to reserve a slot for confirming who I am.
This whole experience is one key reason why I am happy to have PWA as an alternative without all this pain. The only downside to PWA is that not all APIs are available. But that is slowly improving. For a map application, I think that almost all you could dream of is available. What's missing is thing like 'fence' API to let the app know about movements when it is in the background.
Also, don't forget that maintaining 2 versions, one for Android, one for iOS and dealing with the app store's quirkiness is double the misery I've had to deal with.
Finally, if your intent is to have more people using your app, know that surveys show a lot of people (myself included) are tired of having to install a new app for absolutely everything. They are much more likely to open a web site, and then you can gently guide them to put it on their home screen, as Apple calls it, or install in, as Google Android calls it from the web page without having to go to the store.

[userquin]

That's the main reason I started to mantain vite-plugin-pwa, the cost having native/store support is a pain and money.

With Android at least you can develop in any OS and free, for iOS/macOS you need a macOS machine and a license, WTF!.

[BenOnTrack]

Hey guys, don't get me wrong but I'm totally on the same page here - as mentioned above, I have zero intention to learn native coding. I guess the only tradeoff I'm still running at the moment is between PWA and Capacitor (sth like https://github.com/Yermo/maplibre-gl-capacitor-offline -> but I'm not sure all imported libraries are fully open source).

As for pmtiles, I've been looking into that as well, and this svelte-maplibre community has some interesting dev as well dimfeld/svelte-maplibre#59.

[FunMiles]

Going back to the original subject of bundled maps, I have been digging deep into pmtiles. The author of that project (actually several projects), @bdon, and other contributors, have done a fantastic job. Following my deeper diving, it seems that I am going to be doing something that is a lot more involved but that hopefully will be more efficient than my first plan. It means, however, that it will take me some time to complete it.

If you read the specs for the V3 file format, if the file is clustered, it should be possible to efficiently extract arbitrary sub-parts of a pmtiles file, including a planet file. That, in effect is similar to what Google maps allows you to do when you want to download a map for offline usage. Initially, I will not implement that, but will implement a streamed download that allows to go down deeper in level (z index) as the download progresses.

[FunMiles]

I have uploaded a working demo code in the branch dev/pmtiles_pwa. It downloads the file in chunks of 1MB using a range download.
I have removed the data file but the README.md should explain how to generate it.
I wanted to upload a demo with a 90MB datafile on vercel, but due to my not deleting unnecessary files, I've run out of vercel upload bandwidth for the next 24h. I may try to upload again tomorrow.

Note that there is room for a lot of improvements. For example, the download could be done as a single file download without the Range option and then, using the streaming API for fetch, the incoming data could be divided in chunks of 1MB for saving to the database.
The 1MB chunk size is arbitrary but, because the code will only collect at most 2 chunks, any tile that would be more than 1MB could fail if it does not align well with 2 1MB chunks. Any PR fixing this for an arbitrary number of chunks is welcome.
It is also possible to modify the source to allow displaying the map before it is fully downloaded by redirecting to a regular ranged fetch when the data is not yet in the database.

I'll leave that as an exercise 😄
Let me know if you have any issue trying this branch out.

[BenOnTrack]

Hi, I confirm it's working great! I'll need to dig in the code to understand how the chunk work to see if I can help.

Also I've been playing around with the mbtiles/pwa version and I was wondering if that would have worked with multiple mbtiles sources (since they would end up having the same file name z/x/y.pbf). It looks like your pmtiles version stores the source url, so I'll run some tests to see if that would work with multiple incoming .pmtiles (for instance, one per country, or one per category - landcover, poi, road, ...).

I'm super excited to transition to pmtiles, and it already looks like an even smoother experience! I was already impressed by the mbtiles, so I can only be more enthusiastic.

Amazing work!

[FunMiles]

For the zip-based version, you could load several files in the same database as long as the layers they contain are the same, in such case, the content of z/x/y.pbf is independent of the file they come from.

For the pmtiles, I decided to add the url as an indexable entry on the database. So yes, you can load more than one. The trivial use is to switch from one file to another. More refined usage could be to have files with different layers, like with things like poi as you mentioned. In that case, you can modify the custom: adapter code AND the style file to address that. Right now, the code modifies the style file to have custom://<url>/tiles/z/x/y.bpf as the source for vector tiles.
You could have more sources such as custom://<url>/poi/z/x/y.pbf or custom://<url>/road/z/x/y.pbf and check what comes before the /z with a regex and pick from one of several pmTiles tile source object.
In fact, right now, when zoomed out, there are some image tiles (.png type) being retrieved from the network (see natural_earth_shaded_relief in the mystyle.json file). Those files work in the PWA only because they are being cached automatically.
They should really be put in a relief.pmtiles file as well. Maybe in a more efficient format than .png.

On a final note, right now, the database retrieval and all the code to obtain an ArrayBuffer is done on the main GUI thread. It seems to be sufficiently quick than it does not impede the GUI smoothness. If the code were to become more complex and require more compute time/power, it should be moved to a worker (that's how javascript gives you multi-threading). I have not done any measurement, but it seems so smooth that it probably does not take much time.

[BenOnTrack]

I've identified 2 use cases (in my previous project using mbtiles - but it should still apply to pmtiles)

Use Case #1

You have multiple inputs (pmtiles) files that have the same layers. They should all be located under the same maplibre source (for instance openmaptiles)

sources": {
      "openmaptiles": {
         "type": "vector",
         "tiles": [
            "@origin@/input1/{z}/{x}/{y}.pbf",
            "@origin@/input2/{z}/{x}/{y}.pbf",
            ....
         ],
      },

In this case input1 and input2 have the same layers source.

For instance input1 is Colorado state and input2 is California state (both with the same layers - for instance landcover, road,...-> basemap)

Use Case #2

You have multiple inputs (pmtiles) files that have different layers. They should have their own maplibre source

sources": {
      "openmaptiles1": {
         "type": "vector",
         "tiles": [
            "@origin@/input1/{z}/{x}/{y}.pbf"
         ],
      },
     "openmaptiles2": {
         "type": "vector",
         "tiles": [
            "@origin@/input2/{z}/{x}/{y}.pbf"
         ],
      }

In this case, openmaptiles1 and openmaptiles2 will be called according to their source layer that need to be styled.

For instance input1 has a poi-restaurant (but not bar) layer, and input2 has a poi-bar layer (but not restaurant).

Thinking about it, this usecase2 should cover both need.

sources": {
      "openmaptiles1": {
         "type": "vector",
         "tiles": [
            "@origin@/input1-1/{z}/{x}/{y}.pbf",
            "@origin@/input1-2/{z}/{x}/{y}.pbf"
         ],
      },
     "openmaptiles2": {
         "type": "vector",
         "tiles": [
            "@origin@/input2-1/{z}/{x}/{y}.pbf",
            "@origin@/input2-2/{z}/{x}/{y}.pbf"
         ],
      }

As of today the unique pmtiles is named source.pmtiles and is stored under indexdb and fetched via the custom//tiles/... endpoint

I'll need to dig into the code to understand how to make the 2 above use cases work, while making sure all tiles from all inputs stored within indexdb are correctly fetched.

Would the original /tiles/ endpoint fit all needs, should it be replaced by /input1-1, /input1-2, /input2-1,... and how to match those with their input files input1-1.pmtiles, input1-2.pmtiles, input2-1.pmtiles

Maybe the solution would look more like:

sources": {
      "basemap": {
         "type": "vector",
         "tiles": [
            "@origin@/basemap/{z}/{x}/{y}.pbf"
         ],
      },
     "poi-restaurant": {
         "type": "vector",
         "tiles": [
            "@origin@/poi-restaurant/{z}/{x}/{y}.pbf"
         ],
       "poi-food": {
         "type": "vector",
         "tiles": [
            "@origin@/poi-food/{z}/{x}/{y}.pbf"
         ],
      }

but in that case where does the code understands how to match the initial source pmtiles to its indexdb storage and endpoint call :

basemap_colorado.pmtiles and basemap_california.pmtiles would be stored in indexdb under /basemap and fetch via /basemap/z/x/y/

...unless I'm completely out of scope ?

The objective is to have pmtiles inputs, and based on their layers (known in advance), call them within the appropriate maplibre sources to prevent any styling issues.

I believe it should be feasible, and that would ultimately answer one of my biggest need.

[BenOnTrack]

I've managed to fetch multiple sources at once, based on your feedback.
Basically, as of now, I create a PMTiles instance for each .pmtiles file

Then the custom protocol checks multiple Promises, for each of those sources, and return the tile if any.

...

function handleTileRequest(match, pmTiles: PMTiles): Promise<any> {
        const z = parseInt(match[1]);
        const x = parseInt(match[2]);
        const y = parseInt(match[3]);

        return pmTiles.getZxy(z, x, y);
    }

...

 maplibre.addProtocol('pmtiles', (params, callback) => {
            const matchBasemap = params.url.match(/\/basemap\/([0-9]+)\/([0-9]+)\/([0-9]+)\.pbf/);
            const matchRoute = params.url.match(/\/route\/([0-9]+)\/([0-9]+)\/([0-9]+)\.pbf/);

            Promise.all([
                matchBasemap && handleTileRequest(matchBasemap, basemapPMTiles),
                matchRoute && handleTileRequest(matchRoute, routePMTiles)
            ]).then(([tileBasemap, tileRoute]) => {
                if (tileBasemap) {
                    callback(null, tileBasemap.data, null, null);
                } else if (tileRoute) {
                    callback(null, tileRoute.data, null, null);
                } else {
                    console.log('No tiles fetched for either basemap or route');
                    callback(null, null, null, null);
                }
            }).catch(error => {
                console.error(`Error fetching tiles: ${error.message}`);
                callback(error, null, null, null);
            });

            return { cancel: () => {} };
        });

while the style follows the same pattern

"sources": {
      "basemap": {
          "type": "vector",
          "tiles": ["@pmtilesOrigin@/basemap/{z}/{x}/{y}.pbf"]
      },
      "route": {
        "type": "vector",
        "tiles": ["@pmtilesOrigin@/route/{z}/{x}/{y}.pbf"]
    }
  },

I'm not 100% sure it's the right way to do, especially if I'm intending to have a .pmtiles per state/country (similar to OrganicMaps) - not sure it would be smooth if you have 100's of sources for instance.

Another way would be to merge the .pmtiles data into the same PMTiles instance, but again that'd be harder to handle update/delete of specific sources.

[BenOnTrack]

I also have a question regarding the persistence of indexdb through a browser cache clearance ?
If the user, for any reason, decides to clear their browser cache, would that also automatically clear the indexdb or is there a way to prevent that from happening ?

[FunMiles]

Clearing caches does not affect the indexedDB. Only explicitly requesting to remove the app's data. That's akin to requesting to uninstalling an app. It is not something you should worry about. If the user wants to remove an app, it's their choice and there's nothing you can or should do about it.

[BenOnTrack]

Thanks. I finally had some time to dig into indexedDB and understood that it was set per origin (I thought at some point it was a global browser storage, that anyone could easily delete, but it's not). Now that you've been adding the "source" url in indexedDB I assume it would ease file management such as "update" or "delete" of a source.