Interface for converting zxy to range header

[kochis]

Hi there,

I've been trying to understand how the Range header is generated for a given tile, and was wondering if there's an interface, or way to use PMTiles.js to take z/x/y and get just the Range header, without handing off the full request to PMTiles.

The use-case here, is having nodejs as a proxy for serving the tiles directly from S3 or local disk, so there wont be a source provided for PMTIles.

Let me know if that makes sense, happy to provide more info if needed.

Thanks!

[tgecho]

I was looking for this exact thing. I have a half baked forked version that extracts the byte offset calculation logic out of getZxyAttempt and into a getZxyEntry method. It still needs a bit of cleanup, but I'd be happy to put up a PR if @bdon is open.

[bdon]

Yeah, I think it would be refactoring getZxy to skip the part where the tile is fetched. However you do need to also store off the information about what compression method the tile uses to interpret the bytes in the entry. Can look at a PR!

[bdon]

Right, there isn't an interface for that right now, but it would make sense to add something like that - I'm trying to understand the use case here though.

There is still a source, since you need to get the header and directories to fulfill a request; we'll just want to skip the part of actually retrieving and maybe decompressing the tile (but how is what you're trying to do in NodeJS different?)

An implementation of an S3 source exists here, and maybe this deserves to become part of the NPM package:

https://github.com/protomaps/PMTiles/blob/main/serverless/aws/src/index.ts#L87

[tgecho]

I'm trying to understand the use case here though

This enables use in a Lambda@Edge function. You can do the metadata lookup with explicit requests (in an origin request function), then rewrite the actual request to add in required range header. This way the Lambda code never touches the actual tile contents (with the associated gzip/base64 overhead).

An implementation of an S3 source exists here, and maybe this deserves to become part of the NPM package

+1

[bdon]

So that would be leveraging HTTP rewriting behavior specific to Lambda@Edge, right? It's not obvious to me that deployment strategy is significantly faster or more cost effective than the plain Lambda + Function URLs here: http://protomaps.com/docs/cdn/aws

Lambda@Edge is a bit more expensive than plain Lambda, and running in multiple Regions is detrimental because all invocations need to talk to a single-region S3 source anyways. Another big downside is Lambda@Edge rollouts take a long time since they need to be replicated through cloudfront.

[tgecho]

So that would be leveraging HTTP rewriting behavior specific to Lambda@Edge, right?

Correct

It's not obvious to me that deployment strategy is significantly faster or more cost effective than the plain Lambda + Function URLs

That's fair! I'm still early in my exploration, but it seems like there is a noticeable overhead to the base64 encoding (and re-compression, but that could be addressed separately) required to return binary data from the lambda function. I need to back this up with actual test data.

[bdon]

I've observed that a single same-region request between Lambda and S3 can take 40-100 milliseconds (this scales with lambda RAM, my recommendation is 512MB). The base64 encoding should take some minor fraction of this. These only matter on cache misses, of course.

The design you proposed where Lambda@Edge returns a rewritten request could eliminate this last fetch from happening inside the lambda. The fetch would still happen somewhere else in the Cloudfront request chain, so external latency is the same, but you wouldn't be billed for it as part of the Lambda execution duration.

Lambda@Edge is triple the price of base lambda, so I have a hard time seeing it come out ahead. But let me know what you find out!

[bdon]

This new feature might eliminate the base64 encoding step for Lambda: https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/

[kochis]

I'm trying to understand the use case here though.

I think for us, we already have application servers that can handle the requests, and by proxying through our service we can leverage our existing systems for things like auth, rate limiting, monitoring, etc... as well as the ability to easily change which tiles are being used. From a cost / maintenance perspective, this also seems preferable to using lambdas.

you need to get the header and directories to fulfill a request

I'm still relatively new to PMTiles, so perhaps I'm missing some of the pieces here, mainly how the directories fit in. I assumed it was just the range headers that would return the compiled data (in our case, vector tiles in .pbf) that could be send back to the client.

Will take a look at that S3 source, that seems like a good approach 👍

[bdon]

The directories are there to provide the offset/length you need to construct the range request.

It sounds like the PMTiles library along with that S3 Source can accomplish what you need already, with the small hitch that you may need to re-compress the tiles going out of your node server. #96 will eventually allow optimizing this.

If your application is already "serverful" then accessing MBTiles or PMTiles straight from disk will be more economical than S3, likely.

[tomeronen]

kochis i have a very similar use case. A serverful application that used to download mbtiles into disk and then serve them. This brought a lot of complexity because of big amount of mbtiles (can get to around 20,000 diffrent mbtiles). PMtiles looks like a great alternative. will update if i was successful :)

[kochis]

Just to follow up, I wrote an article detailing how I solved this in case anyone is interested: https://ckochis.com/serve-protomaps-tiles-with-nodejs

[bdon]

Great article, thanks for following up!