Realm/package registry & query overhaul

[Kouteki]

Current state

There's no easy way to see a list available realms and packages, making it harder to understand and utilize existing code to build my dapp.

Desired state

As a user ISBAT easily review a list of available realms and packages, so that I could build my dapp faster and easier, with a better end result.

AC

• Build capability to list available packages and realms, as mentioned in Gnoweb: list available realms and packages in the HOME #997
• Implement efficient indexing to list available packages, avoiding O(n) complexity, as mentioned in [META] Coversion layer #3264
• Support pagination
• Stretch goal: filtering/sorting/categorization (no criteria set yet)

[Kouteki]

Other related issues (likely outdated):

• [gnoland] Add nspkgs (namespace packages) ABCI query #2383
• Standardizing and organizing gno packages by categories and publishers #3460
• Implement efficient indexing #3268

[matijamarjanovic]

Hello, I have started working on a possible solution - I scraped the /r directory on github using a scraper written in Python. From the scraped realm set I have removed all test files and have fed each realm's code to ai to get a short description of the realm. The scraped information should then be sent to a registry realm that will store that information.

I understand that this is not ideal but it might prove to be useful. In the future it could be used to manually submit realms with a human written description. Also the transaction with a 20-30k character long realms with descriptions would be pretty expensive but I think this is not a problem since it would be only the time when existing realms and packages are input.

Please let me know if this approach provides a solution at all and whether should I continue working on it. Here's a screenshot of how this might look (before making it pretty and introucing pagination, filters and such):

(keep in mind that the preprompt still needs working on, this is far from final)

PR to the registry so anyone who wants can give a check. I'll also upload the file that scraper generated with the descriptions so people can check how acurate they really are

[thehowl]

Please note: this issue should be concerned predominantly with creating an RPC endpoint to query a list of on-chain packages, possibly with filtering and pagination.

A scraper to index realms is a good approach for a third party service, but it should not be part of our core offering at least for now IMO

[leohhhn]

@thehowl just a point; we agreed on making it an ABCI query instead of a full RPC endpoint.

[thehowl]

yes, apologies, abci query, just mixed up the terminology

[matijamarjanovic]

Please note: this issue should be concerned predominantly with creating an RPC endpoint to query a list of on-chain packages, possibly with filtering and pagination.

Noted, @leohhhn explained the issue to me in greater detail earlier today. Having that said I will be closing the PR on the registry

[moul]

I recall that @gfanton had this on his plate at some point, as the first user of this new query will be gnoweb. @gfanton, do you have an update regarding this?

[gfanton]

I need this to be able to list paths by the prefix gnoweb, so it's definitely on my to-do list. I've started a PoC, but I was limited due to the current store implementation, where there is no way to get a package other than iterating over all of them. Once I finish my current priority, I will address this shortly.