driver fetches flashloan hints for orders #3215
Assignees
Comments
MartinquaXD
changed the title
squadgazzz
added a commit
that referenced
this issue
Jan 27, 2025
# Description As part of the flashloans support, the driver needs be able to retrieve the order's full app data in order to send flashloans hints to solvers(#3216). This PR introduces an app-data retriever that works as follows: - Check whether the app data already exists in the cache and returns it. - Otherwise, it sends a corresponding request to the orderbook API to fetch the app-data. The request is sent using `BoxRequestSharing`, which helps to avoid request duplication. - Since the full app data is an optional field in the `Order` struct, responses with `404 Not Found` are also cached as `None`. There is a small amount of orders with empty full app data in the DB, but they still exist. - In case of any app-data fetching error, the error gets logged. ## Implementation details and considerations - The LRU cache([moka](https://github.com/moka-rs/moka)) is used because, according to the mainnet DB, there is only 2% unique `app_data` among all the orders, so there is no need for a TTL cache since the cache is expected to be hit for the majority of orders. - Based on the DB data, the average full app data size is ~800 bytes. The LRU cache has `2000` capacity, approximately equal to ~1.5MB of memory. - Once app data is fetched, it needs to be stored in the `domain::Order` struct. To avoid creating new order structs, the `AppData` is converted into an enum, which gets updated accordingly. - Already discussed with @MartinquaXD. The full app data is cached even though the flashloan part is only expected to be used further. This can be reconsidered either in this or future PRs. - New metrics to better understand how much time is spent fetching the app_data. Also, added the same for the balance fetching and total preprocessing just for reference. ### Rate limiting **_The following is only valid for colocated solvers since the API is not rate-limited for the services within the same k8s cluster._** <img src="https://github.com/user-attachments/assets/91533bcc-5d7d-4a52-9549-ff7024a7c352" width="400"> <details> <summary>SQL query</summary> ```sql WITH unique_app_data_per_auction AS ( SELECT oe.auction_id, COUNT(DISTINCT o.app_data) AS unique_app_data_count FROM order_execution oe INNER JOIN orders o ON oe.order_uid = o.uid GROUP BY oe.auction_id ), auction_count_by_unique_app_data AS ( SELECT unique_app_data_count, COUNT(*) AS auction_count FROM unique_app_data_per_auction GROUP BY unique_app_data_count ) SELECT unique_app_data_count as unique_app_data_per_auction, auction_count FROM auction_count_by_unique_app_data ORDER BY auction_count DESC; ``` </details> This image represents the following data: The left column shows how many unique app data entries have a single auction, whereas the right shows how many auctions have the corresponding amount of unique app data. The orderbook API RPS is 5. Only 0.05% of auctions have more than 5 unique app data entries. That means there is at most a 0.05% chance that a driver hits the RPS for a single auction, where the actual probability is even lower since some of the hashes will more likely be already cached. That is why the current implementation doesn't contain a rate-limiting mechanism. # Changes - [ ] The app-data retriever. - [ ] A new required driver cli argument `orderbook-url`. - [ ] A mocked orderbook for driver tests that always returns 404. This should be improved in #3216, where driver tests are expected to be added. - [ ] A config that allows switching the feature on/off, as well as configuring the `app-data` cache size. The feature is disabled by default. ## How to test All the current tests pass. New driver tests can be added only once the driver starts sending the collected data to solvers(#3216). ## Related Issues Fixes #3215
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A user is able to create an order with arbitrary metadata by attaching it to the
appdata.To build a system that is easily extensible and doesn't require every piece of metadata that protocol is supposed to understand to the auction we'll make it the responsibility of the driver to fetch the
appdata, parse, and understand it.That means the driver should fetch
/v1/app_data/<app_data_hash>for every order and cache them in memory. The content associated with the hash will never change so we only need to worry about evicting entries that didn't get used for a long time (i.e. orders that already got settled).Also this cache should be shared across all connected solvers as this data is not solver specific.
Finally the cache should not store the raw appdata JSON strings but rather parsed rust structs so that the driver can already use the contained metadata easily.
Fetching the content of missing appdata hashes can probably be done as part of the auction pre-processing (since this is done once per auction for all solvers to share) but please use good judgement to decide on the actual implementation.
Afterwards the parsed
AppDatashould be added to the domain Order struct.For parsing the
AppDatatheapp-datacrate should be used. For this issue it's not necessary that theapp-datacrate already is able to parse the new data. Once that is implemented the driver will be able to parse it right away.The text was updated successfully, but these errors were encountered: