Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Asset System - Prepare merge into main #1292

Merged
merged 113 commits into from
Apr 2, 2024

Conversation

sea212
Copy link
Member

@sea212 sea212 commented Mar 27, 2024

What does it do?

  • Update readme and changelog_for_devs
  • Get code in a mergeable (into main) state

What important points should reviewers know?

Is there something left for follow-up PRs?

What alternative implementations were considered?

Are there relevant PRs or issues?

References

sea212 and others added 30 commits November 6, 2023 09:03
* Update versions (#1168)

* Add updated weights from reference machine (#1169)

* Add license header

Co-authored-by: Chralt <[email protected]>

---------

Co-authored-by: Chralt <[email protected]>
filter admin functions for main-net
* add debug assertions for missing slashes

* place debug_assert for unreserves
add some verify checks to court
…order book markets (#1185)

update contracts call filter
* fix court assertion for resolve_at

* remove unnecessary variable

* mirror mock and actual impl for DisputeResolution
* implement trusted market close

* remove unnecessary benchmark helper function

* Update zrml/prediction-markets/src/lib.rs

Co-authored-by: Malte Kliemann <[email protected]>

* remove unnecessary function

* check market end

* check auto close

* add invalid market status test

---------

Co-authored-by: Malte Kliemann <[email protected]>
* modify events for indexer

* gracefully handle panicers

* handle binary search by key error

* improve style
* handle own existential deposit errors

* use require_transactional

* correct benchmark and test min bet size amounts
* Replace `bmul` and `bdiv` with traited versions

* Restructure directories

* Replace `saturating_*` from neo-swaps

* Fix formatting

* Restructure zrml-swaps math

* Implement and test `b*`

* Fix formatting

* Use new math in orderbook-v1

* Replace checked multiplication with new math

* Use correct rounding in neo-swaps

* Add docs

* Update licenses

* Remove `fixed` module from `primitives`

* Fix formatting

* Update primitives/src/math/fixed.rs

Co-authored-by: Harald Heckmann <[email protected]>

---------

Co-authored-by: Harald Heckmann <[email protected]>
* add env_logger and add force-debug feature

* taplo fmt, fix copyrights

* Update zrml/styx/src/mock.rs

Co-authored-by: Malte Kliemann <[email protected]>

* Update zrml/rikiddo/src/mock.rs

Co-authored-by: Malte Kliemann <[email protected]>

* update comment to clarify logging

---------

Co-authored-by: Malte Kliemann <[email protected]>
* inflate defensively

* add tests

* Update zrml/court/src/tests.rs

Co-authored-by: Malte Kliemann <[email protected]>

* fix test

* fix test name

---------

Co-authored-by: Malte Kliemann <[email protected]>
* integrate market creator fees into orderbook

* edit tests

* update tests

* modify tests

* avoid order side

* Update primitives/src/traits/distribute_fees.rs

Co-authored-by: Malte Kliemann <[email protected]>

* remove asset and account from get_fee api call

* take base asset fees from fill_order

* adjust orderbook for taking fees in fill_order

* adjust without order side for fill_order

* adapt to avoid concept of order side

* adapt benchmarks, tests and weights, restructure

* use DispatchResult

* remove unused import

* do not adjust maker amount for place_order

* correct fuzz of orderbook

* check if maker_amount is zero

* fix order book name in benchmarks

* use remove instead of cancel order book

* correct order book test names

* update READMEs

* fmt

* add tests

* use minimum balance as min order size

* Update zrml/orderbook/README.md

Co-authored-by: Malte Kliemann <[email protected]>

* Update zrml/orderbook/README.md

Co-authored-by: Malte Kliemann <[email protected]>

* Apply suggestions from code review

Co-authored-by: Malte Kliemann <[email protected]>

* prettier orderbook md

* remove comments from benchmarks

* fix tests and benchmarks readibility

* use order book instead of orderbook

* clarify error message

* clarify comments

* rename is to ensure

* error for repatriate

* fix unnecessary clone

* correct mocks behaviour

* improve test

* improve test of near full fill error

* use turbofish syntax

* add filled_maker_amount to event

* split tests

* combine two functions, add docs

* abandon get_fees

* fix order book tests and mock

* remove check for impossible behaviour

* fix toml and benchs

* prepare migration

* add migration for order structure

* return zero fees if transfer fails

* delete unnecessary assert

* fix naming

* fix naming the second

* fix maker fill description

* add scoring rule check to remove order

* fix post_upgrade

* fix terminology

* Update zrml/orderbook/src/migrations.rs

Co-authored-by: Malte Kliemann <[email protected]>

* use storage root check in migration test

* Update zrml/orderbook/src/lib.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/orderbook/src/lib.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/orderbook/src/lib.rs

Co-authored-by: Harald Heckmann <[email protected]>

* delete debug_assert

---------

Co-authored-by: Malte Kliemann <[email protected]>
Co-authored-by: Harald Heckmann <[email protected]>
* Replace `bmul` and `bdiv` with traited versions

* Restructure directories

* Replace `saturating_*` from neo-swaps

* Fix formatting

* Restructure zrml-swaps math

* Implement and test `b*`

* Fix formatting

* Use new math in orderbook-v1

* Replace checked multiplication with new math

* Use correct rounding in neo-swaps

* Add docs

* Update licenses

* Remove `fixed` module from `primitives`

* Fix formatting

* .

* Rewrite math functions

* Remove training wheels

* Fix docs.pdf

* Fix quotes

* Add tests for buying

* Add tests for selling and improve error names

* Update docs

* Check adjusted amount in for numerical bounds

* Remove unused implementations

* Adjust docs

* Add stress test exploring various scenarios

* Add swap fees to stress test

* Add underscore separators

* Clean up

* Benchmark `buy` as function of `asset_count`

* Update benchmarks

* Clippy fix

* Fix benchmark tests

* Update benchmarks of zrml-prediction-markets

* Fix broken comment

* Add comment explaining benchmark spot prices

* Use clearer constants for `amount_in` in tests

* Update zrml/neo-swaps/src/traits/pool_operations.rs

Co-authored-by: Chralt <[email protected]>

* Fix botched merge

* Fix merge

* Update benchmarks

* Update zrml/neo-swaps/docs/docs.tex

Co-authored-by: Harald Heckmann <[email protected]>

* Apply suggestions from code review

Co-authored-by: Harald Heckmann <[email protected]>

* Rename `Fixed` and clean up

* Hotfix exponential function and extend tests

* Complete math test suite

* Fix broken sentence

* Remove unused imports

* Extend `ln` test cases

* Merge & fix formatting

---------

Co-authored-by: Chralt <[email protected]>
Co-authored-by: Harald Heckmann <[email protected]>
* Replace `bmul` and `bdiv` with traited versions

* Restructure directories

* Replace `saturating_*` from neo-swaps

* Fix formatting

* Restructure zrml-swaps math

* Implement and test `b*`

* Fix formatting

* Use new math in orderbook-v1

* Replace checked multiplication with new math

* Use correct rounding in neo-swaps

* Add docs

* Update licenses

* Remove `fixed` module from `primitives`

* Fix formatting

* Add liquidity tree draft

* Fix compiler error in draft

* Clean up `propagate_fees`

* Add docs

* Reorganize

* Clean up, add max iterations

* Remove migrations

* Prepare switch to bounded vector

* Use `BoundedVec`

* Use bounded `BTreeMap` and insert `LiquidityTree` into neo-swaps

* Resolve TODO

* Clean up tests

* Add migration and fix clippy errors

* Make tree depth a generic

* Make tree depth a config parameter

* Update runtime complexities

* Add benchmarking utilities

* Fix runtime complexity for `deploy_pool`

* Add missing lazy propagation

* Fix comment

* Fix error type

* Add `join` benchmarking and fix a bug in `LiquidityTree`

* Clean up benchmarks

* Fix clippy issues

* Remove unnecessary type hint

* Fix bugs in liquidity tree

* Fix comments

* Some fixes in benchmarks

* Implement `BenchmarkHelper`

* Update benchmarks to use the LT

* Clean up and format

* Add testing framework for liquidity trees

* Add first extensive test and fix a bug

* Add more tests

* Clean up test

* Add news tests and use better numerics for ratio calculations

* Make docs more explicit

* Add tests for `exit`

* Add tests for deposit fees

* Add tests for getters

* Clean up tests some more

* Finalize liquidity tree tests

* Clean up comments

* Introduce exit fees

* Rewrite `exit_works`

* Fix liquidity parameter calculation

* Make test a bit more complex

* Test liquidity shares

* Clean up tests

* Update test for destruction

* Enhance `exit_destroys_pool` test

* More cleanup

* Add test for full tree

* Add tests for `join`

* Improve test

* Test withdrawal

* Clean up the test

* Add test for noop

* Add minimum relative liquidity

* Verify that buys deduct the correct amount of fees

* Add last tests

* Add notes about the liquidity tree

* Fix benchmarks

* Fix clippy errors

* Fix benchmarks

* Do more work on benchmarks

* Fix benchmarks, deduce max tree depth

* Remove already solved TODO

* Document macros

* Remove TODO (not a good idea to edit LaTeX docs now)

* Fix `bmul_bdiv`

* Make `bmul_bdiv_*` not implemented

* Double-check that there's a non-zero check for `ratio`

* Fix formatting

* Fix taplo formatting

* Remove TODO

* Remove TODOs and fix documents

* Update primitives/src/math/fixed.rs

Co-authored-by: Chralt <[email protected]>

* Mark `SoloLp` as deprecated

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Make `bmul_bdiv` a little more idiomatic

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Update zrml/neo-swaps/README.md

Co-authored-by: Chralt <[email protected]>

* Rewrite and format `README.md`

* Fix comment about existential deposit

* Remove FIXME

* Add a check against misconfiguration of the pallet

* Fix field docstrings

* Add comment about `assert_ok_with_transaction`

* Update zrml/neo-swaps/src/mock.rs

Co-authored-by: Chralt <[email protected]>

* Format code

* Fix comment

* Prettify extrinsic calls in benchmarks

* Improve code quality of `assert_pool_state!`

* Extend comment about order of abandoned nodes

* Clarify the meaning of "leaf" in `is_leaf`

* Rename `index_maybe` to `opt_index`

* Add unexpected storage overflow error for bounded objects

* Rename `UnclaimedFees` to `UnwithdrawnFees`

* Fix documentation

* Use enum to signal update direction in `update_descendant_stake`

* Add verification to `join_*` benchmarks

* Improve documentation

* Use builtin log

* Remove illegal token from `README.md`

* Update zrml/neo-swaps/src/types/liquidity_tree.rs

Co-authored-by: Chralt <[email protected]>

* Fix unintentional doctest

* Improve `mutate_children`

* Add helpful comment

* Update zrml/neo-swaps/src/types/liquidity_tree.rs

Co-authored-by: Chralt <[email protected]>

* Fix migration

* Fix balances in parachain mock

* use PartialEqNoBound and CloneNoBound for tree

* Fix formatting

* Make `debug_assert!` more clear

* Redesign node assignment

* Clean up comments

* Add some storage overflow errors

* Introduce `LiquidityTreeChildIndices`

* Remove outdated docs

* Rename `update_descendant_stake`

* Remove `Default` usage

* Make liquidity tree types `pub(crate)` where possible

* Make all fields of `Node` only `pub(crate)`

* Make `Node<T>` an associated type of `LiquidityTreeHelper`

* Update zrml/neo-swaps/src/lib.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Fix comment

* Update zrml/neo-swaps/src/types/liquidity_tree.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Fix tests

* Prettify `path_to_node`

* Add max iterations to `path_to_node`

* Add test for complex liquidity tree interactions

* Improve documentation of `LiquidityTreeChildIndices`

* Reorganize crate structure

* Update weights and fix formatting

---------

Co-authored-by: Chralt <[email protected]>
Co-authored-by: Chralt98 <[email protected]>
Co-authored-by: Harald Heckmann <[email protected]>
* Fix padding of foreign fees in Zeitgeist runtime

* Fix padding of foreign fees in Battery Station runtime

* Format

* Update copyright

* Use adjusted_balance function
* Replace `bmul` and `bdiv` with traited versions

* Restructure directories

* Replace `saturating_*` from neo-swaps

* Fix formatting

* Restructure zrml-swaps math

* Implement and test `b*`

* Fix formatting

* Use new math in orderbook-v1

* Replace checked multiplication with new math

* Use correct rounding in neo-swaps

* Add docs

* Update licenses

* Remove `fixed` module from `primitives`

* Fix formatting

* .

* Rewrite math functions

* Remove training wheels

* Fix docs.pdf

* Fix quotes

* Add tests for buying

* Add tests for selling and improve error names

* Update docs

* Check adjusted amount in for numerical bounds

* Remove unused implementations

* Adjust docs

* Add stress test exploring various scenarios

* Add swap fees to stress test

* Add underscore separators

* Clean up

* Benchmark `buy` as function of `asset_count`

* Update benchmarks

* Clippy fix

* Fix benchmark tests

* Update benchmarks of zrml-prediction-markets

* Fix broken comment

* Add comment explaining benchmark spot prices

* Use clearer constants for `amount_in` in tests

* Update zrml/neo-swaps/src/traits/pool_operations.rs

Co-authored-by: Chralt <[email protected]>

* Fix botched merge

* Fix merge

* Update benchmarks

* Remove `pool_*_subsidy`

* Remove `distribute_pool_share_rewards`

* Remove `end_subsidy_phase`

* Remove `destroy_pool_in_subsidy_phase`

* Remove `MinSubsidy*`

* Remove `SubsidyProviders`

* Remove `start_subsidy`

* Rewrite `create_pool`

* Rewrite `swap_exact_amount_in`

* Rewrite `swap_exact_amount_out`

* Rewrite utility functions

* Remove Rikiddo from weight calculation

* Remove Rikiddo from zrml-swaps

* Remove unused errors

* Remove `ScoringRule::Rikiddo...`

* Remove `*SubsidyPeriod`

* Remove Rikiddo-related storage and events

* Remove automatic opening of pools

* Remove `open_pool` from prediction-markets

* Remove `Swaps::close_pool` from prediction-markets

* Remove `clean_up_pool` from prediction-markets

* Remove `clean_up_pool` from `Swaps` trait

* Remove CPMM deployment from prediction-markets

* Remove automatic arbitrage

* Move `market_account` back to prediction-markets

* Remove unused market states

* Fix fuzz tests

* Implement market migration

* Minor changes

* Fix migration behavior

* Remove creator fees from swaps

* Fix try-runtime

* Fix clippy issues

* Remove `LiquidityMining` from swaps

* Fix `get_spot_prices`

* Take first step to remove `MarketCommons` from swaps

* Remove `MarketCommons` from swaps

* Rewrite `PoolStatus`

* Move `Pool*` to swaps

* Use `Bounded*` types in `Pool`

* Finish swaps migration

* Add missing files

* Fix formatting and clippy errors

* Remove `pool_status.rs`

* Ignore doctests

* Fix fuzz tests

* Add prediciton-markets migration

* Test prediction-markets migration

* Finish tests of the market-commons migration

* Add migrations to runtime and fix various errors

* Clean up

* Clean up

* Format code

* Fix pool migration behavior

* Remove `MarketId` from swaps

* Fix formatting

* Fix formatting

* Remove `CPMM` and allow other scoring rules on Battery Station

* Update macros/Cargo.toml

Co-authored-by: Harald Heckmann <[email protected]>

* Update primitives/src/traits/zeitgeist_asset.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/market-commons/src/migrations.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/swaps/src/migrations.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/swaps/src/migrations.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/prediction-markets/src/migrations.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Update zrml/market-commons/src/migrations.rs

Co-authored-by: Harald Heckmann <[email protected]>

* Clean up TODOs/FIXMEs

* Update changelog

* Make more changes to changelog

* Clear zrml-swaps storage

* Remove cfg-if dependency

* Fix formatting

* Trigger CI

* Update copyright notices

* Update docs/changelog_for_devs.md

Co-authored-by: Chralt <[email protected]>

* Make benchmark helper only available if feature flags are set

* Remove `ZeitgeistAsset` trait

* Remove preliminary benchmarks with more steps

* Format code

* Fix copyright notice

---------

Co-authored-by: Chralt <[email protected]>
Co-authored-by: Harald Heckmann <[email protected]>
* Use hotfixed `exp`

* Reorganize tests

* Fix formatting

* Bump versions to v0.4.3

* Update spec versions

* Add missing version bumps

* Format

* Update licenses

---------

Co-authored-by: Malte Kliemann <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* remove dangerous loop

* limit iterations of dangerous loop

* reintegrate last time frame storage item

* wip

* benchmark manual close and open

* fix stall test

* emit event and log for recovery time frame

* add tests

* add trailing comma

* Update zrml/prediction-markets/src/benchmarks.rs

Co-authored-by: Malte Kliemann <[email protected]>

* change should_be_closed condition

* Apply suggestions from code review

Co-authored-by: Malte Kliemann <[email protected]>

* change recursion limit line

* regard period not started yet

* add error tests

* correct benchmarks

* fix after merge and clippy

* use turbofish, add test checks

* remove manually open broken market

* correct errors and call index

* correct wrong error name

* correct position of call index

* correct error position

* update copyrights

* fix after merge

* Update zrml/prediction-markets/src/benchmarks.rs

Co-authored-by: Malte Kliemann <[email protected]>

* set market end for manual close

---------

Co-authored-by: Malte Kliemann <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Reduce bencharm runs of Zeitgeist pallets

Running benchmarks of Zeitgeist pallets on the Zeitgeist reference machine currently takes four days.
…1235)

* Abstract `pool_exit` business logic into `do_*` function

* Add `force_pool_exit` and test

* Install call filters for zrml-swaps
…mutuel (#1223)

* Implement and test `bmul_bdiv_*`

* Use `bmul_bdiv_*` in pallets

* Update copyright
ci(Mergify): configuration update

Signed-off-by: Harald Heckmann <[email protected]>
* Set in-progress when need and rerun CI in merge queue

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
* Avoid mergify deque

* Set review needed only shortly after Mergify commit
* Add tests to neo-swaps that check large buy/sell amounts at high liquidity

* Use new implementation of HydraDX's `exp`

* Add failure tests to neo-swaps's `math.rs`

* Fix formatting

* Update copyright notices

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@sea212 sea212 added s:blocked The pull requests awaits resolution of dependencies (state what it depends on) s:in-progress The pull requests is currently being worked on and removed s:blocked The pull requests awaits resolution of dependencies (state what it depends on) labels Mar 28, 2024
@sea212 sea212 changed the base branch from main to sea212-new-asset-system April 2, 2024 09:21
@sea212 sea212 requested a review from yornaath as a code owner April 2, 2024 09:21
@sea212 sea212 changed the title New Asset System New Asset System - Prepare merge into main Apr 2, 2024
Copy link
Contributor

mergify bot commented Apr 2, 2024

This pull request is now in conflicts. Could you fix it @sea212? 🙏

@mergify mergify bot added s:revision-needed The pull requests must be revised and removed s:in-progress The pull requests is currently being worked on labels Apr 2, 2024
@sea212 sea212 added s:review-needed The pull request requires reviews and removed s:revision-needed The pull requests must be revised labels Apr 2, 2024
@mergify mergify bot added s:revision-needed The pull requests must be revised and removed s:review-needed The pull request requires reviews labels Apr 2, 2024
@sea212 sea212 added s:in-progress The pull requests is currently being worked on and removed s:revision-needed The pull requests must be revised labels Apr 2, 2024
@mergify mergify bot added s:revision-needed The pull requests must be revised and removed s:in-progress The pull requests is currently being worked on labels Apr 2, 2024
@mergify mergify bot added s:in-progress The pull requests is currently being worked on and removed s:revision-needed The pull requests must be revised labels Apr 2, 2024
@sea212 sea212 added s:review-needed The pull request requires reviews and removed s:in-progress The pull requests is currently being worked on labels Apr 2, 2024
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 85.66732% with 218 lines in your changes are missing coverage. Please review.

❗ No coverage uploaded for pull request base (sea212-new-asset-system@6b1aef6). Click here to learn what that means.

Files Patch % Lines
zrml/swaps/src/lib.rs 76.34% 79 Missing ⚠️
zrml/swaps/src/weights.rs 28.20% 56 Missing ⚠️
zrml/court/src/weights.rs 75.91% 33 Missing ⚠️
zrml/parimutuel/src/weights.rs 0.00% 15 Missing ⚠️
zrml/orderbook/src/weights.rs 0.00% 12 Missing ⚠️
zrml/prediction-markets/src/weights.rs 96.02% 7 Missing ⚠️
zrml/court/src/lib.rs 96.66% 4 Missing ⚠️
zrml/styx/src/weights.rs 0.00% 4 Missing ⚠️
zrml/court/src/migrations.rs 97.81% 3 Missing ⚠️
zrml/neo-swaps/src/weights.rs 96.55% 3 Missing ⚠️
... and 1 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@                    Coverage Diff                     @@
##             sea212-new-asset-system    #1292   +/-   ##
==========================================================
  Coverage                           ?   91.81%           
==========================================================
  Files                              ?      191           
  Lines                              ?    34643           
  Branches                           ?        0           
==========================================================
  Hits                               ?    31806           
  Misses                             ?     2837           
  Partials                           ?        0           
Flag Coverage Δ
tests 91.81% <85.66%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sea212 sea212 merged commit ad86d45 into sea212-new-asset-system Apr 2, 2024
18 of 19 checks passed
@sea212 sea212 deleted the sea212-new-asset-system-part-10 branch April 2, 2024 11:13
@sea212 sea212 added s:accepted This pull request is ready for merge and removed s:review-needed The pull request requires reviews labels Apr 2, 2024
@sea212 sea212 removed this from the v0.5.1 milestone Apr 2, 2024
@sea212 sea212 mentioned this pull request Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
s:accepted This pull request is ready for merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants