Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] API Redesign #1607

Closed
wants to merge 62 commits into from
Closed

[WIP] API Redesign #1607

wants to merge 62 commits into from

Conversation

ouziel-slama
Copy link
Contributor

@ouziel-slama ouziel-slama commented Apr 5, 2024

  • Implement API v2
    • ensure that all data is covered
    • ensure that all API v1 endpoints have an equivalent
    • Implement unpack for all message types
  • Deprecate API v1
    • Use X-API-Warn
    • Throw warnings when they're used
    • Put behind a CLI flag
    • Put behind /old/api/
  • Test API v2
  • Document API v2

Quick description of API v2

New flags

  • --api-host (default: "localhost")
  • --api-port (default: 4000)
  • --api-user (default: "api")
  • --api-password (default: "api")
  • --api-no-allow-cors (default: False)
  • --api-not-ready-http-code (default: 503)
  • --enable-v1-api (default: False)

New default port for the API v1: 4100

New Headers

All responses contain the following three headers:

X-COUNTERPARTY-HEIGHT: Last parsed block
X-BACKEND-HEIGHT: backend.getblockcount() + backend.getindexblocksbehind()
X-COUNTERPARTY-READY: X-COUNTERPARTY-HEIGHT >= X-BACKEND-HEIGHT

New Routes

Get Ledger State

  • /blocks

    • /blocks
    • /blocks/<int:block_index>
    • /blocks/<int:block_index>/transactions
    • /blocks/<int:block_index>/events
    • /blocks/<int:block_index>/events/counts
    • /blocks/<int:block_index>/events/<event>
    • /blocks/<int:block_index>/credits
    • /blocks/<int:block_index>/debits
    • /blocks/<int:block_index>/expirations
    • /blocks/<int:block_index>/cancels
    • /blocks/<int:block_index>/destructions
    • /blocks/<int:block_index>/issuances
    • /blocks/<int:block_index>/sends
    • /blocks/<int:block_index>/dispenses
    • /blocks/<int:block_index>/sweeps
  • /transactions

    • /transactions/<tx_hash>
  • /addresses

    • /addresses/<address>/balances
    • /addresses/<address>/balances/<asset>
    • /addresses/<address>/credits
    • /addresses/<address>/debits
    • /addresses/<address>/bets
    • /addresses/<address>/broadcasts
    • /addresses/<address>/burns
    • /addresses/<address>/sends
    • /addresses/<address>/receives
    • /addresses/<address>/sends/<asset>
    • /addresses/<address>/receives/<asset>
    • /addresses/<address>/dispensers
    • /addresses/<address>/dispensers/<asset>
    • /addresses/<address>/sweeps
  • /assets

    • /assets
    • /assets/<asset>
    • /assets/<asset>/balances
    • /assets/<asset>/balances/<address>
    • /assets/<asset>/orders
    • /assets/<asset>/credits
    • /assets/<asset>/debits
    • /assets/<asset>/dividends
    • /assets/<asset>/issuances
    • /assets/<asset>/sends
    • /assets/<asset>/dispensers
    • /assets/<asset>/dispensers/<address>
    • /assets/<asset>/holders
  • /orders

    • /orders/<tx_hash>
    • /orders/<tx_hash>/matches
    • /orders/<tx_hash>/btcpays
  • /bets

    • /bets/<tx_hash>
    • /bets/<tx_hash>/matches
    • /bets/<tx_hash>/resolutions
  • /burns

    • /burns
  • /dispensers

    • /dispensers/<tx_hash>:
    • /dispensers/<tx_hash>/dispenses
  • /events

    • /events
    • /events/<int:event_index>
    • /events/counts
    • /events/<event>
  • /mempool

    • /mempool/events
    • /mempool/events/<event>

Compose and Parse Transactions

  • /transactions/compose/<transaction_name>
  • /transactions/info
  • /transactions/unpack

Get Server State

  • /
  • /healthz

Backend Proxy

  • /backend/addresses/<address>/transactions
  • /backend/addresses/<address>/transactions/oldest
  • /backend/addresses/<address>/utxos
  • /backend/addresses/<address>/pubkey
  • /backend/transactions/<tx_hash>
  • /backend/transactions
  • /backend/estimatesmartfee

Equivalence table between API v1 and v2

Get Ledger State

API v1 API v2
get_assets /assets
/assets/<asset>
/events/ASSET_CREATION
/blocks/<int:block_index>/events/ASSET_CREATION
get_balances /assets/<asset>/balances
/addresses/<address>/balances
/addresses/<address>/balances/<asset>
/assets/<asset>/balances/<address>
get_credits /blocks/<int:block_index>/credits
/addresses/<address>/credits
/assets/<asset>/credits
/events/CREDIT
/blocks/<int:block_index>/events/CREDIT
get_debits /blocks/<int:block_index>/debits
/addresses/<address>/debits
/assets/<asset>/debits
/events/DEBIT
/blocks/<int:block_index>/events/DEBIT
get_bets /addresses/<feed_address>/bets
/bets/<tx_hash>
/events/OPEN_BET
/blocks/<int:block_index>/events/OPEN_BET
get_bet_matches /bets/<tx_hash>/matches
/events/BET_MATCH
/blocks/<int:block_index>/events/BET_MATCH
get_broadcasts /addresses/<feed_address>/broadcasts
/events/BROADCAST
/blocks/<int:block_index>/events/BROADCAST
get_btcpays /orders/<tx_hash>/btcpays
/events/BTC_PAY
/blocks/<int:block_index>/events/BTC_PAY
get_burns /addresses/<address>/burns
/burns
/events/BURN
/blocks/<int:block_index>/events/BURN
get_cancels /blocks/<int:block_index>/cancels
/events/CANCEL_BET
/blocks/<int:block_index>/events/CANCEL_BET
/events/CANCEL_ORDER
/blocks/<int:block_index>/events/CANCEL_ORDER
/events/CANCEL_RPS
/blocks/<int:block_index>/events/CANCEL_RPS
get_destructions /blocks/<int:block_index>/destructions
/events/ASSET_DESTRUCTION
/blocks/<int:block_index>/events/ASSET_DESTRUCTION
get_dividends /assets/<asset>/dividends
/events/ASSET_DIVIDEND
/blocks/<int:block_index>/events/ASSET_DIVIDEND
get_issuances /blocks/<int:block_index>/issuances
/assets/<asset>/issuances
/events/ASSET_ISSUANCE
/blocks/<int:block_index>/events/ASSET_ISSUANCE
get_orders /assets/<asset>/orders
/orders/<tx_hash>
/events/OPEN_ORDER
/blocks/<int:block_index>/events/OPEN_ORDER
get_order_matches /orders/<tx_hash>/matches
/events/ORDER_MATCH
/blocks/<int:block_index>/events/ORDER_MATCH
get_sends /blocks/<int:block_index>/sends
/assets/<asset>/sends
/addresses/<address>/sends
/addresses/<address>/receives
/addresses/<address>/sends/<asset>
/addresses/<address>/receives/<asset>
/events/SEND
/blocks/<int:block_index>/events/SEND
/events/MPMA_SEND
/blocks/<int:block_index>/events/MPMA_SEND
/events/ENHANCED_SEND
/blocks/<int:block_index>/events/ENHANCED_SEND
get_bet_expirations /blocks/<int:block_index>/expirations
/events/BET_EXPIRATION
/blocks/<int:block_index>/events/BET_EXPIRATION
get_order_expirations /blocks/<int:block_index>/expirations
/events/ORDER_EXPIRATION
/blocks/<int:block_index>/events/ORDER_EXPIRATION
get_bet_match_expirations /blocks/<int:block_index>/expirations
/events/BET_MATCH_EXPIRATION
/blocks/<int:block_index>/events/BET_MATCH_EXPIRATION
get_order_match_expirations /blocks/<int:block_index>/expirations
/events/ORDER_MATCH_EXPIRATION
/blocks/<int:block_index>/events/ORDER_MATCH_EXPIRATION
get_bet_match_resolutions /bets/<tx_hash>/resolutions
/events/BET_MATCH_RESOLUTON
/blocks/<int:block_index>/events/BET_MATCH_RESOLUTON
get_rps /events/OPEN_RPS
/blocks/<int:block_index>/events/OPEN_RPS
get_rpsresolves /events/RPS_RESOLVE
/blocks/<int:block_index>/events/RPS_RESOLVE
get_rps_matches /events/RPS_MATCH
/blocks/<int:block_index>/events/RPS_MATCH
get_rps_expirations /blocks/<int:block_index>/expirations
/events/RPS_EXPIRATION
/blocks/<int:block_index>/events/RPS_EXPIRATION
get_rps_match_expirations /blocks/<int:block_index>/expirations
/events/RPS_MATCH_EXPIRATION
/blocks/<int:block_index>/events/RPS_MATCH_EXPIRATION
get_sweeps /addresses/<address>/sweeps
/blocks/<int:block_index>/sweeps
/events/SWEEP
/blocks/<int:block_index>/events/SWEEP
get_dispensers /assets/<asset>/dispensers
/asset/<asset>/dispensers/<address>
/addresses/<address>/dispensers
/addresses/<address>/dispensers/<asset>
/dispensers/<tx_hash>
/events/OPEN_DISPENSER
/blocks/<int:block_index>/events/OPEN_DISPENSER
get_dispenses /blocks/<int:block_index>/dispenses
/dispensers/<tx_hash>/dispenses
/events/DISPENSE
/blocks/<int:block_index>/events/DISPENSE
get_transactions /blocks/<int:block_index>/transactions
/transactions/<tx_hash>
/events/NEW_TRANSACTION
/blocks/<int:block_index>/events/NEW_TRANSACTION
get_messages /blocks/<int:block_index>/events
/blocks/<int:block_index>/events/<event>
/events/<event>
get_messages_by_index /events/<int:event_index>
get_supply /assets/<asset>
get_xcp_supply /assets/XCP
get_asset_info /assets/<asset>
get_block_info /blocks/<int:block_index>
/blocks/<int:block_index>/events
get_blocks /blocks
get_asset_names /assets
get_asset_longnames /assets
get_holder_count /assets/<asset>
get_holders /assets/<asset>/holders
get_dispenser_info /dispensers/<tx_hash>
get_mempool /mempool/events
/mempool/events/<event>
get_element_counts /blocks/<int:block_index>/events/counts
/events/counts
sql NA

Compose and Parse Transactions

API v1 API v2
create_bet /transactions/compose/bet
create_broadcast /transactions/compose/broadcast
create_btcpay /transactions/compose/btcpay
create_burn /transactions/compose/burn
create_cancel /transactions/compose/cancel
create_destroy /transactions/compose/destroy
create_dividend /transactions/compose/dividend
create_issuance /transactions/compose/issuance
create_order /transactions/compose/order
create_send /transactions/compose/send
create_rps /transactions/compose/rps
create_rpsresolve /transactions/compose/rpsresolve
create_sweep /transactions/compose/sweep
create_dispenser /transactions/compose/dispenser
get_tx_info /transactions/info
unpack /transactions/unpack

Get Server State

API v1 API v2
get_running_info /
/healthz /healthz

Backend Proxy

API v1 API v2
search_raw_transactions /backend/addresses/<address>/transactions
get_oldest_tx /backend/addresses/<address>/transactions/oldest
get_unspent_txouts /backend/addresses/<address>/utxos
getrawtransaction /backend/transactions/<tx_hash>
getrawtransaction_batch /backend/transactions
search_pubkey /backend/addresses/<address>/pubkey
fee_per_kb /backend/estimatesmartfee

@ouziel-slama ouziel-slama marked this pull request as draft April 5, 2024 11:01
Copy link

@github-advanced-security github-advanced-security bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pylint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@ouziel-slama
Copy link
Contributor Author

OP updated.

@mattmarcello
Copy link
Contributor

@ouziel-slama, this looks great. Question for you: are these resources paginated?

counterparty-core/counterpartycore/server.py Outdated Show resolved Hide resolved
counterparty-lib/counterpartylib/server.py Outdated Show resolved Hide resolved
counterparty-lib/counterpartylib/lib/api/routes.py Outdated Show resolved Hide resolved
@mattmarcello
Copy link
Contributor

I wonder if mempool should be behind it's own resource, instead of blocks since it only indexes events and events/<event> ( i.e. is not totally symmetrical with other block resources.

/mempool/events
/mempool/events/<event>

@mattmarcello
Copy link
Contributor

Just a few minor thoughts for the /transactions/compose/ API. You are going to need to post data anyway here, so I don't think it's necessary to have /compose. Could just be a POST to /transactions. To that end, would it be more clear to call this endpoint something like /rawtransactions to make it explicit that this is not tantamount to a create and broadcast? Also, maybe it would be nicer from the callers standpoint to pass the transaction type as data in the POST request? Not 100% sure.

curl -d '{"type":"bet", "args": ...betargs }' -H "Content-Type: application/json" -X POST http://localhost:4000/rawtransactions

Compose and Parse Transactions

API v1 API v2 --
create_bet /transactions/compose/bet curl -d '{"type":"bet", "args": ...betargs }' -H "Content-Type: application/json" -X POST http://localhost:4000/rawtransactions
create_broadcast /transactions/compose/broadcast curl -d '{"type":"broadcast", "args": ...broadcastargs }' -H "Content-Type: application/json" -X POST http://localhost:4000/rawtransactions
...

@ouziel-slama
Copy link
Contributor Author

@ouziel-slama, this looks great. Question for you: are these resources paginated?

A basic pagination only for some queries (with the parameters last and limit)

@ouziel-slama
Copy link
Contributor Author

I wonder if mempool should be behind it's own resource, instead of blocks since it only indexes events and events/<event> ( i.e. is not totally symmetrical with other block resources.

/mempool/events
/mempool/events/<event>

I hesitated.. we can say that the mempool contains the next block, it's a mental view, but in the code it is managed like that with a block_index=999999999.. as you wish I can change easily ..

@ouziel-slama
Copy link
Contributor Author

Just a few minor thoughts for the /transactions/compose/ API. You are going to need to post data anyway here, so I don't think it's necessary to have /compose. Could just be a POST to /transactions. To that end, would it be more clear to call this endpoint something like /rawtransactions to make it explicit that this is not tantamount to a create and broadcast? Also, maybe it would be nicer from the callers standpoint to pass the transaction type as data in the POST request? Not 100% sure.

hum.. I am sure that all our requests must be GET and there must be no POST. We open the database in read only mode and no queries modify the database. Also in most cases the url is formed with a script/CLI or something else, I don't think this is a problem.

@adamkrellenstein
Copy link
Member

I wonder if mempool should be behind it's own resource, instead of blocks since it only indexes events and events/<event> ( i.e. is not totally symmetrical with other block resources.

/mempool/events
/mempool/events/<event>

I hesitated.. we can say that the mempool contains the next block, it's a mental view, but in the code it is managed like that with a block_index=999999999.. as you wish I can change easily ..

Yeah I think we should have a dedicated route and later fix that magic number block index, which bothers me :/ (it's my fault originally!)

@mattmarcello
Copy link
Contributor

I wonder if mempool should be behind it's own resource, instead of blocks since it only indexes events and events/<event> ( i.e. is not totally symmetrical with other block resources.

/mempool/events
/mempool/events/<event>

I hesitated.. we can say that the mempool contains the next block, it's a mental view, but in the code it is managed like that with a block_index=999999999.. as you wish I can change easily ..

Interesting. I presume these requests would be errors?

/blocks/999999999/sends
/blocks/mempool/sends

@adamkrellenstein
Copy link
Member

adamkrellenstein commented Apr 11, 2024

Let's include this only into v10.1.2

if api_password:
config.API_PASSWORD = api_password
else:
config.API_PASSWORD = "api" # noqa: S105

Check notice

Code scanning / Bandit

Possible hardcoded password: 'api' Note

Possible hardcoded password: 'api'
if isinstance(result, flask.Response):
response = result
else:
response = flask.make_response(flask.jsonify(result), http_code)

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
flows to this location and may be exposed to an external user.
@ouziel-slama
Copy link
Contributor Author

Note about API design

The idea behind the new API is “one route, one function”. Example:

in api/routes.py:

...
ROUTES = util.prepare_routes(
    {
        ### /blocks ###
        "/blocks": ledger.get_blocks,
        "/blocks/<int:block_index>": ledger.get_block,
        "/blocks/<int:block_index>/transactions": ledger.get_transactions_by_block,
        "/blocks/<int:block_index>/events": ledger.get_events_by_block,
        ...
    }
)
...

in ledger.py:

def get_blocks(db, last: int = None, limit: int = 10):
    """
    Returns the list of the last ten blocks
    :param int last: The index of the most recent block to return
    :param int limit: The number of blocks to return
    """
    ...

The new API uses type annotations and docstrings to generate route arguments, default values, and documentation.

curl http://api:api@localhost:4000/ returns:

{
  "backend_height": 839940,
  "counterparty_height": 839940,
  "network": "mainnet",
  "server_ready": true,
  "version": "10.1.0"
  "routes": [
    ...
    {
      "path": "/blocks"
      "description": "Returns the list of the last ten blocks",
      "args": [
        {
          "default": null,
          "description": "The index of the most recent block to return",
          "name": "last",
          "required": false,
          "type": "int"
        },
        {
          "default": 10,
          "description": "The number of blocks to return",
          "name": "limit",
          "required": false,
          "type": "int"
        }
      ],
      
    },
    ...
  ]
}

this output will be used to automatically generate a blueprint/swagger.

@ouziel-slama
Copy link
Contributor Author

moved here #1716

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants