[2/5] light client http api #12984

nicopernas · 2023-10-03T00:11:14Z

What type of PR is this?

Feature

What does this PR do? Why is it needed?

Implements the Altair spec logic defined in this version:

https://github.com/ethereum/consensus-specs/tree/208da34ac4e75337baf79adebf036ab595e39f15/specs/altair/light-client

It also implements the relevant endpoints specified at https://github.com/ethereum/beacon-APIs

#11815

Which issues(s) does this PR fix?

Partially addresses #11571

Other notes for review

This PR is a follow up to #11815 which was too big to be reviewed. This is the second PR in a series of 5 that I will be submitting sequentially.

The heavy lifting here was done by @qinlz2 and @7AC, so kudos to them.

Full series:

[1/5] [1/5] Light client sync protocol #12853
[2/5] [2/5] light client http api #12984 (this PR)
[3/5]
[4/5]
[5/5] [5/5] Light client sync protocol #12854

#12854 is the last PR in the series that contains all changes so you can go back and see the full picture while reviewing this one.

rkapka · 2023-10-03T10:09:24Z

Hi @nicopernas , can you please open one more PR and move lightclient.go from the blockchain package to core/light-client? The argument is:

they are all stateless methods with no requirement to use the blockchain service

nicopernas · 2023-10-03T14:20:10Z

Hi @nicopernas , can you please open one more PR and move lightclient.go from the blockchain package to core/light-client? The argument is:

they are all stateless methods with no requirement to use the blockchain service

That's no problem at all. If it's not an urgent change, can we do it after the rest of the PRs are merged?? It would save me from rebasing once or twice :)

rkapka · 2023-10-03T16:04:14Z

OK, but in that case I will open an issue for it, just to make sure this is not forgotten.

nicopernas · 2023-10-03T16:24:28Z

OK, but in that case I will open an issue for it, just to make sure this is not forgotten.

Thanks for that!

rkapka · 2023-10-05T12:58:25Z

beacon-chain/rpc/eth/light-client/handlers.go

+)
+
+// GetLightClientBootstrap - implements https://github.com/ethereum/beacon-APIs/blob/263f4ed6c263c967f13279c7a9f5629b51c5fc55/apis/beacon/light_client/bootstrap.yaml
+func (bs *Server) GetLightClientBootstrap(w http.ResponseWriter, req *http.Request) {


All HTTP receivers are named s

rkapka · 2023-10-05T13:01:33Z

beacon-chain/rpc/eth/light-client/handlers.go

+	// Get the block
+	blockRootParam, err := hexutil.Decode(mux.Vars(req)["block_root"])
+	if err != nil {
+		http2.HandleError(w, "invalid block root "+err.Error(), http.StatusBadRequest)


Suggested change

http2.HandleError(w, "invalid block root "+err.Error(), http.StatusBadRequest)

http2.HandleError(w, "invalid block root: "+err.Error(), http.StatusBadRequest)

similarly in other places

rkapka · 2023-10-05T13:02:47Z

beacon-chain/rpc/eth/light-client/handlers.go

+	var blockRoot [32]byte
+	copy(blockRoot[:], blockRootParam)


we have a bytesutil.ToBytes32 function that you can use

rkapka · 2023-10-05T13:08:27Z

beacon-chain/rpc/eth/light-client/handlers.go

+	}
+
+	response := &LightClientBootstrapResponse{
+		Version: ethpbv2.Version(blk.Version()).String(),


it would be better to use the String function defined in runtime/version/fork.go. We want to avoid depending on ethpbv1/ethpbv2 because we plan to remove these packages in the near future. Another advantage is that it returns a lower-case version name, which is aligned with the API spec.

rkapka · 2023-10-05T13:48:32Z

beacon-chain/rpc/eth/light-client/helpers.go

+func CreateLightClientBootstrap(ctx context.Context, state state.BeaconState) (*LightClientBootstrap, error) {
+	// assert compute_epoch_at_slot(state.slot) >= ALTAIR_FORK_EPOCH
+	if slots.ToEpoch(state.Slot()) < params.BeaconConfig().AltairForkEpoch {
+		return nil, fmt.Errorf("invalid state slot %d", state.Slot())


it would be better to be more informative here and say something like "light client bootstrap is not supported before Altair"

rkapka · 2023-10-05T14:45:36Z

beacon-chain/rpc/eth/light-client/helpers.go

+)
+
+const (
+	nextSyncCommitteeBranchNumOfLeaves = 5


This is probably only correct for minimal config, where the sync committee size is 32. For mainnet config the sync committee size is 512 and therefore the proof will be larger. In order to make this work for both, you should define a minimal and mainnet version of this const in the fieldparams package (the name of the package is a little outdated).

This const may be wrongly named. It basically states the number of branches needed for a proof.

This is only relevant the position of the field in the trie leaves. (Where currentSyncCommittee and nextSyncCommittee are next to each other).

So this isn't related to sync committee size (no matter it is 32 or 512), but related to the total number of leaves in the trie.

Yes making this configurable is future proof though. So let's make it configurable.

rkapka · 2023-10-05T14:52:16Z

beacon-chain/rpc/eth/light-client/server.go

+)
+
+type Server struct {
+	BeaconDB    db.ReadOnlyDatabase


There is a lookup.Blocker interface that's probably better suited to use for handlers. You can pass the block root into the Block function and get the same result as you currently do, but unit testing should be much easier as you won't need to save anything to the db.

Good to know. Thanks.

Patched.

rkapka · 2023-10-05T15:01:42Z

beacon-chain/rpc/eth/light-client/handlers_test.go

+	resp := &LightClientBootstrapResponse{}
+	require.NoError(t, json.Unmarshal(writer.Body.Bytes(), resp))
+	require.Equal(t, "CAPELLA", resp.Version)
+	require.Equal(t, hexutil.Encode(header.Header.BodyRoot), resp.Data.Header.BodyRoot)


Given that we have the helper function that prepares the whole bootstrap object, I think it's fine to check only the version here and maybe verify that resp.Data is not nil. But I would still check values of several bootstrap fields by writing a unit test of the helper function.

At this moment, let's just check resp.Data not nil, and existing very basic field checking.
Feel free to create a following task, so we can add more solid unit tests.

rkapka · 2023-10-05T15:03:10Z

beacon-chain/rpc/eth/light-client/helpers.go

+
+	// Return result
+	result := &LightClientBootstrap{
+		Header: &apimiddleware.BeaconBlockHeaderJson{


Similarly comment as for apimiddleware.SyncCommitteeJson, but this time please put the new type inside /eth/shared/structs_blocks.go

rkapka · 2023-10-05T15:05:44Z

beacon-chain/rpc/eth/light-client/structs.go

+	Header                     *apimiddleware.BeaconBlockHeaderJson `json:"header"`
+	CurrentSyncCommittee       *apimiddleware.SyncCommitteeJson     `json:"current_sync_committee"`


these two fields should be of the newly created types that I mentioned in other comments, not the ones in apimiddleware

rkapka · 2023-10-05T15:07:59Z

I reviewed the bootstrap function and will likely continue tomorrow.

rkapka · 2023-10-05T15:27:36Z

I found something missing in your previous PR. In your proto definitions you use current_sync_committee_branch.depth, next_sync_committee_branch.depth and finality_branch.depth, but in order for these variables to work with SSZ, you need to define their values in proto/ssz_proto_library.bzl. Will this be done in one of the next PRs? Because if not and this is an omission, then I don't understand how SSZ generation for these proto messages could ever work.

Looking at these variables, will finality_branch.depth differ between minimal and mainnet? If not, we can simply hardcode it as a number.

Speaking about SSZ generation, API spec requires SSZ support:

Depending on the Accept header it can be returned either as JSON or SSZ-serialized bytes.

This would mean that SSZ generation for LightClientBootstrap and other types is indeed necessary. Which brings a question: do you have SSZ support in APIs coming in a later PR? If not, we can create an issue for it.

rkapka · 2023-10-06T17:36:31Z

beacon-chain/rpc/eth/light-client/helpers.go

+	var nextSyncCommitteeBranch [][]byte
+
+	// update_signature_period = compute_sync_committee_period(compute_epoch_at_slot(block.message.slot))
+	updateSignaturePeriod := uint64(block.Block().Slot()) / slotsPerPeriod


you can use the slots.ToEpoch helper

rkapka · 2023-10-06T17:36:49Z

beacon-chain/rpc/eth/light-client/helpers.go

+	updateSignaturePeriod := uint64(block.Block().Slot()) / slotsPerPeriod
+
+	// update_attested_period = compute_sync_committee_period(compute_epoch_at_slot(attested_header.slot))
+	updateAttestedPeriod := uint64(result.AttestedHeader.Slot) / slotsPerPeriod


beacon-chain/rpc/eth/light-client/helpers.go

rkapka · 2023-10-06T17:45:55Z

beacon-chain/rpc/eth/light-client/handlers.go

+	countParam := req.URL.Query().Get("count")
+	count, err := strconv.ParseUint(countParam, 10, 64)
+	if err != nil {
+		http2.HandleError(w, fmt.Sprintf("got invalid 'count' query variable '%s', err %v", countParam, err),
+			http.StatusInternalServerError)
+		return
+	}


there is a shared.UintFromQuery helper that you can use

rkapka · 2023-10-06T17:46:10Z

beacon-chain/rpc/eth/light-client/handlers.go

+	startPeriodParam := req.URL.Query().Get("start_period")
+	startPeriod, err := strconv.ParseUint(startPeriodParam, 10, 64)
+	if err != nil {
+		http2.HandleError(w, fmt.Sprintf("got invalid 'start_period' query variable '%s', err %v", startPeriodParam, err),
+			http.StatusInternalServerError)
+		return
+	}


rkapka · 2023-10-06T17:53:33Z

beacon-chain/rpc/eth/light-client/handlers.go

+		return
+	}
+
+	lHeadSlot := uint64(headState.Slot())


Can this be renamed to headSlot? We don't use such prefixes anywhere in the codebase. Similarly for other variables in this function

Sure, patched.

nicopernas · 2023-10-09T09:05:21Z

Hey @rkapka thanks a lot for all the good feedback! We are making internal changes to our team structure and I will be looking into something else from now own. Someone else from the team will take over to make sure this and future PRs make it to develop.

rkapka · 2023-10-09T13:00:17Z

beacon-chain/rpc/eth/light-client/handlers.go

+			}
+
+			// Get the block
+			latestBlockHeader := *state.LatestBlockHeader()


No need for the pointer

rkapka · 2023-10-09T13:06:36Z

beacon-chain/rpc/eth/light-client/handlers.go

+			attestedState,
+			finalizedBlock,
+		)
+


Suggested change

rkapka · 2023-10-09T13:10:55Z

beacon-chain/rpc/eth/light-client/helpers.go

+//	    sync_aggregate=block.message.body.sync_aggregate,
+//	    signature_slot=block.message.slot,
+//	)
+func CreateLightClientUpdate(


Have you considered this part of the spec? I don't see this implemented.

Full nodes SHOULD provide the best derivable LightClientUpdate (according to is_better_update) for each sync committee period covering any epochs in range [max(ALTAIR_FORK_EPOCH, current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS), current_epoch] where current_epoch is defined by the current wall-clock time. Full nodes MAY also provide LightClientUpdate for other sync committee periods.

LightClientUpdate are assigned to sync committee periods based on their attested_header.beacon.slot

LightClientUpdate are only considered if compute_sync_committee_period_at_slot(update.attested_header.beacon.slot) == compute_sync_committee_period_at_slot(update.signature_slot)

Only LightClientUpdate with next_sync_committee as selected by fork choice are provided, regardless of ranking by is_better_update. To uniquely identify a non-finalized sync committee fork, all of period, current_sync_committee and next_sync_committee need to be incorporated, as sync committees may reappear over time.

You are correct. There is no implementation for above mentioned text section.
We only implemented the python pseudo spec.

Can we create a separate task to patch this? As this function is individual function, not used by others.

rkapka · 2023-10-09T13:15:42Z

beacon-chain/rpc/eth/light-client/helpers.go

These helpers don't need to be exported

Yes, you are correct.

Patched, only NewLightClientBootstrapFromJSON is exported, and this is used in light client side.

rkapka · 2023-10-09T13:55:14Z

beacon-chain/rpc/eth/light-client/handlers_test.go

+	require.Equal(t, hexutil.Encode(header.Header.BodyRoot), resp.Data.Header.BodyRoot)
+}
+
+func TestLightClientHandler_GetLightClientUpdatesByRange(t *testing.T) {


This endpoint has quite a lot of logic in it - can we have additional test cases?

count > config.MaxRequestLightClientUpdates

startPeriodEndSlot < uint64(config.AltairForkEpoch)*uint64(config.SlotsPerEpoch)

headPeriod < endPeriod

lLastSlotInPeriod > lHeadSlot

len(updates) == 0

rkapka · 2023-10-09T14:39:11Z

beacon-chain/rpc/eth/light-client/handlers.go

+	if block == nil {
+		return nil, fmt.Errorf("latest block is nil")
+	}
+	// Loop through the blocks until we find a block that has super majority of sync committee signatures (2/3)


Are you sure this comment is correct? Isn't supermajority required only when calling from GetLightClientFinalityUpdate?

Good catch, the comment isn't correct.

Patched.

rkapka · 2023-10-09T14:48:21Z

beacon-chain/rpc/eth/light-client/handlers.go

+}
+
+// getLightClientEventBlock - returns the block that should be used for light client events, which satisfies the minimum number of signatures from sync committee
+func (bs *Server) getLightClientEventBlock(ctx context.Context, minSignaturesRequired uint64) (interfaces.ReadOnlySignedBeaconBlock, error) {


Please add some unit tests, two at a minimum:

happy case

number of signatures too low, need to fetch parent block

rkapka · 2023-10-09T14:50:46Z

beacon-chain/rpc/eth/light-client/handlers.go

+}
+
+// getLightClientEventBlock - returns the block that should be used for light client events, which satisfies the minimum number of signatures from sync committee
+func (bs *Server) getLightClientEventBlock(ctx context.Context, minSignaturesRequired uint64) (interfaces.ReadOnlySignedBeaconBlock, error) {


Have you considered caching? Fetching block from the database in a loop is expensive. If your current implementation does not make use of caches, then we should consider adding it in the future.

We didn't consider the cache. Please add a task, so we can improve this later.

Depending on the configuration, typically, satisfying the loop condition isn't hard. (But yes, in extreme error conditions, loop will be triggered severely, and causing performance issue).

BTW, since we switch to use lookup.Blocker, does this component provide cache automatically? If so, then we don't need to add cache.

No, it doesn't. I will add a task.

qinlz2 · 2023-10-11T02:17:16Z

@rkapka This is Lizhang Qin from polymer labs. I will take over this PR work from @nicopernas .

I will address your comments and submit revisions. This will take some time. Thanks!

rkapka · 2023-11-06T21:23:49Z

Hi @qinlz2 , are you able to say how when you might have some time to pick this up? We would like to give light client support higher priority and implement this in the next few months (by "this" I mean not this one PR, but the whole thing).

qinlz2 · 2023-11-07T01:27:39Z

Yes, commit hours for this now. Was deviate from this PR (and the rest of PRs) a little bit.
Coming back now, we will try to meet your timeline. Thanks!

Hi @qinlz2 , are you able to say how when you might have some time to pick this up? We would like to give light client support higher priority and implement this in the next few months (by "this" I mean not this one PR, but the whole thing).

CLAassistant · 2023-11-07T04:55:15Z

All committers have signed the CLA.

qinlz2 · 2023-11-11T03:48:19Z

current_sync_committee_branch.depth

@rkapka In our initial PRs, we only implements JSON support.
For this PRs series we are trying to merge, let's just do JSON part.

Please create task for later SSZ support. Thanks!

qinlz2 · 2023-11-11T03:51:23Z

I found something missing in your previous PR. In your proto definitions you use current_sync_committee_branch.depth, next_sync_committee_branch.depth and finality_branch.depth, but in order for these variables to work with SSZ, you need to define their values in proto/ssz_proto_library.bzl. Will this be done in one of the next PRs? Because if not and this is an omission, then I don't understand how SSZ generation for these proto messages could ever work.

Looking at these variables, will finality_branch.depth differ between minimal and mainnet? If not, we can simply hardcode it as a number.

Speaking about SSZ generation, API spec requires SSZ support:

Depending on the Accept header it can be returned either as JSON or SSZ-serialized bytes.

This would mean that SSZ generation for LightClientBootstrap and other types is indeed necessary. Which brings a question: do you have SSZ support in APIs coming in a later PR? If not, we can create an issue for it.

Between mainnet and min, there is no difference. We can hardcode (and we actually didn't use these at all in the code I think). So I will remove them for now from proto definitions.

qinlz2 · 2023-11-13T06:36:23Z

@rkapka I pushed new revision based on your valuable comments.
Before I rebase to the latest baseline, please help to take a look first, and we will address your comments/advice.
Thanks!

rkapka · 2023-11-16T16:16:38Z

Hey @qinlz2 , everything looks great! My only concern is #12984 (comment) - test names don't align with the test scenarios that I listed. Can you confirm that all test cases are covered? Maybe test names are incorrect?

rkapka · 2023-11-16T16:25:21Z

I updated #12991 to serve as a general tracking issue for future light client improvements.

rkapka · 2023-11-16T16:30:16Z

beacon-chain/rpc/eth/shared/structs.go

+	if sc == nil {
+		return nil
+	}


Can you remove this? The reason is twofold:

we can safely assume structs coming from core code are not nil

even if they are nil, it's better to panic as soon as possible, rather than returning nil,and then panicking somewhere else in code; it will make debugging easier

If you look at other FromConsensus functions, there is no clear pattern. We sometimes check for nil and return errors, sometimes not. We intend to unify this.

Sure, patched.

qinlz2 · 2023-11-17T04:07:09Z

Hey @qinlz2 , everything looks great! My only concern is #12984 (comment) - test names don't align with the test scenarios that I listed. Can you confirm that all test cases are covered? Maybe test names are incorrect?

Yes, all of them are covered and tested. (Due to slight code sequence re-order, the naming for test case won't be exactly the same as your original comment, but all the important if-else branches are tested)

qinlz2 · 2023-11-17T04:18:06Z

@rkapka I update slightly based on your recent comment.
Do you need me to rebase first?

qinlz2 · 2023-11-18T15:53:04Z

Hey @qinlz2 , everything looks great! My only concern is #12984 (comment) - test names don't align with the test scenarios that I listed. Can you confirm that all test cases are covered? Maybe test names are incorrect?

Yes, all of them are covered and tested. (Due to slight code sequence re-order, the naming for test case won't be exactly the same as your original comment, but all the important if-else branches are tested)

@qinlz2 Perfect! In that case let's fix conflicts and merge this beauty :-)

Thanks, @rkapka
I rebased to latest develop branch, manually tested it.

It should be good to go now.

rkapka · 2023-11-20T14:51:24Z

Hey @qinlz2 , please run bazel run //:gazelle -- fix

rkapka · 2023-11-20T18:21:53Z

Also TestGetSpec fails:

assertions.go:37: handlers_test.go:174 Values are not equal, want: 112 (int), got: 113 (int)
handlers_test.go:419: Incorrect key: MAX_REQUEST_LIGHT_CLIENT_UPDATES

qinlz2 · 2023-11-21T05:23:14Z

Also TestGetSpec fails:

assertions.go:37: handlers_test.go:174 Values are not equal, want: 112 (int), got: 113 (int)
handlers_test.go:419: Incorrect key: MAX_REQUEST_LIGHT_CLIENT_UPDATES

@rkapka Thanks, patched both.

rkapka · 2023-11-21T12:27:12Z

@qinlz2 Merged! Thanks a lot for this wonderful work! Looking forward to the next PR 😅

nicopernas requested a review from a team as a code owner October 3, 2023 00:11

nicopernas requested review from kasey, rauljordan and terencechain October 3, 2023 00:11

nicopernas force-pushed the polymer/lightclient-2 branch from 1dc44a0 to e154a1e Compare October 3, 2023 00:14

nicopernas force-pushed the polymer/lightclient-2 branch from 45e25ff to 9efb69e Compare October 3, 2023 15:33

rkapka requested changes Oct 5, 2023

View reviewed changes

rkapka reviewed Oct 6, 2023

View reviewed changes

rkapka requested changes Oct 9, 2023

View reviewed changes

rkapka reviewed Oct 9, 2023

View reviewed changes

rkapka mentioned this pull request Nov 16, 2023

Light Client Support #12991

Open

27 tasks

rkapka reviewed Nov 16, 2023

View reviewed changes

qinlz2 added 20 commits November 18, 2023 23:42

test: unit tests check not-nil

b536a66

fix: avoid NPE

b4c7cfb

fix: patch bazel build

8848e97

refactor: use ToEpoch

2360a58

refactor: use shared.UintFromQuery

af8bbc7

refactor: another shared.UintFromQuery

0396074

refactor: lXXXX variable rename

be9fb29

refactor: use pointer directly

d323cb1

refactor: do not export unneccessary functions

8916e7e

fix: correct comments

14bf4fa

refactor: rename handler Server receiver

9c351ef

fix: fix too early period request handling

45858a1

test: add two more GetLightClientUpdatesByRange unit tests

b1c820a

fix: re-org startPeriod / endPeriod adjustment

cae1990

test: more GetLightClientUpdatesByRange test

9710d83

refactor: allow 0 updates

e70363f

test: len(updates) == 0 test

3796442

test: add getLightClientEventBlock unit tests

00d2b0a

chore: remove no-use checking based on comment

18dbc20

fix: patch the error due to rebase

48cfc79

qinlz2 force-pushed the polymer/lightclient-2 branch from ce21485 to 48cfc79 Compare November 18, 2023 15:48

qinlz2 added 2 commits November 21, 2023 13:08

chore: gazelle fix

30fdabe

fix: patch TestGetSpec unit test

d03fa9a

rkapka approved these changes Nov 21, 2023

View reviewed changes

rkapka merged commit 10ccf18 into prysmaticlabs:develop Nov 21, 2023

	http2.HandleError(w, "invalid block root "+err.Error(), http.StatusBadRequest)
	http2.HandleError(w, "invalid block root: "+err.Error(), http.StatusBadRequest)

		Header *apimiddleware.BeaconBlockHeaderJson `json:"header"`
		CurrentSyncCommittee *apimiddleware.SyncCommitteeJson `json:"current_sync_committee"`

[2/5] light client http api #12984

[2/5] light client http api #12984

Conversation

nicopernas commented Oct 3, 2023 • edited Loading

rkapka commented Oct 3, 2023

nicopernas commented Oct 3, 2023

rkapka commented Oct 3, 2023

nicopernas commented Oct 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkapka commented Oct 5, 2023 • edited Loading

rkapka commented Oct 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicopernas commented Oct 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkapka Oct 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qinlz2 commented Oct 11, 2023

rkapka commented Nov 6, 2023 • edited Loading

qinlz2 commented Nov 7, 2023

CLAassistant commented Nov 7, 2023 • edited Loading

qinlz2 commented Nov 11, 2023

qinlz2 commented Nov 11, 2023

qinlz2 commented Nov 13, 2023

rkapka commented Nov 16, 2023

rkapka commented Nov 16, 2023

rkapka Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qinlz2 commented Nov 17, 2023

qinlz2 commented Nov 17, 2023

qinlz2 commented Nov 18, 2023

rkapka commented Nov 20, 2023

rkapka commented Nov 20, 2023

qinlz2 commented Nov 21, 2023

rkapka commented Nov 21, 2023

nicopernas commented Oct 3, 2023 •

edited

Loading

rkapka commented Oct 5, 2023 •

edited

Loading

rkapka Oct 9, 2023 •

edited

Loading

rkapka commented Nov 6, 2023 •

edited

Loading

CLAassistant commented Nov 7, 2023 •

edited

Loading

rkapka Nov 16, 2023 •

edited

Loading