Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xmas: new runtime driver for directed messaging #617

Draft
wants to merge 72 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
0d2c6a4
wip
liam-fitzgerald Nov 14, 2023
3c4d193
wip
liam-fitzgerald Nov 14, 2023
c6eb231
wip
liam-fitzgerald Nov 14, 2023
ca8f450
wip
liam-fitzgerald Nov 18, 2023
feaca4e
more wip
liam-fitzgerald Nov 18, 2023
ae25315
xmas: improve refcounting
liam-fitzgerald Dec 15, 2023
8624944
wip
liam-fitzgerald Dec 15, 2023
59b5909
wip
liam-fitzgerald Dec 15, 2023
c73485e
wip
liam-fitzgerald Jan 4, 2024
6d0877c
wip
liam-fitzgerald Jan 11, 2024
b4aefe1
wip
liam-fitzgerald Jan 11, 2024
b1790fe
wip
liam-fitzgerald Jan 22, 2024
65e8988
wip
liam-fitzgerald Jan 22, 2024
dd8a9c6
wip
liam-fitzgerald Jan 23, 2024
a151921
xmas: move seralisation into header
liam-fitzgerald Jan 23, 2024
fcd0e4c
add tests
liam-fitzgerald Jan 23, 2024
096af8a
xmas: against jb/dire
liam-fitzgerald Jan 25, 2024
18437e8
xmas: working full flow
liam-fitzgerald Jan 26, 2024
2eb1ce2
xmas: end to end transfer, stubbed congestion control
liam-fitzgerald Jan 26, 2024
1744aa4
Merge remote-tracking branch 'origin/develop' into lf/xmas
liam-fitzgerald Jan 29, 2024
ab89c0e
xmas: refactoring congestion control
liam-fitzgerald Jan 29, 2024
6a11be0
xmas: handle request tables better
liam-fitzgerald Jan 30, 2024
0b10563
wip
liam-fitzgerald Jan 30, 2024
19068f4
wip
liam-fitzgerald Jan 30, 2024
9520641
xmas: improve noun handling speed
liam-fitzgerald Jan 31, 2024
66664fe
xmas: add more congestion control functionality
liam-fitzgerald Feb 2, 2024
0627d3f
xmas: move congestion control to per [ship, lane]
liam-fitzgerald Feb 2, 2024
ef7cca3
wip
liam-fitzgerald Feb 2, 2024
a320608
xmas: finalise congestion control
liam-fitzgerald Feb 5, 2024
4b9cc43
xmas: hacky first pass at blake
liam-fitzgerald Feb 20, 2024
cabf4b3
blake: fix vec impl
liam-fitzgerald Feb 21, 2024
caaabdb
xmas: improve forwarding
liam-fitzgerald Feb 7, 2024
05fb330
xmas: working relaying
liam-fitzgerald Feb 9, 2024
14fa31e
hashtable: add working but leaky u3h_del
liam-fitzgerald Feb 9, 2024
65466f6
hashtable: fix memory leak
liam-fitzgerald Feb 9, 2024
0bd8a47
xmas: better memory mgmt
liam-fitzgerald Feb 22, 2024
e4073d7
xmas: working impl of lss.go
liam-fitzgerald Feb 22, 2024
7ffd05b
xmas: update $name to new format, misc cleanup
joemfb Feb 22, 2024
a42f1eb
xmas: adds $data, updates $page and $poke packets to use it
joemfb Feb 23, 2024
5f7e237
xmas: remove ship from _xmas_etch_pact() signature
joemfb Feb 23, 2024
445645f
xmas: add headers for lss
liam-fitzgerald Feb 23, 2024
c80ed98
xmas: fix ingest return values
liam-fitzgerald Feb 23, 2024
1546812
xmas: removes old $meat types, fixes cursor in _xmas_sift_data()
joemfb Feb 23, 2024
3ba1d10
xmas: parse into correct header struct, assert protocol version
joemfb Feb 23, 2024
dde5c6e
xmas: adds generative roundtrip testing ($poke failing, overflow)
joemfb Feb 23, 2024
c41106e
Merge remote-tracking branch 'origin/jb/xmas' into lf/xmas
liam-fitzgerald Feb 26, 2024
b48e08a
xmas: better blake3 streaming
liam-fitzgerald Feb 26, 2024
bd1d6ca
xmas: stub blake3 lss support
liam-fitzgerald Feb 26, 2024
c787f9f
xmas: enforce lower path length limits in tests, increase samples
joemfb Feb 26, 2024
f7fa447
xmas: adds packet-measurement before etch_*()
joemfb Feb 26, 2024
eb0c726
xmas: fix naming
liam-fitzgerald Feb 26, 2024
7b005d2
xmas: improved logging
liam-fitzgerald Mar 4, 2024
246ec20
xmas: modularise
liam-fitzgerald Mar 4, 2024
8939bd5
xmas: memory fixes
liam-fitzgerald Mar 4, 2024
57dbc7a
xmas: blake non-inline support
liam-fitzgerald Mar 4, 2024
271d4e1
xmas: fix heap corruption
liam-fitzgerald Mar 6, 2024
593558a
xmas: revert unnecessary build changes
joemfb Mar 15, 2024
12a0b63
Merge branch 'develop' into lf/xmas
joemfb Mar 15, 2024
928b24a
xmas: get pact/blake/xmas tests working again
joemfb Mar 15, 2024
de74ab3
xmas: initialize hamt for packet cache, note todos
joemfb Mar 15, 2024
133d3a4
xmas: calculate mugs while etching packets
joemfb Mar 16, 2024
40f015d
xmas: validate mugs while sifting packets
joemfb Mar 16, 2024
e90e5e9
xmas: only try to handle our own effects
joemfb Mar 16, 2024
fc77e92
xmas: target the %mesa vane
joemfb Mar 16, 2024
741634b
xmas: save first fragment after validating inline proof
joemfb Mar 18, 2024
0c2ad80
xmas: inject %page response messages
joemfb Mar 18, 2024
f2da627
xmas: initial %poke handling
joemfb Mar 18, 2024
2bed619
xmas: refactors event queuing in _xmas_hear_page()
joemfb Mar 19, 2024
8c14adb
xmas: adds comments explaining structures, packet handling
joemfb Mar 21, 2024
4619eb5
xmas: restructures %peek and %poke handling
joemfb Mar 21, 2024
bb2f7e1
xmas: separates cache and PIT, unifies %peek and %poke handling
joemfb Mar 23, 2024
7e10047
xmas: separates packet tests
joemfb Apr 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions pkg/c3/motes.h
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
# define c3__bcts c3_s4('b','c','t','s')
# define c3__bczp c3_s4('b','c','z','p')
# define c3__bead c3_s4('b','e','a','d')
# define c3__beam c3_s4('b','e','a','m')
# define c3__bean c3_s4('b','e','a','n')
# define c3__bear c3_s4('b','e','a','r')
# define c3__bede c3_s4('b','e','d','e')
Expand Down Expand Up @@ -539,6 +540,7 @@
# define c3__head c3_s4('h','e','a','d')
# define c3__heal c3_s4('h','e','a','l')
# define c3__hear c3_s4('h','e','a','r')
# define c3__heer c3_s4('h','e','e','r')
# define c3__hela c3_s4('h','e','l','a')
# define c3__helm c3_s4('h','e','l','m')
# define c3__helo c3_s4('h','e','l','o')
Expand Down Expand Up @@ -723,6 +725,7 @@
# define c3__lt c3_s2('l','t')
# define c3__lull c3_s4('l','u','l','l')
# define c3__lunt c3_s4('l','u','n','t')
# define c3__m c3_s1('m')
# define c3__mach c3_s4('m','a','c','h')
# define c3__mack c3_s4('m','a','c','k')
# define c3__made c3_s4('m','a','d','e')
Expand Down Expand Up @@ -750,6 +753,7 @@
# define c3__memo c3_s4('m','e','m','o')
# define c3__menu c3_s4('m','e','n','u')
# define c3__mesh c3_s4('m','e','s','h')
# define c3__mess c3_s4('m','e','s','s')
# define c3__met c3_s3('m','e','t')
# define c3__meta c3_s4('m','e','t','a')
# define c3__mill c3_s4('m','i','l','l')
Expand Down Expand Up @@ -788,6 +792,7 @@
# define c3__mull c3_s4('m','u','l','l')
# define c3__mung c3_s4('m','u','n','g')
# define c3__mut c3_s3('m','u','t')
# define c3__mx c3_s2('m', 'x')
# define c3__n c3_s1('n')
# define c3__na c3_s2('n','a')
# define c3__nail c3_s4('n','a','i','l')
Expand Down Expand Up @@ -851,6 +856,8 @@
# define c3__ovum c3_s4('o','v','u','m')
# define c3__p c3_s1('p')
# define c3__pack c3_s4('p','a','c','k')
# define c3__pact c3_s4('p','a','c','t')
# define c3__page c3_s4('p','a','g','e')
# define c3__pair c3_s4('p','a','i','r')
# define c3__palm c3_s4('p','a','l','m')
# define c3__palq c3_s4('p','a','l','q')
Expand Down Expand Up @@ -1014,6 +1021,7 @@
# define c3__sell c3_s4('s','e','l','l')
# define c3__semp c3_s4('s','e','m','p')
# define c3__send c3_s4('s','e','n','d')
# define c3__sent c3_s4('s','e','n','t')
# define c3__seq c3_s3('s','e','q')
# define c3__serd c3_s4('s','e','r','d')
# define c3__serf c3_s4('s','e','r','f')
Expand Down Expand Up @@ -1321,6 +1329,9 @@
# define c3__wyp c3_s3('w','y','p')
# define c3__wyrd c3_s4('w','y','r','d')
# define c3__xray c3_s4('x','r','a','y')
# define c3__xmas c3_s4('x','m','a','s')
# define c3__x c3_s1('x')
# define c3__xx c3_s2('x','x')
# define c3__yell c3_s4('y','e','l','l')
# define c3__yelp c3_s4('y','e','l','p')
# define c3__z c3_s1('z')
Expand Down
142 changes: 135 additions & 7 deletions pkg/noun/hashtable.c
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,134 @@ u3h_put(u3p(u3h_root) har_p, u3_noun key, u3_noun val)
}
}

static c3_o _ch_slot_del(u3h_slot*, u3_noun, c3_w, c3_w, c3_w*);

/* _ch_buck_del(): delete from bucket
*/
static c3_o
_ch_buck_del(u3h_slot* sot_w, u3_noun key, c3_w *use_w)
{
u3h_buck* hab_u = u3h_slot_to_node(*sot_w);
c3_w fin_w = hab_u->len_w;
c3_w i_w;
//
// find index of key to be deleted
//
for ( i_w = 0; i_w < hab_u->len_w; i_w++ ) {
u3_noun kov = u3h_slot_to_noun(hab_u->sot_w[i_w]);
if ( c3y == u3r_sing(key, u3h(kov)) ) {
fin_w = i_w;
u3z(kov);
break;
}
}

// no key found, no-op
if ( fin_w == hab_u->len_w ) {
return c3n;
}

{
hab_u->len_w--;
u3_assert(c3y == u3h_slot_is_noun(hab_u->sot_w[fin_w]));
//u3z(u3h_slot_to_noun(hab_u->sot_w[fin_w]));
for ( i_w = fin_w; i_w < hab_u->len_w; i_w++ ) {
hab_u->sot_w[i_w] = hab_u->sot_w[i_w + 1];
}

*use_w -= 1;
return c3y;
}
}

static c3_o _ch_some_del(u3h_slot*, u3_noun, c3_w, c3_w, c3_w*);

static c3_o
_ch_slot_del(u3h_slot* sot_w, u3_noun key, c3_w lef_w, c3_w rem_w, c3_w* use_w) {
if ( c3y == u3h_slot_is_noun(*sot_w) ) {
u3_noun kev = u3h_slot_to_noun(*sot_w);
*sot_w = 0;
u3z(kev);
*use_w -= 1;
return c3y;
} else {
return _ch_some_del(sot_w, key, lef_w, rem_w, use_w);
}
}

static c3_o
_ch_node_del(u3h_slot* sot_w, u3_noun key, c3_w lef_w, c3_w rem_w, c3_w* use_w)
{
u3h_node* han_u = (u3h_node*) u3h_slot_to_node(*sot_w);

c3_w bit_w, inx_w, map_w, i_w;

lef_w -= 5;
bit_w = (rem_w >> lef_w);
rem_w = CUT_END(rem_w, lef_w);
map_w = han_u->map_w;
inx_w = _ch_popcount(CUT_END(map_w, bit_w));

// nothing at slot, no-op
if ( !BIT_SET(map_w, bit_w) ) {
return c3n;
}
c3_w ken_w;
u3h_slot kes_w;
u3h_slot don_w = han_u->sot_w[inx_w];
c3_w len_w = _ch_popcount(map_w);
if ( len_w == 2 && ((ken_w = (0 == inx_w) ? 1 : 0),
(kes_w = han_u->sot_w[ken_w]),
(c3y == u3h_slot_is_noun(don_w) && c3y == u3h_slot_is_noun(kes_w))) ) {
*sot_w = kes_w;
u3z(u3h_slot_to_noun(don_w));
*use_w -= 1;
u3a_wfree(han_u);
return c3y;
}
c3_o ret_o = _ch_slot_del(&han_u->sot_w[inx_w], key, lef_w, rem_w, use_w);
if ( c3y == ret_o && c3y == u3h_slot_is_null(han_u->sot_w[inx_w]) ) {
han_u->map_w &= ~(1 << bit_w);
for ( i_w = inx_w; i_w < len_w; i_w++ ) {
han_u->sot_w[i_w] = han_u->sot_w[i_w + 1];
}
}
return ret_o;
}


static c3_o
_ch_some_del(u3h_slot* sot_w, u3_noun key, c3_w lef_w, c3_w rem_w, c3_w* use_w)
{
if ( 0 == lef_w ) {
return _ch_buck_del(sot_w, key, use_w);
}
//u3h_node* han_u = (u3h_node*)han_v;
//if ( _ch_popcount(han_u))
return _ch_node_del(sot_w, key, lef_w, rem_w, use_w);
}



/** TODO: this has a bug where the freeing a hamt that this has been called with failes
* most likelly memroy mgmt
*/
void
u3h_del(u3p(u3h_root) har_p, u3_noun key)
{
u3h_root* har_u = u3to(u3h_root, har_p);
c3_w mug_w = u3r_mug(key);
c3_w inx_w = (mug_w >> 25);
c3_w rem_w = CUT_END(mug_w, 25);
u3h_slot* sot_w = &(har_u->sot_w[inx_w]);

if ( c3y == u3h_slot_is_null(*sot_w) ) {
return;
} else {
_ch_slot_del(sot_w, key, 25, rem_w, &(har_u->use_w));
}
}

/* _ch_uni_with(): key/value callback, put into [*wit]
*/
static void
Expand Down Expand Up @@ -667,6 +795,7 @@ u3h_get(u3p(u3h_root) har_p, u3_noun key)
static void
_ch_free_buck(u3h_buck* hab_u)
{
//fprintf(stderr, "free buck\r\n");
c3_w i_w;

for ( i_w = 0; i_w < hab_u->len_w; i_w++ ) {
Expand All @@ -678,7 +807,7 @@ _ch_free_buck(u3h_buck* hab_u)
/* _ch_free_node(): free node.
*/
static void
_ch_free_node(u3h_node* han_u, c3_w lef_w)
_ch_free_node(u3h_node* han_u, c3_w lef_w, c3_o pin_o)
{
c3_w len_w = _ch_popcount(han_u->map_w);
c3_w i_w;
Expand All @@ -687,17 +816,16 @@ _ch_free_node(u3h_node* han_u, c3_w lef_w)

for ( i_w = 0; i_w < len_w; i_w++ ) {
c3_w sot_w = han_u->sot_w[i_w];

if ( _(u3h_slot_is_noun(sot_w)) ) {
if ( _(u3h_slot_is_null(sot_w))) {
} else if ( _(u3h_slot_is_noun(sot_w)) ) {
u3z(u3h_slot_to_noun(sot_w));
}
else {
} else {
void* hav_v = u3h_slot_to_node(sot_w);

if ( 0 == lef_w ) {
_ch_free_buck(hav_v);
} else {
_ch_free_node(hav_v, lef_w);
_ch_free_node(hav_v, lef_w, pin_o);
}
}
}
Expand All @@ -721,7 +849,7 @@ u3h_free(u3p(u3h_root) har_p)
else if ( _(u3h_slot_is_node(sot_w)) ) {
u3h_node* han_u = u3h_slot_to_node(sot_w);

_ch_free_node(han_u, 25);
_ch_free_node(han_u, 25, i_w == 57);
}
}
u3a_wfree(har_u);
Expand Down
7 changes: 7 additions & 0 deletions pkg/noun/hashtable.h
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,13 @@
u3_weak
u3h_git(u3p(u3h_root) har_p, u3_noun key);

/* u3h_del(); delete from hashtable.
**
** `key` is RETAINED
*/
void
u3h_del(u3p(u3h_root) har_p, u3_noun key);

/* u3h_trim_to(): trim to n key-value pairs
*/
void
Expand Down
55 changes: 55 additions & 0 deletions pkg/noun/hashtable_tests.c
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
/// @file

#include "noun.h"
#define TEST_SIZE 100000

// defined in noun/hashtable.c
c3_w _ch_skip_slot(c3_w mug_w, c3_w lef_w);
Expand All @@ -14,6 +15,59 @@ _setup(void)
u3m_pave(c3y);
}

/* _test_put_del():
*/
static c3_i
_test_put_del()
{
u3p(u3h_root) har_p = u3h_new();
c3_i ret_i = 1;

c3_w i_w;
for ( i_w = 0; i_w < TEST_SIZE; i_w++ ) {
u3_noun key = u3i_word(i_w);
u3_noun val = u3nc(u3_nul, u3k(key));
u3h_put(har_p, key, val);
u3z(key);
}
fprintf(stderr, "inserted");

for ( i_w = 0; i_w < TEST_SIZE; i_w++ ) {
u3_noun key = u3i_word(i_w);
u3_weak val = u3h_get(har_p, key);
if ( val == u3_none ) {
fprintf(stderr, "failed insert\r\n");
ret_i = 0;
}
u3z(key);
u3z(val);
}
fprintf(stderr, "presence");
c3_w del_w[4] = {30, 82, 4921, 535};

for ( i_w = 0; i_w < 4; i_w++ ) {
u3_noun key = u3i_word(del_w[i_w]);
u3h_del(har_p, key);
u3z(key);
}
fprintf(stderr, "deleted");

for ( i_w = 0; i_w < 4; i_w++ ) {
u3_noun key = u3i_word(del_w[i_w]);
u3_weak val = u3h_get(har_p, key);
if ( u3_none != val ) {
fprintf(stderr, "failed delete\r\n");
ret_i = 0;
break;
}
}
fprintf(stderr, "presence two");
u3h_free(har_p);
fprintf(stderr, "freed");

return ret_i;
}

/* _test_bit_manipulation():
*/
static c3_i
Expand Down Expand Up @@ -220,6 +274,7 @@ _test_hashtable(void)
ret_i &= _test_skip_slot();
ret_i &= _test_cache_trimming();
ret_i &= _test_cache_replace_value();
ret_i &= _test_put_del();

return ret_i;
}
Expand Down
5 changes: 5 additions & 0 deletions pkg/noun/jets/k.h
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,11 @@
u3_weak
u3kdb_put(u3_noun a, u3_noun b, u3_noun c);

/* u3kdb_del(): map del for key `b`
*/
u3_weak
u3kdb_del(u3_noun a, u3_noun b);

/* u3kdb_has(): test for get.
*/
u3_noun
Expand Down
2 changes: 2 additions & 0 deletions pkg/noun/jets/q.h
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
u3_noun u3qdb_any(u3_noun, u3_noun);
u3_noun u3qdb_apt(u3_noun);
u3_noun u3qdb_bif(u3_noun, u3_noun);
u3_noun u3qdb_del(u3_noun, u3_noun);
u3_noun u3qdb_dif(u3_noun, u3_noun);
u3_noun u3qdb_gas(u3_noun, u3_noun);
u3_noun u3qdb_get(u3_noun, u3_noun);
Expand All @@ -102,6 +103,7 @@

u3_noun u3qdi_apt(u3_noun);
u3_noun u3qdi_bif(u3_noun, u3_noun);
u3_noun u3qdi_del(u3_noun, u3_noun);
u3_noun u3qdi_dif(u3_noun, u3_noun);
u3_noun u3qdi_gas(u3_noun, u3_noun);
u3_noun u3qdi_has(u3_noun, u3_noun);
Expand Down
Loading
Loading