Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use NATS #8118

Open
wants to merge 341 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
341 commits
Select commit Hold shift + click to select a range
db5011e
nats: incorporate open-files service into project so it automatically…
williamstein Feb 1, 2025
2936021
nats sync: backend syncdocs working with proper doctype
williamstein Feb 1, 2025
bc2d855
nats auth: implement function to remove project permissions
williamstein Feb 1, 2025
a18e29f
nats: add generic pub/sub with event emitter, which I'm now using to …
williamstein Feb 1, 2025
a72862d
nats cursors -- make it work :-)
williamstein Feb 2, 2025
538188a
Merge branch 'master' into nats
williamstein Feb 2, 2025
82ac7c0
enable nats-based synctable for database changefeeds; also make the i…
williamstein Feb 2, 2025
5ea6e0c
nats: integrate hub database service; refactor and use getAll more
williamstein Feb 2, 2025
b4567d3
nats: typescript fix and reminder
williamstein Feb 2, 2025
851c7a6
nats: improve docs and initial configuration
williamstein Feb 2, 2025
02b0af2
nats: tiny bit more docs
williamstein Feb 2, 2025
937f5b8
don't recomend nats env
williamstein Feb 2, 2025
9028029
nats database changefeeds -- rewrite how they work to address several…
williamstein Feb 2, 2025
8412f2d
nats database synctable: implement a major optimization
williamstein Feb 2, 2025
090ebfc
sync editing -- commit more frequently
williamstein Feb 2, 2025
6d17cb8
nats: eliminate use of hub websocket for changefeeds... but
williamstein Feb 2, 2025
6b43eb7
nats query client: fix return format so now crm works fine :-)
williamstein Feb 2, 2025
4653cc2
nats: work in progress refactoring touch, touch_project
williamstein Feb 2, 2025
3dd81a2
nats: eliminate old touch codepath -- use new api
williamstein Feb 3, 2025
e97c18b
user_tracking: switch to nats api; at admin setting so it is disabled…
williamstein Feb 3, 2025
092bfc5
remove web browser prometheus reporting
williamstein Feb 3, 2025
5e8856e
nats/api: only use new api for managing api keys... and
williamstein Feb 3, 2025
c12c036
delete ancient deprecated code for zendesk in the app
williamstein Feb 3, 2025
197fe9a
nats: rewrite streaming LLM output to use ephemeral nats jetstreams a…
williamstein Feb 3, 2025
5875d37
nats: implement a very cool key:value store wrapper, which we're goin…
williamstein Feb 4, 2025
c5af663
nats: framework for system config -- not used yet
williamstein Feb 4, 2025
f9237d3
nats hub: add authtoken rpc
williamstein Feb 4, 2025
bb106ff
database: rewrite central log save in typescript
williamstein Feb 4, 2025
acb2a1a
nats: switch admin user impersonate to new api
williamstein Feb 4, 2025
bb57170
nats hub -- rewrite user search
williamstein Feb 4, 2025
2df7477
nats: rewrite getNames to use new api
williamstein Feb 4, 2025
fa63cde
nats: user new api for getBalance
williamstein Feb 4, 2025
21f831d
Merge branch 'master' into nats
williamstein Feb 4, 2025
867f350
oops -- metrics-recorder is of course used for more than just browser…
williamstein Feb 4, 2025
76a6610
nats: new api for create project
williamstein Feb 4, 2025
8c04d87
nats: add new api support for read/write text file to project
williamstein Feb 4, 2025
c90e4e3
nats: improve the kv a little more
williamstein Feb 4, 2025
108f3fe
nats: improve kv store even more
williamstein Feb 4, 2025
6ff85e5
fix some typescript issue with kv
williamstein Feb 5, 2025
10390fa
nats: work in progress rewriting copy path between projects...
williamstein Feb 5, 2025
e0a4c00
nats: first attempt at simple browser api.
williamstein Feb 5, 2025
c46e0c3
nats: clean up how sign in cookies are set; also set a new account_id…
williamstein Feb 5, 2025
2ef3d16
nats: fix changefeeds (they were hanging due to a refactor)
williamstein Feb 5, 2025
fcf157c
nats: make my req/reply api services be registered NATS microservices…
williamstein Feb 6, 2025
c103fd8
nats: locking down the JWT permissions rules
williamstein Feb 6, 2025
6c8c6b9
nats: fix terminal streams permissions
williamstein Feb 6, 2025
743f46d
nats auth: get rid of redundant code for specifying rules
williamstein Feb 6, 2025
e17448c
account prefs save error crashed server so fix this and make error be…
williamstein Feb 6, 2025
3e8bb44
nats: unify and improve the default connection parameters
williamstein Feb 6, 2025
9770162
nats: trying a little "eventually consistent KV" POC
williamstein Feb 6, 2025
f4579bc
nats kv: work in progress
williamstein Feb 7, 2025
8041513
nats: more work on eventually consistent key value store
williamstein Feb 7, 2025
b28442e
nats distributed key value store -- rename and document
williamstein Feb 7, 2025
eb977e3
nats: adding a stream wrapper
williamstein Feb 7, 2025
c69423e
nats stream class: much, much better
williamstein Feb 8, 2025
d4deba6
nats: create dstream = distributed eventually consistent stream class
williamstein Feb 8, 2025
6ff903b
nats: add stream limits
williamstein Feb 8, 2025
98b6ac2
nats filtered stream: implement all the limits
williamstein Feb 8, 2025
3c461b9
nats streams: support start_seq
williamstein Feb 8, 2025
18b8433
nats streams: better cache
williamstein Feb 8, 2025
bae0e05
nats: easy use of account/project streams from backend
williamstein Feb 8, 2025
7c1793e
nats stream: ability to fill in old values
williamstein Feb 8, 2025
88bde9a
nats streams: s[n] notation; new public streams (hub can write; proje…
williamstein Feb 8, 2025
017067e
nats: user friendly account/project/public kv an dkv
williamstein Feb 8, 2025
58d7066
nats: make it so our streams/kv's can have *arbitrary* names, rather …
williamstein Feb 9, 2025
fa1aff2
nats kv: make it possible to use *arbitrary strings* for keys
williamstein Feb 9, 2025
34f1457
nats: switching to new stream for terminal -- work in progress that d…
williamstein Feb 9, 2025
1f2a475
fix a circular reference issue
williamstein Feb 9, 2025
e583350
nats: new streams were hanging on startup
williamstein Feb 9, 2025
f058779
nats terminal: limit history size (in bytes)
williamstein Feb 9, 2025
e76dfca
nats auth: no longer use -terminal stream
williamstein Feb 9, 2025
21636fd
nats: implements limits for kv
williamstein Feb 9, 2025
0991640
nats dkv: implement limits; also handling of rejected changes due to …
williamstein Feb 9, 2025
fa07410
nats dstream: add reject event support
williamstein Feb 9, 2025
df98481
nats: add better time support to kv and stream
williamstein Feb 9, 2025
0a7eb65
nats: rewrite open files tracker to use new dkv
williamstein Feb 9, 2025
9cd3c07
nats: do not throttle dkv and dstream unless there are errors (e.g., …
williamstein Feb 9, 2025
f363cd0
nats: switch to using new dstream for patches table :-)
williamstein Feb 9, 2025
c1d2f6c
nats: broken (!) work in progress switching database changefeeds to n…
williamstein Feb 9, 2025
c903fa1
nats changefeeds -- fix some bugs with rewrite; this makes sense fina…
williamstein Feb 10, 2025
0874950
nats: implemented new "dko" = distributed key:object store, where whe…
williamstein Feb 10, 2025
4339bb8
switch to using new synctable based on new dko.
williamstein Feb 10, 2025
08b7302
nats: fully switched everything to my new kv/stream implementations, …
williamstein Feb 10, 2025
ac0b8ff
nats: subtle issues with deleting from non-atomic distributed changef…
williamstein Feb 11, 2025
5e30be6
messages table: don't merge in messages, since then deleting doesn't …
williamstein Feb 11, 2025
d8f25dc
nats: greatly speedup reading initial kv out of nats
williamstein Feb 11, 2025
6115ccc
nats install: improve install process, document better, newest nats-s…
williamstein Feb 11, 2025
13e61f0
nats: start unit testing; even found a bug in creating a change handler.
williamstein Feb 11, 2025
664afdf
nats dkv testing -- work in progress unit testing
williamstein Feb 11, 2025
3e61607
nats dkv -- more unit tests
williamstein Feb 11, 2025
3e08025
nats: unit testing 3-way merge conflict implementation
williamstein Feb 11, 2025
2e1b556
nats kv -- more unit tests, especially involving 3-way merge
williamstein Feb 11, 2025
e124221
nats: fix some bugs revealed by testing and caused by testing
williamstein Feb 12, 2025
b553711
nats: fix major issue with hangs at scale (due to lack of auth)
williamstein Feb 12, 2025
00a3737
nats database synctable -- free synctables (avoid eventemitter leaks,…
williamstein Feb 12, 2025
01b5fc6
nats: solve the req/reply "inbox security" issue for users via inbox …
williamstein Feb 12, 2025
f4cdd2a
database user tracker: properly fix subtle bug that would cause issue…
williamstein Feb 12, 2025
195d54a
Merge branch 'master' into nats
williamstein Feb 12, 2025
29f1f5b
use test namespace for testing nats kv and streams.
williamstein Feb 12, 2025
26a7221
nats dstream: fix some bugs and create some unit tests
williamstein Feb 12, 2025
4994b84
nats dstream: add unit tests and fix bugs
williamstein Feb 13, 2025
e4c10ec
nats: adjust parallel param and make comment in unit tests about speed
williamstein Feb 13, 2025
825a16c
nats dkv: better testing null/undefined values (and fix bug testing r…
williamstein Feb 13, 2025
f3c53f6
nats: fix the @cocalc/sync test suite by not making *it* use nats
williamstein Feb 13, 2025
1e60bd4
fixing unit tests
williamstein Feb 13, 2025
d426771
delete the exec shell code api test
williamstein Feb 13, 2025
ae89578
admin impersonation: fix some bugs in my new implementation
williamstein Feb 13, 2025
c6961fc
nats open file data: improving it
williamstein Feb 13, 2025
9ee9323
nats sync: fix opening non-string first time
williamstein Feb 13, 2025
139bb21
nats: unit test open-files
williamstein Feb 13, 2025
a6ae9c4
nats sync objects: mostly improving caching and testing
williamstein Feb 13, 2025
db264ec
nats + trimetravel -- make timetravel close syncdoc it opens
williamstein Feb 14, 2025
6f30fa5
nats: create nice abstraction for services
williamstein Feb 14, 2025
b8bdad4
nats -- active jupyter
williamstein Feb 14, 2025
aae1da6
nats: ipywidgets support
williamstein Feb 14, 2025
bbed478
nats: wire up code formatter
williamstein Feb 14, 2025
e6b638f
nats: better call error messages
williamstein Feb 14, 2025
eb0d1b4
nats service: add a typed option
williamstein Feb 14, 2025
12957dd
switch jupyter to use new typed api
williamstein Feb 14, 2025
0e32a10
jupyter api: slightly better typing
williamstein Feb 14, 2025
8691b75
nats: auth -- fix service permissions
williamstein Feb 14, 2025
8159cae
nats sync: fix issue with changefeeds not working
williamstein Feb 15, 2025
a3bc1ea
nats: sage worksheets
williamstein Feb 15, 2025
9deb844
nats sync: expire ipywidgets
williamstein Feb 15, 2025
92ad19d
nats terminal: fixing some issues with initial load and sizing
williamstein Feb 15, 2025
d80b55d
nats: rewriting more terminal functionality
williamstein Feb 15, 2025
75801e3
nats: different approach to typed service RPC with terminal
williamstein Feb 15, 2025
5df9575
nats: yet another attempt at nice typed RPC api microservices
williamstein Feb 15, 2025
5ccf778
nats jupyter api: more typing
williamstein Feb 15, 2025
0e932d1
nats terminal: properly implementing browser side
williamstein Feb 16, 2025
f08bc33
terminal: mostly implement "kick"
williamstein Feb 16, 2025
98b24b9
nats services: incorporate stats/info/ping/waitFor
williamstein Feb 16, 2025
7b215d6
nats: improve terminal startup robustness
williamstein Feb 16, 2025
16b8e0e
nats terminal: implement 'kick'
williamstein Feb 16, 2025
c61fcb4
nats terminal: deleting old code, mostly
williamstein Feb 16, 2025
b636af3
nats: make initial load of stream much faster and always load it all
williamstein Feb 16, 2025
9124f53
dstream -- support optional typing
williamstein Feb 16, 2025
586dfc3
add typing support to DKO, DKV, KV, DStream, Stream
williamstein Feb 16, 2025
cdf0f1c
nats: work in progress rewriting listings
williamstein Feb 17, 2025
3a2c32b
nats: implementing new directory listings watcher
williamstein Feb 17, 2025
2e9c543
nats listings: packaged client
williamstein Feb 17, 2025
0338401
nats listings -- start integrating with frontend
williamstein Feb 17, 2025
f1dbea2
make 'cc.current()' better to aid in debugability of cocalc
williamstein Feb 17, 2025
b16a35c
nats: work in progress on revamping directory listings state
williamstein Feb 17, 2025
708fe11
open-files: dealing with deleted files
williamstein Feb 17, 2025
1b760e4
nats: add monitoring to kv store
williamstein Feb 18, 2025
eec42ba
nats kv: confirmed that my monitor works
williamstein Feb 18, 2025
ca7ea3c
nats-based delete file management: deal with some edge cases exposed …
williamstein Feb 18, 2025
4214fec
nats file deletion handling: work in progress
williamstein Feb 18, 2025
cd3a551
nats: handling deleted files
williamstein Feb 18, 2025
b7c2ca2
nats: starting project and initial listing -- improve
williamstein Feb 18, 2025
d1bb47e
nats compute servers: change frontend to use new nats based manager
williamstein Feb 19, 2025
2f539f0
nats compute servers -- make project open-files use the new manager i…
williamstein Feb 19, 2025
921b1aa
nats compute servers: improve file and terminal switching
williamstein Feb 19, 2025
f2dfc8e
nats: workaround typescript issue
williamstein Feb 19, 2025
ed4433d
nats project connection: automatically reconnect until success
williamstein Feb 19, 2025
9f95980
nats: fix project auth
williamstein Feb 19, 2025
4738cb9
nats: implement time sync using nats jetstream
williamstein Feb 19, 2025
ee4d657
nats time and open files: use sync'd time explicitly
williamstein Feb 19, 2025
b5e6bc5
nats: change delete in open-files so can properly resolve merge confl…
williamstein Feb 19, 2025
03ca5bc
nats open files: track exactly which backend has file opened and when
williamstein Feb 19, 2025
4bbd937
nats: start wokr on streaming file get service
williamstein Feb 19, 2025
2f038d0
nats: implement async generator to read file from compute server or p…
williamstein Feb 20, 2025
d3a8f84
nats: raw files -- work in progress
williamstein Feb 20, 2025
352961a
nats streaming file download: the project/compute server part seems done
williamstein Feb 20, 2025
fef2012
nats file download: more integration throughout the system
williamstein Feb 20, 2025
70e75a9
nats file read -- add auth
williamstein Feb 20, 2025
1465dd4
"raw url" --> "files url" for download -- refactor code so url always…
williamstein Feb 20, 2025
ec94533
nats: create streaming file write service (no testing)
williamstein Feb 21, 2025
5be29e1
nats file write: add example and fix the one (and only?) mistake
williamstein Feb 21, 2025
c2b84be
nats unit tests: reorg sync; write new tests for time and writeFile
williamstein Feb 21, 2025
1e2a704
nats: write unit tests of streaming reads
williamstein Feb 21, 2025
7917aee
nats: fix some unit tests and add some more docs
williamstein Feb 21, 2025
2bc2c08
nats file upload: starting work integrating this with hub (work in pr…
williamstein Feb 21, 2025
be306d4
add env variable to not run nextjs
williamstein Feb 21, 2025
68cbe8c
nats upload: more work in progress
williamstein Feb 21, 2025
65ebf25
nats hub file upload: this approach sucks
williamstein Feb 21, 2025
3d02486
nats file upload: this works (but is still not pretty)
williamstein Feb 21, 2025
4b142f3
nats hub upload: I think this is the hard part, done.
williamstein Feb 21, 2025
91c36da
project: completely remove raw file server! (major security enhanceme…
williamstein Feb 22, 2025
c6aadf8
nats upload: better error handling
williamstein Feb 22, 2025
f9a0a52
nats upload: support directory trees properly
williamstein Feb 22, 2025
35d2664
nats upload: automatically start upload service in project
williamstein Feb 22, 2025
16eddb4
nats upload: show error when there's only one chunk
williamstein Feb 22, 2025
8593e6d
llm: rewrite to use multiresponse instead of jetstream (work in progr…
williamstein Feb 22, 2025
1e9b91e
llm integration: get it working end-to-end with nats multi response
williamstein Feb 23, 2025
566d595
sync: make ephemeral syncstring and testing more sensibly not use nats
williamstein Feb 23, 2025
c4f4be3
tiny comments while looking at some of the project status code
williamstein Feb 23, 2025
d7c6a19
jupyter/nats: fix "close and halt" not working
williamstein Feb 23, 2025
8f0dcd6
jupyter api: rewrite all client code so it gets type checked with typ…
williamstein Feb 23, 2025
af2ef40
nats db query: convert timestamps to Date objects client side (due to…
williamstein Feb 23, 2025
e6821e6
nats: integrate with connection indicator and modal
williamstein Feb 23, 2025
17e4b2f
fix fallout of Date processing
williamstein Feb 23, 2025
6b85edf
fix cursor bug with Jupyter which was -- amusingly -- only visible wh…
williamstein Feb 23, 2025
dbfd905
nats sync: make inventories for our small distributed eventually cons…
williamstein Feb 24, 2025
e0dad23
nats sync inventory: implement dkv part
williamstein Feb 24, 2025
41e41fe
nats inventory: fix little issues
williamstein Feb 24, 2025
bd07b0d
nats: implement inventory for dstream's
williamstein Feb 24, 2025
6452b7a
test mode: disable nats inventory by default; also express frustratio…
williamstein Feb 24, 2025
2a3306c
nats: add desc to inventory
williamstein Feb 24, 2025
98a27e7
nats inventory: make it better
williamstein Feb 24, 2025
1647415
Merge branch 'master' into nats
williamstein Feb 27, 2025
61745de
starting work on new file-server
williamstein Feb 28, 2025
9184e96
file-server: work in progress implementing it
williamstein Feb 28, 2025
f29dcd8
file-server: more work
williamstein Feb 28, 2025
1e1f366
file-server zfs reorg
williamstein Mar 1, 2025
37b6678
file-server: nfs share configuration
williamstein Mar 1, 2025
16628f9
file-server: working on snapshots
williamstein Mar 1, 2025
b966035
file-server: trim snapshots, also getting all modified files
williamstein Mar 1, 2025
83eab72
file-server: function to trim all active project snapshots
williamstein Mar 2, 2025
5a8299e
file-server: minor refactor
williamstein Mar 2, 2025
c920fca
file-server: code to snapshot all recently active projects, if need be
williamstein Mar 2, 2025
af57d84
file-server: implemented 'clone', by just making create project to a …
williamstein Mar 2, 2025
64432c5
file-server: code refactoring (move out nfs)
williamstein Mar 2, 2025
220531c
file-server: more refactoring
williamstein Mar 2, 2025
1c1f9f5
file-server: more refactoring
williamstein Mar 2, 2025
39a4afb
file-server: much better database interface and typescript, etc. -- w…
williamstein Mar 2, 2025
cf7bc77
file-server: getting it to work again
williamstein Mar 2, 2025
d22d538
file-server: get archive/non-archive to work
williamstein Mar 2, 2025
6f0b40d
file-server: misc
williamstein Mar 2, 2025
fd4ef90
file-server: add progress meter for dearchive
williamstein Mar 2, 2025
b6ffbfd
file-server: implement automated bup backups
williamstein Mar 3, 2025
d1cdaaf
nats: fix base url issue in browser client
williamstein Mar 3, 2025
7a219aa
python3
williamstein Mar 4, 2025
75efc14
nats: make the server and port be customizable via env variable
williamstein Mar 4, 2025
54474e8
files-system: ability to force snapshot creation
williamstein Mar 4, 2025
6873b6c
nats nsc: disable logging
williamstein Mar 4, 2025
6db3d7f
Merge branch 'nats' of github.com:sagemathinc/cocalc into nats
williamstein Mar 4, 2025
57901f1
Merge branch 'master' into nats
williamstein Mar 4, 2025
2d81cde
file-server: add a unit test
williamstein Mar 4, 2025
2bdeb8c
file-server tests: only run on system with zpool installed
williamstein Mar 4, 2025
3efeb4d
only test zfs pool if have zfs
williamstein Mar 4, 2025
9f9e099
file-system: much better unit testing of "create project"
williamstein Mar 4, 2025
2044345
upgrade nats-server; unit test archiving of projects
williamstein Mar 4, 2025
d48ea25
file-server: work in progress on NFS unit test
williamstein Mar 4, 2025
b135967
file-server: fix the nfs unit tests
williamstein Mar 4, 2025
f3fb2b7
file-server: docs, improve database schema, fix create
williamstein Mar 4, 2025
b149078
rewrite file-server code to support multiple named filesystems owned …
williamstein Mar 5, 2025
fee208e
terminal: clearer error message
williamstein Mar 5, 2025
dc034aa
file-server: more tests and better organization and support for owner…
williamstein Mar 5, 2025
08dcbbd
file-system: make bup backups read from a snapshot instead of live fi…
williamstein Mar 5, 2025
99f799f
archive filesystem -- always verify that the archive stream is valid
williamstein Mar 5, 2025
5f32b96
file-server: implement send/recv/recompact for streaming replication …
williamstein Mar 5, 2025
d25a3ff
file-system: add stream maintenance
williamstein Mar 5, 2025
a485a00
file-system: implement a replication system (not tested at all yet)
williamstein Mar 6, 2025
5850722
file-server: add ability to change the context easily with the prefix…
williamstein Mar 6, 2025
f893d5c
file-system: a working unit test of pull sync replication!
williamstein Mar 6, 2025
dec8f56
file-system: adding more unit tests
williamstein Mar 6, 2025
47484b9
file-server: unit tested pull/replication functionality!
williamstein Mar 6, 2025
7e75522
file-server sync: keep backups of remote sqlite3 file
williamstein Mar 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
nats: primus -- going to give upon this approach
williamstein committed Jan 30, 2025
commit 445d0483cb1eeb4fea2bdd318316585807c4d118
2 changes: 1 addition & 1 deletion src/packages/frontend/project/websocket/api.ts
Original file line number Diff line number Diff line change
@@ -60,7 +60,7 @@ export class API {
private getChannel = async (channel_name: string) => {
const natsConn = await webapp_client.nats_client.primus(this.project_id);
// TODO -- typing
return natsConn.channel(channel_name) as Channel;
return natsConn.channel(channel_name) as unknown as Channel;
};

call = async (mesg: Mesg, timeout: number) => {
12 changes: 11 additions & 1 deletion src/packages/nats/primus.ts
Original file line number Diff line number Diff line change
@@ -60,7 +60,10 @@ const connections: { [key: string]: Primus } = {};
export function getPrimusConnection(opts: PrimusOptions): Primus {
const key = getKey(opts);
if (connections[key] == null) {
console.log("getPrimus", key, "CREATING", opts);
connections[key] = new Primus(opts);
} else {
console.log("getPrimus", key, "already have it", opts);
}
return connections[key];
}
@@ -126,6 +129,12 @@ export class Primus extends EventEmitter {
constructor({ subject, channelName = "", env, role, id }: PrimusOptions) {
super();

console.log("PRIMUS Creating", {
subject,
id,
channel: channelName,
});

this.subject = subject;
this.channelName = channelName;
this.env = env;
@@ -159,6 +168,7 @@ export class Primus extends EventEmitter {
return;
}
this.state = "closed";
console.log("destroy", getKey(this));
delete connections[getKey(this)];
for (const sub of this.subs) {
sub.close();
@@ -283,7 +293,7 @@ export class Primus extends EventEmitter {
};

channel = (channelName: string) => {
return new Primus({
return getPrimusConnection({
subject: this.subject,
channelName,
env: this.env,