Summary of major tech debts

[viniarck]

This is just a summary of existing major tech debts on kytos-ng and which future potential epics to start discussing about them and address them. Major tech debts:

Front-end

1. Vue 2 needs to be migrated to Vue 3

• There's an ongoing effort current in progress

Back-end

0. @listen_to potential out of order events execution is getting cumbersome and tricky to maintain

• Alternative solutions: use @alisten_to or @listen_to(pool="dynamic") if you can afford, however the current MongoDB client with pymongo is sync.
• Refactor existing @listen_to as @alisten_to when there's no blocking dependency down (including threading.Lock instances, those need to be asyncio.Lock, although managing asyncio.Lock is much easier to reason about since all the yield/await points are explicit so certain non suspending that functions that get called might not even need to have a lock mid iteration), asyncio is not a silver bullet though, but compared with the threading use case it should be not only lighter in terms of resources but easier to maintain and reason.
• RabbitMQ can also help here but it's a large infrastructure dependency that would also need a ton of refactoring, trying to add a RabbitMQ-like message broker on Kytos-ng at the moment, would feel like a bazooka to kill a fly, sure it can also help with message persistence, but as of now when event handlers when implemented without race conditions will work reliably and we have a basic support for dead letter queues too.

1. Keep moving more towards asyncio, try to refactor existing synchronous threadpool based endpoints to async. This is related to 1, but for HTTP endpoints that front-end and other API clients use.

2. NApps aren't microservices and kytosd can't be stateful clustered easily without a massive refactoring that would take years. Kytos-ng can be categorized as a majestic monolith, which is great too. A Napp will run in a shared Python runtime, it's just a dynamic lodead "plugin", it's a Python package that gets loaded dynamically. There's room to simplify service-to-service / NApp-to-NApp communication in certain use cases that's a request is being used, concrete example: mef_eline when trying to find paths via pathfinder

• A little bit of context for future readers: Each NApp also doesn't have it's own database and neither it's own HTTP server, the uvicorn/starlette app instance is shared accross all NApps, so there's no full fault-tolerance in terms of process/HTTP/database. Also, many NApp depends on shared resources on core such as Interface, Switch and Link (on topology) and operation on these are using shared memory. The original intention was great to try to make a NApp as a microservice-like via HTTP for service to service communication, but at the moment we get all the downsides of this type of communication without much of the benefits. If one day this were to be a full blown real microservices architecture it could be done if it were to be business justifiable, but it would need a team much bigger, and all NApps to core communication via HTTP too (this is what uonos is doing with gRPC), and much harder and deployment orchestration, and since we don't have any scaling issues that can justify this, we have room to simplify service-to-service communication by not using HTTP to service-to-service and only use it for the front-end (or external 3rd party clients) and service-to-service/NApp-to-NApp via Python callables and/or via the event bus when it makes sense. We don't have any major bottleneck in production use cases, uvicorn/starlette theoretically handle dozens of thousands requests/sec, and MongoDB can scale horizontally also handling dozens of thousands of ops/sec.
• Potential simplification: start allowing NApps to expose/registry which callables either a def func(*args, **kwargs) or async def func(*args, **kwargs) that can be called by any other NApp with Pydantic model validations or trying to reuse validations from openapi-core. A NApp that needs to call other NApp would try to get whether or not the callable has been registered as start using it. There's also the fact that NApps get started in certain order, but long as you start to call it after you get a loaded event it should work without exceptions, and then with rate limiting in the future we can also plug rate limit here.

3. Kytos event bus is meant to be pub sub fanout. Such as when indeed a one or more NApps need to subscribe and subscribe for changes. It shouldn't not be primarily used for background 1-to-1 / channel like communication between two NApps when you need to avoid an HTTP request overhead unless the underlying resource indeeds only responds asynchronously after some time.

• Good usages of 1-to-1: sending flow mods since the error is asynchronously and depends on a switch response anyway.
• Good usages of 1-to-1: fire-and-forget use cases when you know the payload won't fail and when the handling NApp can also deal with errors and ensure an eventual execution.
• Non optimal usages of 1-to-1 events: trying to use a event for a function that can reply immediately or a func that finds network path like on pathfinder. This can work too, but it's a lot of extra code and leg work for virtually no benefit at all, not to mention asynchronously error handling on a different code path. (This can be solved with the proposed simplification on point 2)

4. Start support an additional MongoDB driver such as motor once it's been confirmed it's prod grade at least equal or superior performance (this is related to point 1 on this list).

5. Migrate to MongoDB 7, MongoDB 5 EOL in late 2024.

e2e testing

1. e2e test suite deserved a bit more of attention and review, xfail tests, reruns and overall house keeping.
2. automated on-demand stress tests to really try to catalyze reasonable scalability a bit more than used in prod. (This isn't actually tech debt, but the absence of it sometimes make it harder to quickly stress test certain changes or features that are being developed)
3. e2e upgrade db tests (using an old DB collection), including running DB specific upgrade scripts.

cc'ing @jab1982 for his information, and the core team @italovalcy @Ktmi @Alopalao @rmotitsuki. If you guys also have noticed any major paint point feel free to add in the comments, and have any other suggestion, as we can afford to try to mitigate certain tech, the first step is to start mapping just so they can be potentially considered and prioritized one day. Once they are, it'll be further categorized in a epic and further broken down into tasks.

[Ktmi]

I've actually been looking into async mongo specifically for tyring to move as much code over to async. As you've mentioned one option is motor. It uses very similar syntax to the existing pymongo driver, as it just an async wrapper that runs pymongo in a separate thread. Alternatively, we could wrap our existing calls to the db to run in an executor, and create a db threadpool. Personally, I would suggest the latter, as it reduces the area of code we would have to modify, and ensures maximum compatibility.

As for inter-napp communication, the direction I've been thinking of moving this is into a having generic references to objects or collections that we can then attempt to resolve actions on, similar to graphQL. For example, you want to add a flow to a table on a particular switch, you would ask the switch for the references to its tables, then use the table reference to insert the flow. At the top level, to access a reference you would use the NApp's reference, provided by the controller. Optionally you could create a wrapper class for the references which would automatically make the calls to the resolvers when trying to access an attribute.

[viniarck]

Alternatively, we could wrap our existing calls to the db to run in an executor, and create a db threadpool.

Yes, also being able to fully manage this pool and gracefully shuts it down can be great. We also use this pattern on start_shell_async, and it's definitely Pythonic and well supported. To be honest, I was also expecting motor to be a bit more asyncio first class citizen, specially now in 2024, but yes, probably for their business use case they wanted to have a thin wrapper async layer compatibility being good enough including other ones like tornado that used be popular back in the days. Let's keep this mind, and see if in the future motor will evolve, if by the time we prioritize this migration and it's still the same, then we might as well stick with pymongo in an executor managed by us. In the final result when developing this, let's also benchmark against motor to ensure that at least our implementation is on par, which should be based on their current implementation.

[viniarck]

At the top level, to access a reference you would use the NApp's reference, provided by the controller.

Yes, that'd be great. Allowing NApps to invoke the (exported) callables accordingly. It's certainly also desirable to have a fine control of what is allowed, of course in Python land, this is just a logical boundary, but at least also explicitly marking/decorating/or registrying and then accessing/getting this NApp reference/callable references becomes a bit more explicit and kind of controlled of what it's expected to be called by other NApps (which is also great for developers to also quickly figure out what they can call directly on other NApps).

Yes the idea is to "resolve" a callable, let's try to really make this as dead simple as possible, as it should be just a simple callable call anyway. Then we can also discuss if/how/when Pydantic or any other validation should be used. Good thing about this is that also for existing functionality via endpoints, where applicable we can extract a new func, make it "exportable" and then reuse it as on the API endpoints for front-end or other clients.

[viniarck]

Great suggestions, David.

[Alopalao]

Great summary. I was only aware of the out of order events since I hit it a couple of times with testing. callables is a great addition but are there any plans to automatically run NApps in order despite them being installed out of order?

[Ktmi]

NApps list their dependencies within the kytos.json file. It should be possible to build a dependency graph based on this, then preform a reverse topological sort to find the order which NApps should be loaded.

[viniarck]

Side note:

For future reference it was discussed on this issue #163, and kytos.json last updated time was reasonable enough and good compromise.

Speaking of which, a minor pain point relying on kytos.json stat.ST_MTIME is that when you bump kytos-ng versions then you need to retouch these files. There's room to potentially slighly improve here too. Let us know if you've encountered any other pain point though @Alopalao .

[viniarck]

Front-end

Vue 2

• Vue 2 (EOL on Dec 31st, 2023) needs to be migrated to Vue 3 (epic_ui_towards_vue3 in-progress).

Usability

• SPA: No easy way to display a specific component directly from the URL, such as a view of a k-info-panel, such as an EVC detail. No easy way to export this URL view.
• Could've leverage more link colorings and colored status in the network graph

Code maintenance

• No unit tests
• No e2e browser tests
• No linter and code formatter
• Fix dependencies vulnerabilities (from npm audit)
• Potentially migrate to TypeScript for increased type safety
• Maintain existing k-* components and make them reusable like k-table
• Be able to run components locally with full hot reload with npm server to develop, iterate and style them quickly (without having to refresh the page)
• Try to have richer and traceable error messages when there are HTML component errors

[viniarck]

Back-end

Thread and asyncio platform usage history

• Prior to 2022.1:

  • API endpoints: unbounded threads.
  • Event handlers: unbounded threads.
  • DB: threaded storehouse.
  • Requests lib requests: blocking IO.

• 2023.1:

  • API endpoints: thread pools and asyncio.
  • Event handlers: thread pools and asyncio.
  • DB: pymongo blocking IO.
  • Requests lib httpx: both blocking IO and asyncio.

• 2024.1:

  • Same as 2023.1 except API endpoints expose additional configuration for the api threadpool size

• Kytos-ng core has evolved significantly towards building a core asyncio-compatible. Since version 2023.1 the only remaning blocking IO is pymongo.

• Once a asyncio DB client is added or an asyncio threaded executor is added then it finally unlocks NApps to be fully asyncio compatible, epic_asyncio_mongo_client (this goal is not too far away).

• General goal: keep moving more towards asyncio usage on both Kytos-ng core and NApps.

Thread race conditions

• @listen_to decorator potential out of order events execution can be hard to reason and maintain, when the event order matters or when it's stateful (resulted in dozens of major bugs)

    @listen_to('.*.switch.interface.link_up')
    def on_interface_link_up(self, event):
        interface = event.content['interface']
        with self._intfs_lock[interface.id]:
            if (
                interface.id in self._intfs_updated_at
                and self._intfs_updated_at[interface.id] > event.timestamp
            ):
                return
            self._intfs_updated_at[interface.id] = event.timestamp
            self.handle_link_up(interface)

    @listen_to('.*.switch.interface.link_down')
    def on_interface_link_down(self, event):
        interface = event.content['interface']
        with self._intfs_lock[interface.id]:
            if (
                interface.id in self._intfs_updated_at
                and self._intfs_updated_at[interface.id] > event.timestamp
            ):
                return
            self._intfs_updated_at[interface.id] = event.timestamp
            self.handle_link_down(interface)

• Solutions and guidelines:

• a) Refactor existing @listen_to as @alisten_to when applicable. Newer NApps should use as much asyncio as possible

  • a1) You can't have sync IO blocking the event loop (either refactor them too or wrap in asyncio threadpool executor)
  • a2) asyncio.Lock is needed if you need to ensure that subsequent awaits (yield) execute in order sequentially
  • a3) asyncio.Semaphore is needed if you need to limit the tasks concurrency of a handler

• b) Use @listen_to with a pool="dynamic_single" to turn a handler as single threaded

  • It's only well suited if it doesn't have many IO ops or if the IO is done in bulk

a) @alisten_to example:

    @alisten_to('.*.switch.interface.[link_up|link_down]')
    async def on_interface_link_event(self, event):
        interface = event.content['interface']
        if event.name.endswith("link_down"):
            self.handle_link_down(interface)
        elif event.name.endswith("link_up"):
            self.handle_link_up(interface)
        self.do_notify(event)

a2) @alisten_to asyncio.Lock per interface example:

    @alisten_to('.*.switch.interface.[link_up|link_down]')
    async def on_interface_link_event(self, event):
        interface = event.content['interface']
        async with self._intfs_lock[interface.id]:
            if event.name.endswith("link_down"):
                await self.handle_link_down(interface)
            elif event.name.endswith("link_up"):
                await self.handle_link_up(interface)
            await self.do_notify(event)

b) @listen_to with pool="dynamic_single", single threaded execution:

    @listen_to('.*.switch.interface.[link_up|link_down]', pool='dynamic_single')
    def on_interface_link_event(self, event):
        interface = event.content['interface']
        if event.name.endswith("link_down"):
            self.handle_link_down(interface)
        elif event.name.endswith("link_up"):
            self.handle_link_up(interface)
        self.do_notify(event)

Other examples:

• of_core and of_lldp uses a2) mixed
• pathfinder uses b)
• telemetry_int uses a2), when it was creaed from strach asyncio tasks were used as much as possible

asyncio isn't a silver bullet, but for typical IO threaded concurrent cases it's lighter weight and easier to maintain and reason in Python land.

Kytos-ng is a majestic monolith with plugins NApps

Context:

• NApps aren't microservices and kytosd can't be stateful clustered easily without a massive refactoring that would take years.
• Kytos-ng can be categorized as a majestic monolith, which is great too.
• A Napp will run in a shared Python runtime, it's just a dynamic lodead "plugin".

Tech debt:

• There's room to simplify NApp-to-NApp communication in certain use cases where a request is being used, concrete example: mef_eline when trying to find paths via pathfinder, or when mef_eline tries tries to send flows via flow_manager.
• Kytos-ng core should expose the NApps references on controller.napps[(napp_org, napp_name)] to be get, and then any public callable from managers/* and controllers/* packages can be called, eliminating the request cost, and consequently also simplifying part of the error handling of the request itself too.

Typical NApp high level code architecture organization:

         Presentation                         Logic                        Data Models
+-----------------------------+   +-----------------------------+   +----------------------+
|                             |   |                             |   |                      |
|   API & Event Handlers      |<->|  Controllers & Managers     |<->|   Database Models    |
|                             |   |        API Models           |   |                      |
+-----------------------------+   +-----------------------------+   +----------------------+

• Prerequisites:
• NApps who have too much logic on its main.py should move it to managers/*, and then on main.py API routes or event handlers reuse and remap exceptions to be compatible with either HTTP responses and or the event error handling. This is a relatively large refactoring in some cases. It can be done incrementally.
• If returning data from shared in memory data, return a shallow copy to avoid other mutability issues, or even a deepcopy or custom copy in certain cases when it makes sense. Managers responses should try to be as close to the API as possible to leverage the documentation too.
• This still needs a blueprint and issue discussion but that's the general idea.
• This is also a step that tries to take the most advantage of the current architecture while making it clear that NApps will not ever be fully distributed or turn into full blown microservices.

Move topology links to kytos core

• Similar to how Swiches and Interfaces are managed via core to keep their references easily available, links should be managed via the core too
• cross-ref/tech-debt: move self.links to kytos core topology#203
• mef_eline could start using the correct Links references too

Kytos event bus is meant to be pub sub fanout

Kytos event bus is meant to be pusb sub fanout subscription like.

• Good usages of 1-to-1: flow_manager sending flow mods since the error is asynchronously and depends on a switch response anyway.
• Good usages of 1-to-1: fire-and-forget use cases when you know the payload won't fail and when the handling NApp can also deal with errors and ensure an eventual execution.
• Non optimal usages of 1-to-1 events or difficult to maintain: trying to use a event for a function that can reply immediately or a func that finds network path like on pathfinder.
• Threaded event handlers should also use the appropriate thread pool depending on its purpose.

Migrate to MongoDB 7, MongoDB 5 EOL by late 2024.

• Done in 2024.1
• MongoDB 7.0 will be EOL by August 2026 (still have plenty of time as of Dec 2024)

Migrate ES to 9.X from 7.17

• ES is an optional dependency but part of our development infrastructure
• Kibana, Filebeat and APM server should be upgraded too
• Probably by 2026 when ES 9.X is available (early 2026 might be strategic to upgrade MongoDB too)
• Side note: Italo has tried to upgrade to version 9, APM didn't work, it needs integration needs to be updated and certain env vars need to be covered too to support TLS

Upgrade Python dependencies

• openapi-core is working but outdated
• Most of the current dependencies can have their minor version bumped to get minor upstream fixes
• Python can still stay on 3.11 in 2025, unless we have other strong reason to upgrade it

Maintain core projects on PyPI

• Kytos-ng core projects kytos, python-openflow, kytos-utils ideally should be on PyPI too to facilitate distribution and installation in general
• It might not be worth it if we decide to rebrand Kytos-ng to something else for discoverability and other collaborations

NApps installation and marketplace

• Reasses if there's any interest in making it easier and make certain functionalities of kytos-utils to work again, for instance, instead of the Kytos NApps server (that used to be maintained by SPRACE), but instead clone/install from GitHub instead.
• Right now installing everything by cloning from GitHub is OK, but for better DX/UX experience and discoverability, it be cool it if kytos-utils worked with GitHub as its back-end
• It might only be worth it when the goal is to try to increase the project development partnerships and other general collaboration

e2e (end-to-end) testing

• e2e test suites deserved a bit more of attention and review, xfail tests, reruns and overall house keeping.

Related things that aren't necessarily tech debt, but they would be great to have:

• e2e upgrade db tests (using an old DB collection), including running DB specific upgrade scripts.
• Automated on-demand stress tests to really try to catalyze reasonable scalability a bit more than used in prod
• Make available AmLight like topology and trying to mimic similar resources scalability with mininet (this can be useful for upgrades, and general bug reproduction)

New documentation

• epic_documentation_ng / ReadTheDocs
• Be aware of potential name rebranding for web searchability and collaboration

Final thoughts

When it comes to addressing the tech debts they can be thought from two main ways:

1. Fix the existing tech debt in a next or current version based on what is already clear in terms of requirements. Example, removing requests from certain NApp-to-NApp communication, or moving topology Link(s) instances to core, they can be done on their own regardless.
2. Fix the existing tech debt only when there's a major adjacent or related change. For instance, if building a new feature that's IO related, if possible to introduce it with asyncio, plan for it and do it; or if a threaded part is being hard to maintain or reason, consider planning and replacing it with an asyncio equivalent version.

Finally, if that's being replaced is IO based, stress testing should also be done to ensure that the performance and reliability is at least or par or better. Tech debts being mitigated should be planned for too, similar to other features or fixes, especially if their effort scope is significant.

[jab1982]

@viniarck , That's a great compilation for our next all-hands meeting.

You covered well the software development deficits. One thing we need to consider is new functionalities coming for our network operators and their impact on our software workflow. For instance, we need to decouple from OpenFlow and support more generic southbound protocols (including SONIC). We need network functionalities as well, such as port mirroring, INT, point-to-multipoint, and support for ACL. All of those lead to support for multi-tables and inter-napps flow dependency and manipulation.

Also, we should consider authentication/authorization, better topology monitoring (BFD), new applications and integration with FPGAs and smartnics, and faster path protection.

[viniarck]

Thanks for your review and inputs @jab1982.

Yes, this discussion here is only for "1 ) General Kytos-ng architecture pain points (tech debts)" the first part of the epic_future

The second paragraph regarding the core, will be covered on "2) Initial design considerations to support new remote southbound APIs on Kytos-ng", that will be an initial design considerations to start supporting the use cases we'll go for.

For network functionalities or new applications that might be or not core we should probably break down into new initiatives or other epics, too much for the scope here.

Also, we should consider authentication/authorization, better topology monitoring (BFD), new applications

Same as the previous reply, part can be new features.

and integration with FPGAs and smartnics, and faster path protection.

Integration with FPGAs and smartnics definitely deserves to be treated in the same bucket as southbound-ish related. Let's see how the use cases will be unfolding and then consider how it can be integrated.

[HeriLFIU]

I don't know if I should have made another discussion, but I want to share some ideas as to how we can make the UI easier to maintain for the future.

Front-end

Removal of the Dynamic Loading of Components

I think that loading the components dynamically does more harm than good. We should move all of the UI into the main UI repo and load it all at once. We can keep the vue3-sfc-loader as a side component in case we ever need it for the future or even use it to test components. We could also use Vue 3 async components and distribute components as needed.
Async components allow you to load components when needed and webpack will split a bundle based on these async components into separate files.
If someone would want to add additional UI for the NApps, we document how to do so, and they can add it to the main UI repo.

Dev Server

There is no quick way to test components within the main UI repo. To test changes made to the main UI repo, you need to recompile the UI and place the build within the Kytos web_ui folder.
If the dynamic loading of the components were to be removed, then most of the UI would function within the dev server; what would be missing are the endpoints, but from what I read, they can be mocked, so we can just mock all of the backend and force it to send data from a linear topology or something similar.

Mocking the backend

I read it from these two sources, but in short you can do it by using the proxy option within the webpack config.
https://www.georgestefanis.com/blog/2020/04/29/how-to-mock-the-backend-with-webpack-dev-server/
https://soofka.pl/entry/mocking-and-proxying-http-requests-from-localhost-with-webpack-dev-server-or-express

Testing

Currently, it is very time-consuming and error-prone to just click all of the UI buttons individually. Especially since sometimes they perform a request, and this can branch out, fail, or succeed. This is why it would be good to have unit testing for these little cases. I have not yet tested it, but I'm trying to get a working prototype for unit tests with the Vitest testing framework.

Formatter and Linter

After finishing implementing the testing framework, we can also add a linter and formatter to keep the code well organized.

Additional Notes

Some of the components may need refactoring.
The UI can be made to look better and more responsive in some areas.
I've heard that some would like the information within the UI to be easier to share. I have not thought of a concrete solution for this, but as I've seen and heard mentioned, the best way would probably be to keep the current configuration and data of the UI within the URL.

[HeriLFIU]

While writing unit tests for the components, the time can also be taken to optimize and refactor the code; new Vue 3 features can also be implemented.

[viniarck]

If someone would want to add additional UI for the NApps, we document how to do so, and they can add it to the main UI repo.

That's unfortunately is a deal breaker for us, with the current architecture we should empower developers (including developers who aren't core developers) to still be able to create their UI components without having to add code to the ui, decoupling the development responsibilities too, since not every NApp our core team is responsible to maintain.

[viniarck]

Great points @HeriLFIU, appreciated your contributions, we'll keep these points in minds when evolving the UI and prioritizing new initiatives. Feel free to keep augmenting anything else in this document since it'll be used for upcoming plannings and we'll also discuss about it in the All Hands meeting on Jan 30, 2025.

[HeriLFIU]

I see,
and ok, ill keep the document up to date if anything else pops up, thanks Vinicius

[viniarck]

Right. If one day we decide to pull the plug and no longer support dynamic loaded Vue files from NApps, to simplify code maintenance, and re-thinking other major points where it can simplify for us, then we can for sure try to pursue this direction, but since it impacts on 3rd party collaboration with other developers we need to keep that aligned with other partnership and collaboration goals with the project. Definitely something to discuss in the upcoming too.

[viniarck]

Other tech_debt ones in the back-end worth highlighting too:

#532 (this is priority_major as of Feb 2025)

#534