[Feature]: Full development environment

[jonbarrow]

Checked Existing

• I have checked the repository for duplicate issues.

What feature do you want to see added?

An easy to setup and run development environment that runs essentially the entire network locally

Why do you want to have this feature?

Currently it's very difficult for new people to contribute as there are hundreds of repositories and many many servers which all talk to each other. Many of our servers/tools also lack instructions for how to run/build them, and it's not clear how services communicate with each other. All of this combined makes it very difficult for new people to join and get setup

Any other details to share? (OPTIONAL)

This issue is related to both PretendoNetwork/Inkay#57 and PretendoNetwork/Martini#13, but has a much larger scope. Discussion about a full development environment was beginning to happen on the Inkay issue which was not a good place for that to happen. This issue is where future discussions of a full development environment should take place

Some basic requirements for this would be, but not limited to:

• Pulling down all the relevant repositories (should this be EVERY repository? Or do we cherry pick?)
• Providing a way to redirect requests to the local servers in a way that allows for them to both be easily ran AND worked on
• Any relevant custom patches
• Some sort of bootstrap step to setup dependencies like Postgres and Mongo and such

All of these requirements already have some suggested solutions

For redirecting requests we have https://github.com/PretendoNetwork/mitmproxy-nintendo, which is already designed to redirect requests from Nintendo's servers to our production ones. By modifying pretendo_addon.py and changing the systems hosts file you can easily point requests to a local server. It was also pointed out by @DaniElectra on Discord that mitimproxy now has better support for older TLS versions, which also improves this:

As for the custom patches, we have 2 solutions for this already:

• Wii U: https://github.com/PretendoNetwork/Meowth is a suite of stand-alone SSL patches. SSL patching is required to connect to local servers. Inkay provides this patching for our use case, but it also changes the URLs to our production ones which may not be desirable to everyone. Using Meowth gives developers a way to have the SSL patches they need but without the opinionated URL patches. Meowth also has the advantage of not needing to use a custom SSL certificate for Miiverse development on the Wii U, which was always a pain point (in the past to do local Miiverse development, you would need to build a custom version of Inkay that used your own SSL certificate)
• 3DS: We unfortunately do not have a Meowth equivalent for the 3DS, which means we are much more limited. The system itself also imposes some limitations. Under normal operation only 1 local account is present at any given time. This makes it hard to have a "developer account" separate from your main accounts, like we can easily do on the Wii U. Nimbus abuses an unused feature of the 3DS that does allow for multiple accounts to exist, by creating a 2nd local account under the test environment and swapping between the two local accounts. This is done to isolate the accounts and give them their own sets of URLs, which our patches can then patch without affecting the normal production URLs of the "Nintendo" account. The 3DS has a 2rd account environment however called dev which should also have its own URLs. We should make a dedicated developer tool for this, which is essentially just a more powerful Nimbus/https://github.com/zaksabeast/3ds-Friend-Account-Manager that lets developers make and manage a 3rd local account, and update our URL patches accordingly? This is the best we can do right now I think. Alternatively, we know that there is a frd sysmodule decomp/reimplementation being worked on (this is not public so I won't say any more than that). Once/if that becomes public, we could possibly use this to replace our URL patches entirely and instead swap the URLs on the fly. This idea has been brought up internally before, since https://github.com/zaksabeast/3ds-friends-sysmodule exists, but this reimplementation is incomplete (lacks all NEX functionality) so it can't be used. However this new reimplementation should have NEX support, so maybe that's more viable? Regardless, one issue still remains: custom SSL certificates for Miiverse. Since we don't have a Meowth equivalent for the 3DS, this means that the Miiverse app does not have any SSL patches and we must use custom ones for local development. This is still a barrier to entry, which hopefully we can solve in the future (either on the consoles or off)

As for pulling down the repos, I had originally suggested submodules (despite knowing they were not the best tool for the job). This was (understandably) shot down. However I think I've found a better tool? I discovered https://gerrit.googlesource.com/git-repo recently which seems to be designed for this sort of workflow. You provide the tool a "manifest" file which contains the links to remote repositories and the paths they should have on disk, and this tool helps manage them. It's used as part of Android development so if it's good enough for Google, it should be good enough for us? We can just have Repo use the manifest to pull down the repositories needed for the development environment

Then we just need a way to run said servers. Maybe something like https://github.com/MatthewL246/pretendo-docker?

My proposed structure for a developer-environment repository would look something like:

developer-environment/
├── patches/
│   └── 3ds/
│       └── nimbus-or-whatever/
├── proxy/
│   └── mitmproxy-nintendo files here
├── projects/
│   ├── account/
│   ├── nex-go/
│   ├── miiverse-api/
│   └── etc.
└── orchestration/
    └── server runner idk

Where projects would NOT come pre-populated and would instead be created using Repo

Working with Miiverse (and other browser-based apps) is something that has been mentioned a number of times. These are pretty large barriers to get past, since they have a combination of issues. These include, but are not limited to:

• Requiring custom SSL certificates/patches (solved on Wii U, but not 3DS)
• Being slow to iterate on (lack of modern web app features resulting in a lot of testing needing to be done, but the consoles/apps are slow to restart and there is no browser tools like a console or refresh button to help with this)

A number of suggestions for this have been proposed. See #5 for more details. We essentially have 3 options:

1. Only use real hardware (most accurate, but the worst to setup and use)
2. Use console emulators (lacking in accurate options since HLE emulation is always going to have some issues and can be somewhat heavy, and not made for this sort of work. However this does work, if setup correctly)
3. Write our own mini browser that matches the specifications of these apps (possibly the most amount of effort to maintain, however would be the best in terms of DX. We know the WebKit versions the apps use, and I've gotten fairly far in compiling the Wii U version, but it still fails. It shows a lot of promise however)

[AToska21]

I've worked with the repo tool (I develop Android, where repo is used for everything.)
The repo tool works perfectly for this purpose. Just create a manifest with the repositories, run "repo sync" and everything is downloaded!

(small note: the projects directory would have to be pre-created, though you could run a script to initialize repo which cannot be done directly on github and sync everything.)

[jonbarrow]

(small note: the projects directory would have to be pre-created, though you could run a script to initialize repo which cannot be done directly on github and sync everything.)

Does this mean just the folder would need to exist, or do the repos need to also exist pre-cloned? I was under the impression that Repo would pull down the repos itself as part of the syncing process (though if not then it's still not a huge deal like you said, can just bootstrap it)

[AToska21]

(small note: the projects directory would have to be pre-created, though you could run a script to initialize repo which cannot be done directly on github and sync everything.)

Does this mean just the folder would need to exist, or do the repos need to also exist pre-cloned? I was under the impression that Repo would pull down the repos itself as part of the syncing process (though if not then it's still not a huge deal like you said, can just bootstrap it)

The folder would need to exist. You could include a bootstrap sh file that would run repo init and repo sync. The repos would be synced by Repo, no need to include them in git. (that's a lot of "repo")

[AToska21]

I've made an example manifest file just to show how it'd work.

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
    <remote name="github" fetch="https://github.com/" />

    <default revision="master" remote="github" sync-j="4" />

    <project name="PretendoNetwork/account" path="account" />
    <project name="PretendoNetwork/friends" path="friends" revision="nex-go-rewrite" />
</manifest>

You'd define the "remote" as GitHub or any other git service. Then, if you wanted to, you'd define some defaults such as the default branch, remote and how many threads should be used to sync. Then you define the repositories, or "projects" as repo calls them. You could define specific branches, or not define any (defaults used). The "path" is where it would be stored. For example, if repo was init in dev-env/projects, and you set the path for the "account" project as just "account", the server would be located in dev-env/projects/account.