Migrate nfEngineDemoData to our server

[lookeypl]

Google Drive sometimes performs minor fails when trying to pack and send entire archive with deps. This makes using it much more difficult - we need our own solution to keep nfEngineDemoData.

• Provide easy way to download nfEngineDemoData. Use our recently rented VPS.
• Make sure that this works in all cases and can be easily used to download all the files. Avoid using git repository, however if there is any small revision control system for binary data, use it.
• Update README.md

UPDATE 2015.09.25:
It looks like Git LFS is still in very early development stage, plus LFS server probably will clutter our storage space on VPS. After discussion, here is new, proposed way of keeping the files on nfprojects.org (Advanced features does not have to be developed right away, however might prove useful in the future - these are marked with [Adv] tag):

• A Git-like Python script in Scripts directory which will perform data synchronization between VPS and our project:
  • Script should use native modules to communicate with VPS and download the data (no module download should be needed).
  • Script should have following modes of work (names are subject to change):
    • INIT: with -i/--init arg. Used to initialize the Data folder right after cloning nfEngine repo from GitHub/GerritHub. If the folder already exists, after questioning the user its contents are removed and redownloaded to provide us with some update capabilities.
    • [Adv] PUSH: with -p/--push arg. Pushes current data from the directory to server
    • [Adv] UPDATE: with -u/--update arg. Downloads data from server to repo, if needed
    • [Adv] STATUS: without any arg. Compares current Data folder contents with whatever is available on the server and prints differences (which files were modified, which exist locally but not on FTP server, etc).
  • [Adv] To implement UPDATE/PUSH/STATUS modes, script should keep the information about current version of files and match it with version of files on VPS (ex. by using a file with file list, their modification date, if they were removed etc). Implementation details are to discuss.
  • [Adv] The script probably will be used for more than one folder with Data in the future. Support multiple folders with large files.
• VPS should be properly configured:
  • FTP server launched together with VPS and allowing access to the data (ex. on address ftp://nfprojects.org/nfenginedemodata)
  • Access rights:
    • All should be able to read the data (even people without an account on our VPS)
    • VPS owners should have rights to write files onto the server (probably a group should be created - Linux DAC should handle it)
  • When someone commences PUSH mode from within the script, the server must ask for authentication before an upload commences (this will probably do itself automatically with proper Linux DAC configuration on the server, however it is an requirement so it must be verified if it happens). Adding someones account to the group with access should be the only move to enable pushing large files to VPS.
• Documentation updates: script description in Wiki, "Building the project" chapters on README.md and Wiki

[lookeypl]

GitHub is nice enough to provide a new Git extension, which might help us in this issue.

Use the extension. Check what is needed to host our files on nfprojects.org and make them compatible with Git LFS.

[mkulagowski]

I tried to get it to work and was close to success.

1. Downloaded, configured (a lil' bit) and started it on our VPS.
2. Made temp repo locally, installed git-lfs and tried to upload one file
3. Didn't manage to upload file, but server got POST queries made by my git-lfs.

All questions I'll shoot straight at the lfs-test-server creators:
git-lfs/lfs-test-server#23

[mkulagowski]

SETTING UP SERVER:

1. Download test-server binary from: https://github.com/github/lfs-test-server/releases
2. Change LFS_HOST to nfprojects.org:8080
3. Change LFS_ADMINUSER and LFS_ADMINPASS to known values
4. Enter http://nfprojects.org:8080/mgmt and add a user - these credential You'll use when prompter for credentials during push
5. Change LFS_CONTENTPATH to destination folder (else it will be held in lfs-content subdirectory)
6. start lfs-test-server and nevaaah stop, oh noooee!

SETTING UP LOCAL MACHINE:

1. Download git-lfs installator from: https://github.com/github/git-lfs/releases
2. Unpack it into your local git repo, install
3. Open git bash in your local git repo and write git lfs init
4. Make it track files by e.x. git lfs track "*.zip" or edit .gitattributes file manually
5. Change .git/config file so that it have:
   [lfs]
url = "http://nfprojects.org:8080/some_path"

Should work. Writing git lfs ls-files will list files managed by git-lfs.
While pushing and asked for credentials use those acquired in p.4 of SETTING UP SERVER.

If you have any questions don't hesitate to ask.

[lookeypl]

Nice job!

Can you create a nice-and-neat Wiki page in our project? This is very important info and it will be easily accessible there for future references.

[lookeypl]

I'm taking over the issue and will try to make this work for our future tasks.

[lookeypl]

Minor notes from my side:

• LFS Test Server now has SysV Service script written by me. I will upload it to our nfEngine repo for backup purposes
• Informations on how to add Git LFS support will come with commit migrating the data to our server
• After downloading client-side of git lfs you can simply copy it on Windows to [path_to_git_installation]/bin

LFS Test Server service has known bugs/issues:

• Logging is printed to stdout, not to syslog. Might be a minor pain in the a** if you are doing some jobs as root on our VPS.
• LFS TS might not start automatically after VPS restart - needs checking and, eventually, fixing
• LFS TS has admin credentials written into service file - this is probably LFS TS fault (duh, it's a "test" server), we need to either search for another LFS server, or write our own.

Moving on to uploading test data to LFS

[lookeypl]

After some hiatus from this task I finally did some more research about it and here is what follows:

• Git LFS is a nice solution, however lfs-test-server is not. The test server is indeed, test, and has major flaws:
  • Authorization credentials - to keep the data on LFS server, we need to create a user. These credentials are used both to download and upload data. These need to be separated.
  • Integration with Code Review - uploading mutliple patch sets, each with different version of a heavy file, will flood our VPS.
  • Each file upload requires login - something like public-private key pair authorization would help here.
• Git LFS versions files, which is a nice thing, however this might create a problem with VPS disk space in the future. Something more lightweight without versioning would be a nice solution for now (maybe an FTP server?). This needs investigation.

Right now there are more important tasks to do, so I'm unassigning myself. Maybe, if not me, then someone else will take care of it in the future.

[lookeypl]

Another set of updates after discussion. It looks like the issue is finally coming to an end.

[mkulagowski]

It would be nice to actually write here a comment, concerning what is done and how:

• We have FTP server running on our VPS.
• For public access we have guest and anonymous acc - it's read-only, so downloading is permitted.
• For full control we have accounts created - they are fully customizable along with their passwords, while logged onto VPS.
• FTP is chrooted to a certain directory, so no access to files outside is possible.

First version of python script is currently close to being merged - it contains initialization.
As for the rest of modes: all apart from push can be made using guest acc, PUSH will be made using created account for FTP (so user will be automatically prompted for password, while script is trying to establish connection with server).

Next commit will contain STATUS option, as it is basis for the others, along with some clean-ups for the script and arguments management.

[Witek902]

👍

[mkulagowski]

FTP synchronisation script was merged here and here.
Right now script has INIT, PUSH and UPDATE modes. Instead of keeping everything in a file, the script just compare listings of files along with their modification date. PUSH mode will ask for credentials with proper rights in order to store/delete files.

FTP server works (mostly) - there is only one group, that should be allowed (by proftpd) to write - ftpers. Anonymous/guest virtual read-only accounts are created, also any other user can log in, but no root-login is allowed. TLS is supported.

What should be done next? STATUS mode and?