From 1187a38bb373e9aab65e1da2d1cf374daf43d195 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 18:44:33 -0400 Subject: [PATCH 1/9] added documentation to add a custom binary --- README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/README.md b/README.md index 5f115f5c..b58dd240 100644 --- a/README.md +++ b/README.md @@ -126,3 +126,28 @@ you can execute (it's important that `--jobs` comes before `--engine`): ~~~ gg force --jobs 100 --engine lambda src/frontend/mosh-server ~~~ + +### Adding a Custom Binary + +Make sure that the binary you are using is a self-contained x86-64 Linux ELF executable or shared object. +~~~ +$ file custombinary +custombinary: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=884d9f49a5af3f06df13203d71a980a53de6437a, stripped +~~~ + +gg needs to be informed about the command syntax so that it can create thunks for the corresponding input and output files. This is done by adding a wrapper to `gg/src/models/wrappers/` corresponding to your command syntax. For example, if your command looks like +~~~ +./custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt +~~~ +You can add a file `gg/src/models/wrappers/custombinary` with the following content: +~~~ +#!/bin/bash +model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@infile --outputfile=@outfile" "$@" +~~~ + +Then, gg will be able to understand your command which you can execute this way: +~~~ +$ gg init +$ gg infer custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt +$ gg force o.txt +~~~ \ No newline at end of file From 5570b7c1180ada8ecc408ada567de1c337192b27 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 19:17:07 -0400 Subject: [PATCH 2/9] minor fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b58dd240..12ad0699 100644 --- a/README.md +++ b/README.md @@ -149,5 +149,5 @@ Then, gg will be able to understand your command which you can execute this way: ~~~ $ gg init $ gg infer custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt -$ gg force o.txt +$ gg force output1.txt ~~~ \ No newline at end of file From c651feb6215faba6273bc98ae4366314fa2e9a49 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 19:22:09 -0400 Subject: [PATCH 3/9] added documentation for wrapper file syntax --- README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/README.md b/README.md index 12ad0699..7da191ff 100644 --- a/README.md +++ b/README.md @@ -150,4 +150,15 @@ Then, gg will be able to understand your command which you can execute this way: $ gg init $ gg infer custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt $ gg force output1.txt +~~~ + +### Syntax for wrapper files + +model-generic currently supports only positional arguments and optional arguments. It doesn't support operators like |, >, <, etc. +use `@infile` for any input file that is read, and `@outfile` for any output file that is written to. You can use @ for any argument that gg should ignore like numeric or string arguments. + +A sample wrapper file with all supported argument types: +~~~ +#!/bin/bash +model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@infile --outputfile=@outfile" "$@" ~~~ \ No newline at end of file From 751b781c8c4127113287027e5198694da75ee18e Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 19:22:57 -0400 Subject: [PATCH 4/9] minor addition --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7da191ff..88b6820a 100644 --- a/README.md +++ b/README.md @@ -157,7 +157,7 @@ $ gg force output1.txt model-generic currently supports only positional arguments and optional arguments. It doesn't support operators like |, >, <, etc. use `@infile` for any input file that is read, and `@outfile` for any output file that is written to. You can use @ for any argument that gg should ignore like numeric or string arguments. -A sample wrapper file with all supported argument types: +A sample wrapper file with all supported argument types by model-generic: ~~~ #!/bin/bash model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@infile --outputfile=@outfile" "$@" From 5a00a348db9302ead9f96a06a69b1f6935deeadf Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 23:34:57 -0400 Subject: [PATCH 5/9] added python binary conversion doc --- README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/README.md b/README.md index 88b6820a..4b768fac 100644 --- a/README.md +++ b/README.md @@ -161,4 +161,31 @@ A sample wrapper file with all supported argument types by model-generic: ~~~ #!/bin/bash model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@infile --outputfile=@outfile" "$@" +~~~ + +### Using gg with your own Python script + +Once a python script is compiled to a single file binary, gg can be used in the same way described above. To compile a python script to a binary, a library like [Nuitka](http://nuitka.net/doc/user-manual.html) or Cython can be used. Pyinstaller has its quirks and so doesn't work with gg out of the box because it assumes that the current directory is the location of the executable. A way to use gg with pyinstaller is to create links to the files being referred to. For example, to run this command through gg: +~~~ +$ gg infer pyinstallerbinary input.txt output.txt +~~~ +the thunks will have to be created manually thus: +~~~ +gg create-thunk \ + --value $(gg hash input.txt) \ + --output output.txt \ + --executable $(gg hash argtest) \ + --placeholder output.txt \ + --link input.txt=$(gg hash input.txt) \ + --link argtest=$(gg hash argtest) \ + $(gg hash argtest) + argtest input.txt output.txt +~~~ +To pass optional arguments to your command that gg should ignore, you need to tell it explicitly where the create-thunk options end and your arguments begin; this can be done by passing -- right before passing the positional arguments: +~~~ +gg create-thunk \ + --value $(gg hash input.txt) \ + ... + --output output.txt \ + -- $(gg hash binary) argtest input.txt output.txt --any-option-you-like test ~~~ \ No newline at end of file From 14b687f02cd1eeeb5449154c295e826b89771bd1 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Tue, 28 Jul 2020 23:42:09 -0400 Subject: [PATCH 6/9] added different python script compilation docs --- README.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4b768fac..c87ff9b5 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,23 @@ model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@inf ### Using gg with your own Python script -Once a python script is compiled to a single file binary, gg can be used in the same way described above. To compile a python script to a binary, a library like [Nuitka](http://nuitka.net/doc/user-manual.html) or Cython can be used. Pyinstaller has its quirks and so doesn't work with gg out of the box because it assumes that the current directory is the location of the executable. A way to use gg with pyinstaller is to create links to the files being referred to. For example, to run this command through gg: +Once a python script is compiled to a single file binary, gg can be used in the same way described above. To compile a python script to a binary, a library like [Nuitka](http://nuitka.net/doc/user-manual.html) or Cython can be used. + +Nuitka compilation: +~~~ +$ python -m nuitka --follow-imports customscript.py -o singlefilebinary +~~~ +Cython compilation: +~~~ +$ cython customscript.py --embed +$ gcc -Os -I /usr/include/python3.6m -o singlebinary /home/saurabh/.local/bin/customscript.c -lpython3.6m -lpthread -lm -lutil -ldl +~~~ + +Pyinstaller has its quirks and so doesn't work with gg out of the box because it assumes that the current directory is the location of the executable. +~~~ +$ pyinstaller customscript.py --onefile --distpath . +~~~ +A way to use gg with pyinstaller is to create links to the files being referred to. For example, to run this command through gg: ~~~ $ gg infer pyinstallerbinary input.txt output.txt ~~~ From a956a8299a111622eb829f37603ea6bab637e9d6 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Wed, 29 Jul 2020 15:04:57 -0400 Subject: [PATCH 7/9] removed = from sample commands --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c87ff9b5..fa1e8a15 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ custombinary: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamicall gg needs to be informed about the command syntax so that it can create thunks for the corresponding input and output files. This is done by adding a wrapper to `gg/src/models/wrappers/` corresponding to your command syntax. For example, if your command looks like ~~~ -./custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt +./custombinary 65 input1.txt output1.txt --arg 23 --inputfile input2.txt --outputfile output2.txt ~~~ You can add a file `gg/src/models/wrappers/custombinary` with the following content: ~~~ @@ -148,7 +148,7 @@ model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@inf Then, gg will be able to understand your command which you can execute this way: ~~~ $ gg init -$ gg infer custombinary 65 input1.txt output1.txt --arg=23 --inputfile=input2.txt --outputfile=output2.txt +$ gg infer custombinary 65 input1.txt output1.txt --arg 23 --inputfile input2.txt --outputfile output2.txt $ gg force output1.txt ~~~ From 5354d8136fde04b1c15df03dbdc1fb7d4e2e40bb Mon Sep 17 00:00:00 2001 From: drunksaint Date: Wed, 29 Jul 2020 15:50:51 -0400 Subject: [PATCH 8/9] added information about boolean flags --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index fa1e8a15..3dd7e745 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,7 @@ custombinary: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamicall gg needs to be informed about the command syntax so that it can create thunks for the corresponding input and output files. This is done by adding a wrapper to `gg/src/models/wrappers/` corresponding to your command syntax. For example, if your command looks like ~~~ -./custombinary 65 input1.txt output1.txt --arg 23 --inputfile input2.txt --outputfile output2.txt +./custombinary 65 input1.txt output1.txt --flag --arg 23 --inputfile input2.txt --outputfile output2.txt ~~~ You can add a file `gg/src/models/wrappers/custombinary` with the following content: ~~~ @@ -148,14 +148,14 @@ model-generic "/path/to/custombinary @ @infile @outfile --arg=@ --inputfile=@inf Then, gg will be able to understand your command which you can execute this way: ~~~ $ gg init -$ gg infer custombinary 65 input1.txt output1.txt --arg 23 --inputfile input2.txt --outputfile output2.txt +$ gg infer custombinary 65 input1.txt output1.txt --flag --arg 23 --inputfile input2.txt --outputfile output2.txt $ gg force output1.txt ~~~ ### Syntax for wrapper files model-generic currently supports only positional arguments and optional arguments. It doesn't support operators like |, >, <, etc. -use `@infile` for any input file that is read, and `@outfile` for any output file that is written to. You can use @ for any argument that gg should ignore like numeric or string arguments. +use `@infile` for any input file that is read, and `@outfile` for any output file that is written to. You can use @ for any argument that gg should ignore like numeric or string arguments. Boolean flags (`--flag`) should be excluded from the wrapper. A sample wrapper file with all supported argument types by model-generic: ~~~ From 70d3b12ea8e3136355d1221b1b4bd92f9e455314 Mon Sep 17 00:00:00 2001 From: drunksaint Date: Wed, 29 Jul 2020 20:59:36 -0400 Subject: [PATCH 9/9] bolder custom binary header --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3dd7e745..125d00ab 100644 --- a/README.md +++ b/README.md @@ -127,7 +127,7 @@ you can execute (it's important that `--jobs` comes before `--engine`): gg force --jobs 100 --engine lambda src/frontend/mosh-server ~~~ -### Adding a Custom Binary +## Adding a Custom Binary Make sure that the binary you are using is a self-contained x86-64 Linux ELF executable or shared object. ~~~