Code

Compile libmariandecoder to wasm#6

Code

Closed

motin wants to merge 160 commits intomasterbrowsermt/marian-dev:masterfrom compile-wasmbrowsermt/marian-dev:compile-wasmCopy head branch name to clipboard

+1,922-308

Commits

Commits on Oct 14, 2020

• Add makefile and command for attempting marian-decoder wasm compilation

  motin

  committed
  7c4bd68

  Copy full SHA for 7c4bd68

• Avoid sentencepiece header pollution

  motin

  committed
  29e2430

  Copy full SHA for 29e2430

Commits on Oct 15, 2020

• Set up native and wasm cmake-based builds via docker

  motin

  committed
  a834788

  Copy full SHA for a834788

• Prevent sentencepience header pollution (the "VERSION" file in the sentencepiece directory gets included when #include <version> is invoked via emscripten/system/include/libcxx)

  motin

  committed
  154c4b1

  Copy full SHA for 154c4b1

• sentencepiece: Don't require Threads if compiling wasm

  motin

  committed
  5d772bf

  Copy full SHA for 5d772bf

Commits on Nov 12, 2020

• Avoid header pollution (include <version> failed before this change)

  motin

  committed
  7aea659

  Copy full SHA for 7aea659

• Some cleanup

  motin

  committed
  9ff99fb

  Copy full SHA for 9ff99fb

• cmake_cmd += -DUSE_DOXYGEN=off

  motin

  committed
  d07cc58

  Copy full SHA for d07cc58

• Verify that relevant parts of sentencepiece compiles to wasm

  motin

  committed
  c48be77

  Copy full SHA for c48be77

• Clarify docker-folder readme somewhat

  motin

  committed
  b34de06

  Copy full SHA for b34de06

• First round of disabled dependencies for wasm compilation + produce libmariandecoder instead of libmarian

  motin

  committed
  4056730

  Copy full SHA for 4056730

Commits on Nov 13, 2020

• Merge branch 'master' into compile-wasm

  Show description for 0ba5e7f
  motin

  committed
  0ba5e7f

  Copy full SHA for 0ba5e7f

• Avoid header pollution (include <version> failed before this change)

  motin

  committed
  7570ef2

  Copy full SHA for 7570ef2

• Make it possible to build without a .git directory

  motin

  committed
  82e5f5b

  Copy full SHA for 82e5f5b

• Append rather than set CMAKE_MODULE_PATH since emscripten also wants to modify it

  motin

  committed
  8326954

  Copy full SHA for 8326954

• Start each compilation with a clean build dir

  motin

  committed
  2ef26b6

  Copy full SHA for 2ef26b6

• Revert a previous header-pollution workaround (which now has been addressed in another manner)

  motin

  committed
  6ae2f56

  Copy full SHA for 6ae2f56

• Use VERBOSE=1 for make commands

  motin

  committed
  80466cb

  Copy full SHA for 80466cb

Commits on Nov 14, 2020

• Make the COMPILE_WASM option explicit

  motin

  committed
  8de767f

  Copy full SHA for 8de767f

Commits on Nov 16, 2020

• Disable all release-build flags when compiling to wasm

  motin

  committed
  241eaab

  Copy full SHA for 241eaab

• Install wasm protobuf in /usr/opt instead of /root

  motin

  committed
  4e8ce2a

  Copy full SHA for 4e8ce2a

Commits on Nov 18, 2020

• Avoid setting compile option BUILD_ARCH to 'native'

  Show description for 9f826f0
  abhi-agg

  committed
  9f826f0

  Copy full SHA for 9f826f0

• Enable USE_SSE2 compile definition for WASM builds

  Show description for 4e6ebe0
  abhi-agg

  committed
  4e6ebe0

  Copy full SHA for 4e6ebe0

• Removed use of 'register' storage class

  Show description for d15a2e8
  abhi-agg

  committed
  d15a2e8

  Copy full SHA for d15a2e8

• Update intgemm submodule

  Show description for e908d50
  abhi-agg

  committed
  e908d50

  Copy full SHA for e908d50

• Updated intgemm submodule

  Show description for 0398c9e
  abhi-agg

  committed
  0398c9e

  Copy full SHA for 0398c9e

• Updated intgemm submodule

  Show description for 85088a1
  andrenatal

  committed
  85088a1

  Copy full SHA for 85088a1

• Removing zstr from file_stream.h

  andrenatal

  committed
  237459e

  Copy full SHA for 237459e

• Remove FAISS

  andrenatal

  committed
  54282d1

  Copy full SHA for 54282d1

Commits on Nov 19, 2020

• Stubbing a path to make Pathie compile

  andrenatal

  committed
  d627a77

  Copy full SHA for d627a77

• Compile marian-decoder.cpp

  Show description for 3f3aaeb
  abhi-agg

  committed
  3f3aaeb

  Copy full SHA for 3f3aaeb

• Avoid linking sentencepiece_train for wasm builds

  abhi-agg

  committed
  824aa48

  Copy full SHA for 824aa48

• Added __unix__ preprocessor around unistd.h

  Show description for 3c1c54b
  abhi-agg

  committed
  3c1c54b

  Copy full SHA for 3c1c54b

• Added more sources for WASM compilation

  Show description for 9ed75ea
  abhi-agg

  committed
  9ed75ea

  Copy full SHA for 9ed75ea

• Fix two small typos

  motin

  committed
  a8e4a89

  Copy full SHA for a8e4a89

• Put faiss cmake inclusions in if-statement rather than removing

  motin

  committed
  5e13d5d

  Copy full SHA for 5e13d5d

Commits on Nov 20, 2020

• Putting file_stream and pathie back

  andrenatal

  committed
  db63f57

  Copy full SHA for db63f57

• __emscripten__ guards on file_stream.cpp

  andrenatal

  committed
  8905596

  Copy full SHA for 8905596

• __emscripten__ guards on cnpy.h

  andrenatal

  committed
  2d8ac08

  Copy full SHA for 2d8ac08

• __emscripten__ guards on sentencepiece_vocab.cpp

  andrenatal

  committed
  6bc8986

  Copy full SHA for 6bc8986

• __emscripten__ guards on logging.cpp

  andrenatal

  committed
  cd6ce50

  Copy full SHA for cd6ce50

• Updating sentencepiece submodule commit id

  andrenatal

  committed
  f409bb5

  Copy full SHA for f409bb5

• Increasing parallel compilation again

  andrenatal

  committed
  8e2b96b

  Copy full SHA for 8e2b96b

• Adding test pages

  andrenatal

  committed
  21a584a

  Copy full SHA for 21a584a

• Cleanup src/CMakeLists.txt

  abhi-agg

  committed
  fdc6fd6

  Copy full SHA for fdc6fd6

• CMAKE flag to generate marian-decoder.html

  Show description for 35c9458
  abhi-agg

  committed
  35c9458

  Copy full SHA for 35c9458

• Removed VERBOSE=1 from build

  Show description for 2133798
  abhi-agg

  committed
  2133798

  Copy full SHA for 2133798

• modifying run-test to use the generated glue page

  andrenatal

  committed
  95f7431

  Copy full SHA for 95f7431

Commits on Nov 21, 2020

• Revert "Removed VERBOSE=1 from build"

  Show description for accc4ca
  motin

  committed
  accc4ca

  Copy full SHA for accc4ca

• Revert "Increasing parallel compilation again"

  Show description for f8991b1
  motin

  committed
  f8991b1

  Copy full SHA for f8991b1

• Use emrun instead of local python server (avoids use of shared array buffer)

  motin

  committed
  b9fe97b

  Copy full SHA for b9fe97b

• Some cleanup

  motin

  committed
  860e826

  Copy full SHA for 860e826

• Enabled packaging and loading of pre-packaged model files

  motin

  committed
  8e522f0

  Copy full SHA for 8e522f0

• Add docs for make clean-wasm

  motin

  committed
  709a87a

  Copy full SHA for 709a87a

• Add hardcoded stdin input (untested since another exception prevents testing at this point)

  motin

  committed
  fee0f70

  Copy full SHA for fee0f70

Commits on Nov 24, 2020

• Attempt to add ability to compile marian decoder only for non-wasm builds

  motin

  committed
  ede62d3

  Copy full SHA for ede62d3

• Adjust some compile flags/guards

  motin

  committed
  62ad7a2

  Copy full SHA for 62ad7a2

Commits on Nov 26, 2020

• Revert a change that was removed in previous commits

  abhi-agg

  committed
  e25effb

  Copy full SHA for e25effb

• Moved logic of BUILD_ARCH being set to native at proper place

  Show description for 18454e6
  abhi-agg

  committed
  18454e6

  Copy full SHA for 18454e6

• First fix for the broken wasm build

  abhi-agg

  committed
  a379190

  Copy full SHA for a379190

• Final fix for broken wasm build

  Show description for 86d6db9
  abhi-agg

  committed
  86d6db9

  Copy full SHA for 86d6db9

• Clean up src/CMakeLists.txt file

  abhi-agg

  committed
  360e1f4

  Copy full SHA for 360e1f4

• More Clean up for src/CMakeLists.txt

  abhi-agg

  committed
  5601daa

  Copy full SHA for 5601daa

• Clean up src/3rd_party/CMakeLists.txt

  abhi-agg

  committed
  99a00f7

  Copy full SHA for 99a00f7

• Fixed the leftover COMPILE_WASM flag

  abhi-agg

  committed
  eb13702

  Copy full SHA for eb13702

• COMPILE_WASM cleanup in CMakeLists.txt

  abhi-agg

  committed
  c3cc293

  Copy full SHA for c3cc293

Commits on Nov 27, 2020

• Replaced __EMSCRIPTEN__ with COMPILE_DECODER_ONLY compile definition

  Show description for ea146f9
  abhi-agg

  committed
  ea146f9

  Copy full SHA for ea146f9

• Revert change in src/3rd_party/catch.hpp

  Show description for de233e8
  abhi-agg

  committed
  de233e8

  Copy full SHA for de233e8

• Makefile modification for correct native compilation

  Show description for 551d1a9
  abhi-agg

  committed
  551d1a9

  Copy full SHA for 551d1a9

• src/common/logging.cpp: More explicit return message

  Show description for be2fd77
  abhi-agg

  committed
  be2fd77

  Copy full SHA for be2fd77

• Separate clean and build targets for native marian decoder compilation

  Show description for 2715214
  abhi-agg

  committed
  2715214

  Copy full SHA for 2715214

• Disabled VERBOSE builds

  Show description for 26582a6
  abhi-agg

  committed
  26582a6

  Copy full SHA for 26582a6

• Fixed a minor typo

  abhi-agg

  committed
  0c376c7

  Copy full SHA for 0c376c7

Commits on Nov 30, 2020

• Disabled FASTOPT for WASM compilation

  Show description for 27bb1a7
  abhi-agg

  committed
  27bb1a7

  Copy full SHA for 27bb1a7

Commits on Dec 1, 2020

• Resolve WASM runtime failure "Cannot enlarge memory arrays"

  Show description for f4d6d90
  abhi-agg

  committed
  f4d6d90

  Copy full SHA for f4d6d90

• Confirm that stdin is fed correctly via the emulation layer

  motin

  committed
  df69dd5

  Copy full SHA for df69dd5

• Add docs for make compile-and-run-test-stdin-wasm

  motin

  committed
  b901034

  Copy full SHA for b901034

Commits on Dec 2, 2020

• Don't set intrinsics arbitrarily to -msse4.1 in a build scenario that doesn't happen

  motin

  committed
  a4c7b3c

  Copy full SHA for a4c7b3c

• Move marian decoder emscripten link flags away from emcc flags, so that we don't have to recompile everything when tweaking link flags

  motin

  committed
  dc6b392

  Copy full SHA for dc6b392

• Allow compiling without using pthreads + various tweaks

  motin

  committed
  1df8bfa

  Copy full SHA for 1df8bfa

• Some more use pthreads guards

  motin

  committed
  3b72eda

  Copy full SHA for 3b72eda

• Add some stderr output to trace execution

  motin

  committed
  95265e7

  Copy full SHA for 95265e7

• Replaced -DMKL with -DUSE_MKL compile option

  Show description for 1d3e02b
  abhi-agg

  committed
  1d3e02b

  Copy full SHA for 1d3e02b

• Switch O2 to O0 to prevent potential issues that stems from optimizations

  motin

  committed
  3e0b707

  Copy full SHA for 3e0b707

• Temporary hardcoding in batch generator to see if wasm code runs further

  motin

  committed
  7ee3968

  Copy full SHA for 7ee3968

• Attempt to get readable c++ exception messages in wasm land

  motin

  committed
  f6995c6

  Copy full SHA for f6995c6

• Include an ABORT in main() to see what happens

  motin

  committed
  db0f150

  Copy full SHA for db0f150

• Fix makefile syntax typo

  motin

  committed
  54856ef

  Copy full SHA for 54856ef

• Revert "Include an ABORT in main() to see what happens"

  Show description for 6e981ac
  motin

  committed
  6e981ac

  Copy full SHA for 6e981ac

• Remove temporary hardcodings related to batch size

  motin

  committed
  05e85a8

  Copy full SHA for 05e85a8

Commits on Dec 3, 2020

• Add try catch in marian decoder main() to avoid the exception bubbling up to JS/WASM layer

  motin

  committed
  286609c

  Copy full SHA for 286609c

• Added onnxjs submodule

  Show description for 24e4567
  abhi-agg

  committed
  24e4567

  Copy full SHA for 24e4567

• Updated "onnxjs" submodule

  abhi-agg

  committed
  a4fa487

  Copy full SHA for a4fa487

• Removed specific GIT_REVISION and SHA from docker/Makefile

  Show description for ded6d97
  abhi-agg

  committed
  ded6d97

  Copy full SHA for ded6d97

• Cmake modifications to add wasm compatible sgemm

  Show description for a36ba54
  abhi-agg

  committed
  a36ba54

  Copy full SHA for a36ba54

• Enabled flag to compile with WASM compatible SGEMM (onnxjs)

  abhi-agg

  committed
  64d8d3a

  Copy full SHA for 64d8d3a

• Updated onnxjs submodule

  abhi-agg

  committed
  ae86530

  Copy full SHA for ae86530

• Added actual sgemm routine from onnxjs

  Show description for e1929ee
  abhi-agg

  committed
  e1929ee

  Copy full SHA for e1929ee

• Updated submodule onnxjs

  abhi-agg

  committed
  a429008

  Copy full SHA for a429008

Commits on Dec 4, 2020

• Revert "Removed specific GIT_REVISION and SHA from docker/Makefile"

  Show description for cf4e48c
  motin

  committed
  cf4e48c

  Copy full SHA for cf4e48c

• Updated onnxjs submodule

  Show description for 268903d
  abhi-agg

  committed
  268903d

  Copy full SHA for 268903d

• Chnaged to SSSE3 for wasm compilation

  Show description for 4f48416
  abhi-agg

  committed
  4f48416

  Copy full SHA for 4f48416

• Cleaner way of running Batch Gnenerator syncronously

  abhi-agg

  committed
  9e115a9

  Copy full SHA for 9e115a9

• Added USE_PTHREADS as compile definition

  Show description for 9e17540
  abhi-agg

  committed
  9e17540

  Copy full SHA for 9e17540

• Enabled use of -pthread compile option for native builds

  Show description for 9c2457d
  abhi-agg

  committed
  9c2457d

  Copy full SHA for 9c2457d

• Added Aborts in ThreadPool class for testing

  Show description for 70f54a3
  abhi-agg

  committed
  70f54a3

  Copy full SHA for 70f54a3

• Added USE_PTHREADS=off for 'compile-decoder-only-native' target

  Show description for d795907
  abhi-agg

  committed
  d795907

  Copy full SHA for d795907

Commits on Dec 5, 2020

• Add make benchmark-decoder-only-native

  motin

  committed
  9fc2e5f

  Copy full SHA for 9fc2e5f

• Small cleanup in native image Dockerfile

  motin

  committed
  a7e18a4

  Copy full SHA for a7e18a4

• Fix typo in benchmark command

  motin

  committed
  0ec1301

  Copy full SHA for 0ec1301

• Re-enable call stacks in native builds

  motin

  committed
  bc7b89d

  Copy full SHA for bc7b89d

Commits on Dec 6, 2020

• Restore wasm build

  motin

  committed
  1a515ce

  Copy full SHA for 1a515ce

• Crank up wasm compilation from O0 to O3

  motin

  committed
  e7d711d

  Copy full SHA for e7d711d

• Set up instructions and config for wasm benchmarking

  motin

  committed
  e188e96

  Copy full SHA for e188e96

• Updated benchmarking config

  motin

  committed
  552569e

  Copy full SHA for 552569e

• Add commands to compile decoder only using onednn (as per https://github.com/browsermt/marian-dev/pull/8/files)

  motin

  committed
  bcc7f2f

  Copy full SHA for bcc7f2f

• Temporarily set mini/maxi batch sizes to 1/1 in esen speed benchmark script

  motin

  committed
  50886ca

  Copy full SHA for 50886ca

• Set OMP_NUM_THREADS=1 esen speed benchmark script

  motin

  committed
  f3b3a82

  Copy full SHA for f3b3a82

Commits on Dec 7, 2020

• Add make benchmark-wasm-compatible-decoder-only-native to readme

  motin

  committed
  2405bd0

  Copy full SHA for 2405bd0

• Some makefile cleanup

  motin

  committed
  0b13431

  Copy full SHA for 0b13431

• Set mini/maxi batch sizes to 32/100 for benchmarking

  motin

  committed
  7e680a9

  Copy full SHA for 7e680a9

• Use shortlist in benchmarks

  motin

  committed
  c42bf24

  Copy full SHA for c42bf24

• Add 100 and 300 top lines versions of newstest2013.es, use 300 by default in benchmarks

  motin

  committed
  d81a401

  Copy full SHA for d81a401

• Fix "wasm_compatible_decoder_only_native_cmake_cmd" Makefile target

  Show description for a6aa75b
  abhi-agg

  committed
  a6aa75b

  Copy full SHA for a6aa75b

• Clean up docker/Makefile

  Show description for 3a3f174
  abhi-agg

  committed
  3a3f174

  Copy full SHA for 3a3f174

• Improved documentation for Makefile targets

  abhi-agg

  committed
  3f2fb98

  Copy full SHA for 3f2fb98

• Use top 300 lines of newstest2013 in esen speed benchmark script

  motin

  committed
  a19f37e

  Copy full SHA for a19f37e

Commits on Dec 9, 2020

• Fix typo that causes emscripten stdout buffer not to be reset

  motin

  committed
  de9ed4e

  Copy full SHA for de9ed4e

• Add make commands for echoing compiling and benchmarking outside of docker

  motin

  committed
  3744e20

  Copy full SHA for 3744e20

• Fix stdin

  motin

  committed
  1a122e2

  Copy full SHA for 1a122e2

• Preserve function names in WASM compiled code

  Show description for db280f1
  abhi-agg

  committed
  db280f1

  Copy full SHA for db280f1

Commits on Dec 10, 2020

• Disable exception catching (this breaks the build but allows us to progress further)

  motin

  committed
  6db5455

  Copy full SHA for 6db5455

Commits on Dec 14, 2020

• Avoid deleting CMakeCache.txt for a build command

  Show description for ed73881
  abhi-agg

  committed
  ed73881

  Copy full SHA for ed73881

• Enabled Exception Catching for WASM builds

  Show description for 0e0042f
  abhi-agg

  committed
  0e0042f

  Copy full SHA for 0e0042f

• Added Logging in fastopt for debugging wasm run time failures

  abhi-agg

  committed
  b288e62

  Copy full SHA for b288e62

• Enabled FASTOPT for WASM builds

  Show description for c2ac607
  abhi-agg

  committed
  c2ac607

  Copy full SHA for c2ac607

Commits on Dec 16, 2020

• Added more logs in fastopt to debug wasm FASTOPT run time failures

  abhi-agg

  committed
  57d297b

  Copy full SHA for 57d297b

Commits on Dec 17, 2020

• Fix to resolve run time failures for FASTOPT enabled wasm builds

  Show description for 78e5fcb
  abhi-agg

  committed
  78e5fcb

  Copy full SHA for 78e5fcb

• Revert "Added more logs in fastopt to debug wasm FASTOPT run time failures"

  Show description for 51f9401
  abhi-agg

  committed
  51f9401

  Copy full SHA for 51f9401

• Revert "Added Logging in fastopt for debugging wasm run time failures"

  Show description for c640b0b
  abhi-agg

  committed
  c640b0b

  Copy full SHA for c640b0b

• Refactoring of FastOpt::makeScalar() function

  Show description for 7584bed
  abhi-agg

  committed
  7584bed

  Copy full SHA for 7584bed

• Cleaned up options.h file

  Show description for 7435c91
  abhi-agg

  committed
  7435c91

  Copy full SHA for 7435c91

• Attempt to compile with -fno-exceptions

  Show description for 2a785e2
  abhi-agg

  committed
  2a785e2

  Copy full SHA for 2a785e2

• Updated submodule sentencepiece

  Show description for 14699fd
  abhi-agg

  committed
  14699fd

  Copy full SHA for 14699fd

• Enable WASM builds without exceptions

  Show description for f0350f9
  abhi-agg

  committed
  f0350f9

  Copy full SHA for f0350f9

• Revert "Enabled Exception Catching for WASM builds"

  Show description for f853a82
  abhi-agg

  committed
  f853a82

  Copy full SHA for f853a82

Commits on Dec 18, 2020

• Set also the native wasm compatible decoder to compile without exceptions

  motin

  committed
  8d3f310

  Copy full SHA for 8d3f310

• Guard a try/catch in translator/output_printer.h

  motin

  committed
  58581bd

  Copy full SHA for 58581bd

• Only add -fno-exceptions if in strict mode (should allow compiling something that ought to run without exceptions being thrown in course of ordinary operation)

  motin

  committed
  0961e08

  Copy full SHA for 0961e08

• Restore build

  motin

  committed
  25aff1e

  Copy full SHA for 25aff1e

• Compile wasm without assertions

  motin

  committed
  7eeea41

  Copy full SHA for 7eeea41

Commits on Jan 7, 2021

• Guard try/catch in rest of the marian source code

  Show description for 9ec0876
  abhi-agg

  committed
  9ec0876

  Copy full SHA for 9ec0876

• Removed try/catch from marian-decoder.cpp

  abhi-agg

  committed
  7a1dff4

  Copy full SHA for 7a1dff4

Commits on Jan 8, 2021

• Guard try/catch in some of marian's 3rd_party sources

  Show description for ac51c16
  abhi-agg

  committed
  ac51c16

  Copy full SHA for ac51c16

• Update intgemm submodule

  Show description for ace098f
  abhi-agg

  committed
  ace098f

  Copy full SHA for ace098f

• Enabled simd shuffle pattern for intgemm compilation

  abhi-agg

  committed
  afaf49f

  Copy full SHA for afaf49f

Commits on Jan 12, 2021

• Clarified wasm docs slightly

  motin

  committed
  87eebe0

  Copy full SHA for 87eebe0

Commits on Jan 13, 2021

• Updated benchmarking scripts and options (focused on intgemm)

  motin

  committed
  b570b3d

  Copy full SHA for b570b3d

• Update intgemm8bit and alphas benchmark scripts to match wasm benchmarks

  motin

  committed
  65ae3b9

  Copy full SHA for 65ae3b9

Commits on Mar 16, 2021

• Updating package-files-wasm directive in README

  andrenatal

  authored
  2799822

  Copy full SHA for 2799822