Skip to content

Commit

Permalink
fix && improvement: rc1 for official master
Browse files Browse the repository at this point in the history
  • Loading branch information
@BTNC
BTNC committed Oct 31, 2016
1 parent 2917d2b commit a57322d
Show file tree
Hide file tree
Showing 14 changed files with 239 additions and 469 deletions.
1 change: 1 addition & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ build/
install/
*/*/build
extra/luafilesystem/lfs.so
win-files/3rd/
105 changes: 36 additions & 69 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,88 +1,55 @@
Self-contained Torch installation for windows
============

## Prerequisites

#### Must have
- MSVC, anyone of the following choices is sufficient
- [Visual C++ Build Tools](http://landinghub.visualstudio.com/visual-cpp-build-tools)
- [Visual Studio Express](https://www.visualstudio.com/vs/visual-studio-express/)
- [Visual Studio Community](https://www.visualstudio.com/vs/community/)
- [Git](https://git-scm.com/download/win)
- [CMake](https://cmake.org/download/#latest)
- [Conda](http://conda.pydata.org/docs/download.html), manage dependencies like openblas, jpeg, qt, etc

#### If use CUDA
[![Build Status](https://travis-ci.org/torch/distro.svg?branch=master)](https://travis-ci.org/torch/distro)

- [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)
- [CUDNN](https://developer.nvidia.com/cudnn), if use dnn
Self-contained Torch installation
============

#### Optional
#### Please refer to the [Torch installation guide](http://torch.ch/docs/getting-started.html#_) for details on how to make a fresh install of Torch on Linux or MacOS.
#### If on windows with msvc, please refer to this [guide](win-files/README.md) for details on installation and usage.

- [MKL](https://software.intel.com/intel-mkl), better performance blas/lapack library
- [Gnuplot](https://sourceforge.net/projects/gnuplot/files/latest)
- [GraphicsMagick](https://sourceforge.net/projects/graphicsmagick/files/latest), optional for image package

## Install
Open "Visual Studio Native Tools Command Prompt" and run:
```bat
install.bat
## Repo content
#### Dependencies
Globally installed dependencies can be installed via:
```bash
bash install-deps
```
By default Torch will be installed under install\ with LuaJIT 2.1 and openlblas from conda environment 'torch'.
There are a few customizable environment variables listed on top of install-deps.bat. There is no need to run
install-deps.bat before run install.bat, it sets variables in global and it will be called directly by install.bat.
*Do not* use lua instead of luajit since lua version Torch will use luaffifb for ffi which has bugs on windows
and has poor performance.

## Use
In order to use Torch in a Self-contained way, a few helper cmd will be installed under the installation directory:
- torchenv.cmd: setup Torch environment including PATH, LUA\_PATH, LUA\_CPATH, CUDNN\_PATH
- luajit.cmd: a wrapper of luajit.exe with Torch environment
- luarocks.cmd: a wrapper of luarocks.bat with Torch environment
- cmake.cmd: a wrapper of cmake.exe which helps package installation with MSVC

#### Use luajit.cmd and luarocks.cmd directly
```bat
path_to_Torch\luajit -ltorch -e "torch.test()"
#### Lua and Torch
The self-contained Lua and Torch installations are performed via:
```bash
./install.sh
```
luarocks install should be run in "Visual Studio Native Tools Command Prompt" or consoles with MSVC setup
```bat
path_to_Torch\luarocks install dpnn
```
Torch manages a repo for all packages' rockspecs, however the rockspecs may be not up-to-date. dpnn is one of that
case when I tried. Main problem is that luarocks on windows does not support commands in multiple lines. Instead,
the latest source of [dpnn](https://github.com/Element-Research/dpnn) should be git cloned, cd dpnn and run:
```bat
path_to_Torch\luarocks make rocks\dpnn-scm-1.rockspec

By default Torch will install LuaJIT 2.1. If you want other options, you can use the command:
```bash
# If a different version was installed, used ./clean.sh to clean it
TORCH_LUA_VERSION=LUA51 ./install.sh
TORCH_LUA_VERSION=LUA52 ./install.sh
```
It will automatically install not installed dependencies.

#### Run torchenv.cmd, then use availabe Torch executables
```bat
path_to_Torch\torchenv.cmd
th
## Update
To update your already installed distro to the latest `master` branch of `torch/distro` simply run:
```bash
./update.sh
```
Trepl on windows should work similarly as on linux or macos.

## Clean or Uninstall
To remove all the temporary compilation files:
```bat
clean.bat
## Cleaning
To remove all the temporary compilation files you can run:
```bash
./clean.sh
```
This will run "git checkout -f" for all packages, so *do not* do modifications in those packages directory, leave them as
synced with github. Do modifications in another cloned folder, and run luarocks.cmd for seperate installation.

To remove the installation:
```bat
uninstall.bat
To remove the installation run:
```bash
# Warning: this will remove your current installation
rm -rf ./install
```
This will remove the install\ directory in addition to clean.bat. Since it has no knowledge where Torch is installed, it
is up to the user to remove the installation directory if Torch is installed in a different place.
You may also want to remove the `torch-activate` entry from your shell start-up script (`~/.bashrc` or `~/.profile`).

## Test
You can test that all libraries are installed properly by running:
```bat
test.bat
```bash
./test.sh
```

Tested on Windows 10, Visual Studio Community 2015, Anaconda4, Cuda8.0, MKL2017
Tested on Ubuntu 14.04, CentOS/RHEL 6.3 and OSX
24 changes: 6 additions & 18 deletions clean.bat
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,17 @@
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::


set GIT_CMD=
for /f "delims=" %%i in ('where git') do (
set GIT_CMD=%%i
goto :AFTER_GIT
)
:AFTER_GIT

if "%GIT_CMD%"=="" goto :HELP

set TORCH_DISTRO=%cd%
set TORCH_DISTRO=%~dp0.

for /d %%G in ("%TORCH_DISTRO%\win-files\3rd\wineditline-*") do rmdir /s /q "%%~G"\build

cd %TORCH_DISTRO%\win-files\3rd\dlfcn-win32 && git clean -fdx

cd %TORCH_DISTRO%\exe\luajit-rocks && git clean -fdx
cd %TORCH_DISTRO%\exe\lua-5.1.5 && del /q *.obj *.o *.lib *.dll *.exp *.exe
cd %TORCH_DISTRO%\exe\lua-5.2.4 && del /q *.obj *.o *.lib *.dll *.exp *.exe
cd %TORCH_DISTRO%\exe\lua-5.3.3 && del /q *.obj *.o *.lib *.dll *.exp *.exe
cd %TORCH_DISTRO%\exe\luajit-2.0 && git clean -fdx
cd %TORCH_DISTRO%\exe\luajit-2.1 && git clean -fdx
cd %TORCH_DISTRO%\exe\luarocks && git clean -fdx

cd %TORCH_DISTRO%\extra\luafilesystem && git clean -fdx
Expand Down Expand Up @@ -55,11 +50,4 @@ cd %TORCH_DISTRO%\extra\graphicsmagick && git clean -fdx
cd %TORCH_DISTRO%\extra\totem && git clean -fdx

echo Cleaning is finished
goto :END

:HELP
echo Git is not installed globally in this system.
echo Please run torch-activate.cmd before run clean.bat so that git in your installation can be found.

:END
@endlocal
83 changes: 55 additions & 28 deletions install-deps.bat
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,20 @@ REM set LAPACK_LIBRARIES=D:\\Libraries\\lib\libopenblas.dll.a
:: where to find cudnn library
REM set CUDNN_PATH=D:\NVIDIA\CUDNN\v5.1\bin\cudnn64_5.dll

:: whether update dependencies if already setup, default to not update
REM set TORCH_UPDATE_DEPS=

:::: End of customization ::::


set ECHO_PREFIX=+++++++
set TORCH_SETUP_FAIL=1

:::: validate lua version ::::

:: [TODO] currently luajit lua luarocks are installed from source. they can be changed to use luajit-rocks when
:: luajit-rocks is ready for windows

if "%TORCH_LUA_VERSION%" == "" set TORCH_LUA_VERSION=LUAJIT21
if "%TORCH_LUA_VERSION%" == "LUAJIT21" (
set TORCH_LUAJIT_VERSION=2.1
Expand All @@ -54,15 +60,15 @@ if "%TORCH_LUA_VERSION%" == "LUAJIT20" (
set TORCH_LUAROCKS_LUA=5.1
)
if "%TORCH_LUA_VERSION%" == "LUA53" (
set TORCH_LUA_SOURCE=lua-5.3
set TORCH_LUA_SOURCE=lua-5.3.3
set TORCH_LUAROCKS_LUA=5.3
)
if "%TORCH_LUA_VERSION%" == "LUA52" (
set TORCH_LUA_SOURCE=lua-5.2
set TORCH_LUA_SOURCE=lua-5.2.4
set TORCH_LUAROCKS_LUA=5.2
)
if "%TORCH_LUA_VERSION%" == "LUA51" (
set TORCH_LUA_SOURCE=lua-5.1
set TORCH_LUA_SOURCE=lua-5.1.5
set TORCH_LUAROCKS_LUA=5.1
)
if "%TORCH_LUA_SOURCE%" == "" (
Expand All @@ -72,7 +78,7 @@ if "%TORCH_LUA_SOURCE%" == "" (

:::: Setup directories ::::

set TORCH_DISTRO=%cd%
set TORCH_DISTRO=%~dp0.
if "%TORCH_INSTALL_DIR%" == "" set TORCH_INSTALL_DIR=%TORCH_DISTRO%\install
set TORCH_INSTALL_BIN=%TORCH_INSTALL_DIR%\bin
set TORCH_INSTALL_LIB=%TORCH_INSTALL_DIR%\lib
Expand Down Expand Up @@ -127,51 +133,73 @@ echo %ECHO_PREFIX% Createing conda environment '%TORCH_CONDA_ENV%' for Torch7 de
conda create -n %TORCH_CONDA_ENV% -c conda-forge vc --yes
set TORCH_CONDA_LIBRARY=%CONDA_DIR%envs\\%TORCH_CONDA_ENV%\\Library

set PATH=%TORCH_CONDA_LIBRARY%\bin;%PATH%;

set TORCH_CONDA_PKGS=%TORCH_DISTRO%\win-files\check_conda_packages_for_torch.txt
conda list -n %TORCH_CONDA_ENV% > %TORCH_CONDA_PKGS%

findstr "openblas" "%TORCH_CONDA_PKGS%" >nul
if not errorlevel 1 set TORCH_SETUP_NO_BLAS=
if "%TORCH_SETUP_NO_BLAS%" == "1" (
echo %ECHO_PREFIX% Installing openblas by conda, since there is no blas library specified
conda install -n %TORCH_CONDA_ENV% -c ukoethe openblas --yes || goto :Fail
)

echo %ECHO_PREFIX% Installing other dependencies by conda for image, qtlua, etc
conda install -n %TORCH_CONDA_ENV% -c conda-forge jpeg libpng zlib libxml2 qt=4.8.7 --yes

echo %ECHO_PREFIX% Installing tools by conda
set PATH=%TORCH_CONDA_LIBRARY%\bin;%PATH%;

for /f "delims=" %%i in ('where git') do (
set GIT_CMD=%%i
goto :AFTER_GIT
findstr "jpeg" "%TORCH_CONDA_PKGS%" >nul
if errorlevel 1 set TORCH_DEPENDENCIES=%TORCH_DEPENDENCIES% jpeg
findstr "libpng" "%TORCH_CONDA_PKGS%" >nul
if errorlevel 1 set TORCH_DEPENDENCIES=%TORCH_DEPENDENCIES% libpng
findstr "zlib" "%TORCH_CONDA_PKGS%" >nul
if errorlevel 1 set TORCH_DEPENDENCIES=%TORCH_DEPENDENCIES% zlib
findstr "libxml2" "%TORCH_CONDA_PKGS%" >nul
if errorlevel 1 set TORCH_DEPENDENCIES=%TORCH_DEPENDENCIES% libxml2
findstr "qt" "%TORCH_CONDA_PKGS%" >nul
if errorlevel 1 set TORCH_DEPENDENCIES=%TORCH_DEPENDENCIES% qt=4.8.7

del /q %TORCH_CONDA_PKGS%

if not "%TORCH_DEPENDENCIES%" == "" (
echo %ECHO_PREFIX% Installing %TORCH_DEPENDENCIES% by conda for Torch7
conda install -n %TORCH_CONDA_ENV% -c conda-forge %TORCH_DEPENDENCIES% --yes
)
:AFTER_GIT
if "%GIT_CMD%" == "" conda install -n %TORCH_CONDA_ENV% -c conda-forge git --yes

for /f "delims=" %%i in ('where cmake') do (
set CMAKE_CMD=%%i
goto :AFTER_CMAKE
)
if "%CMAKE_CMD%" == "" (
echo %ECHO_PREFIX% Installing cmake by conda
conda install -n %TORCH_CONDA_ENV% -c conda-forge cmake --yes
)
:AFTER_CMAKE
if "%CMAKE_CMD%" == "" conda install -n %TORCH_CONDA_ENV% -c conda-forge cmake --yes

set NEW_PATH=%TORCH_CONDA_LIBRARY%\bin;%TORCH_CONDA_LIBRARY%\mingw64\bin;%TORCH_CONDA_LIBRARY%\usr\bin;%NEW_PATH%
set NEW_PATH=%TORCH_CONDA_LIBRARY%\bin;%NEW_PATH%

:::: git clone luajit-rocks ::::
:::: git clone luarocks ::::

echo %ECHO_PREFIX% Git clone luajit-rocks for its tools
echo %ECHO_PREFIX% Git clone luarocks for its tools
cd %TORCH_DISTRO%\exe\
if not exist luajit-rocks\.git git clone https://github.com/torch/luajit-rocks.git

set PATH=%TORCH_DISTRO%\exe\luajit-rocks\luarocks\win32\tools;%PATH%;
if not exist luarocks\.git git clone https://github.com/keplerproject/luarocks.git luarocks
set PATH=%TORCH_DISTRO%\exe\luarocks\win32\tools\;%PATH%;

:::: install lua ::::

echo %ECHO_PREFIX% Installing %TORCH_LUA_SOURCE%
cd %TORCH_DISTRO%\exe\luajit-rocks\%TORCH_LUA_SOURCE%\src
cd %TORCH_DISTRO%\exe\
if not "%TORCH_LUAJIT_VERSION%" == "" (
if not exist %TORCH_LUA_SOURCE%\.git git clone -b v%TORCH_LUAJIT_VERSION% http://luajit.org/git/luajit-2.0.git %TORCH_LUA_SOURCE% || goto :Fail
cd %TORCH_LUA_SOURCE% && ( if "%TORCH_UPDATE_DEPS%" == "1" git pull ) & cd src
) else (
wget -nc https://www.lua.org/ftp/%TORCH_LUA_SOURCE%.tar.gz --no-check-certificate || goto :Fail
7z x %TORCH_LUA_SOURCE%.tar.gz -y >NUL && 7z x %TORCH_LUA_SOURCE%.tar -y >NUL && cd %TORCH_LUA_SOURCE%\src
)
if not "%TORCH_LUAJIT_VERSION%"=="" (
call msvcbuild.bat || goto :FAIL
copy /y jit\* %TORCH_INSTALL_BIN%\lua\jit\
copy /y luajit.h %TORCH_INSTALL_INC%\luajit.h
set LUAJIT_CMD=%TORCH_INSTALL_DIR%\luajit.cmd
) else (
del /q *.obj *.o *.lib *.dll *.exp *.exe
cl /nologo /c /O2 /W3 /D_CRT_SECURE_NO_DEPRECATE /MD /DLUA_BUILD_AS_DLL *.c || goto :FAIL
ren lua.obj lua.o || goto :FAIL
ren luac.obj luac.o || goto :FAIL
Expand All @@ -193,7 +221,7 @@ for %%g in (lua.h,luaconf.h,lualib.h,lauxlib.h) do copy /y %%g %TORCH_INSTALL_IN
echo %ECHO_PREFIX% Installing luarocks
cd %TORCH_DISTRO%\exe\
if not exist luarocks\.git git clone https://github.com/keplerproject/luarocks.git luarocks
cd luarocks && git fetch & call install.bat /F /Q /P %TORCH_INSTALL_ROC% /SELFCONTAINED /FORCECONFIG /NOREG /NOADMIN /LUA %TORCH_INSTALL_DIR% || goto :FAIL
cd luarocks && ( if "%TORCH_UPDATE_DEPS%" == "1" git pull ) & call install.bat /F /Q /P %TORCH_INSTALL_ROC% /SELFCONTAINED /FORCECONFIG /NOREG /NOADMIN /LUA %TORCH_INSTALL_DIR% || goto :FAIL
for /f %%a in ('dir %TORCH_INSTALL_ROC%\config*.lua /b') do set LUAROCKS_CONFIG=%%a
set LUAROCKS_CONFIG=%TORCH_INSTALL_ROC%\%LUAROCKS_CONFIG%
echo rocks_servers = { >> %LUAROCKS_CONFIG%
Expand All @@ -202,7 +230,6 @@ echo [[https://raw.githubusercontent.com/rocks-moonscript-org/moonrocks-mirror
echo } >> %LUAROCKS_CONFIG%

set LUAROCKS_CMD=%TORCH_INSTALL_DIR%\luarocks.cmd
set NEW_PATH=%TORCH_INSTALL_ROC%\tools\;%NEW_PATH%

:::: install wineditline ::::

Expand All @@ -217,7 +244,7 @@ cmake -E make_directory build && cd build && cmake .. -G "NMake Makefiles" -DCMA
echo %ECHO_PREFIX% Installing dlfcn-win32 for thread package
cd %TORCH_DISTRO%\win-files\3rd\
if not exist dlfcn-win32\.git git clone https://github.com/dlfcn-win32/dlfcn-win32.git
cd dlfcn-win32 && git pull
cd dlfcn-win32 && ( if "%TORCH_UPDATE_DEPS%" == "1" git pull )
cmake -E make_directory build && cd build && cmake .. -G "NMake Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=..\ && nmake install
set WIN_DLFCN_INCDIR=%TORCH_DISTRO:\=\\%\\win-files\\3rd\\dlfcn-win32\\include
set WIN_DLFCN_LIBDIR=%TORCH_DISTRO:\=\\%\\win-files\\3rd\\dlfcn-win32\\lib
Expand All @@ -241,7 +268,7 @@ if not "%LUAJIT_CMD%" == "" (
echo %ECHO_PREFIX% Creating torch-activate.cmd lua.cmd luac.cmd luarocks.cmd cmake.cmd
)

set NEW_PATH=%TORCH_INSTALL_BIN%;%TORCH_INSTALL_ROC%;%TORCH_INSTALL_ROC%\tools;%TORCH_INSTALL_ROC%\systree\bin;%NEW_PATH%;%%PATH%%;;
set NEW_PATH=%TORCH_INSTALL_DIR%;%TORCH_INSTALL_BIN%;%TORCH_INSTALL_ROC%;%TORCH_INSTALL_ROC%\tools;%TORCH_INSTALL_ROC%\systree\bin;%NEW_PATH%;%%PATH%%;;
set NEW_LUA_PATH=%TORCH_INSTALL_ROC%\lua\?.lua;%TORCH_INSTALL_ROC%\lua\?\init.lua;%TORCH_INSTALL_ROC%\systree\share\lua\%TORCH_LUAROCKS_LUA%\?.lua;%TORCH_INSTALL_ROC%\systree\share\lua\%TORCH_LUAROCKS_LUA%\?\init.lua;;
set NEW_LUA_CPATH=%TORCH_INSTALL_ROC%\systree\lib\lua\%TORCH_LUAROCKS_LUA%\?.dll;;

Expand Down Expand Up @@ -299,11 +326,11 @@ echo :G_NMake >> %CMAKE_CMD%
echo shift >> %CMAKE_CMD%
echo cmake.exe .. -G "NMake Makefiles" %%* >> %CMAKE_CMD%

set TORCH_SETUP_FAIL=0
echo %ECHO_PREFIX% Setup succeed!
goto :END

:FAIL
set TORCH_SETUP_FAIL=1
echo %ECHO_PREFIX% Setup fail!

:END
Loading

0 comments on commit a57322d

Please sign in to comment.