-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathebtut.txt
1969 lines (1470 loc) · 72.5 KB
/
ebtut.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
= EasyBuild Notes
:doctype: book
:toc:
:icons:
:sectlinks
:source-highlighter: pygments
== Introduction
This is a tutorial on how to construct and use `easyconfig` files in the
EasyBuild system. EasyBuild compiles packages based on the contents of these files
so it is very important to get them right.
This tutorial will proceed by examples, with each containing one or more
features not explained in previous examples as well as more in-depth looks
at features that were only briefly explained previously.
We will proceed from the simple to the complex to gain a full understanding
of how to construct `easyconfig` files.
There are also sections in the latter part of the document that go into much
more detail about some of the features.
This is based on the very useful tutorial at:
https://hprc.tamu.edu/wiki/SW:EasyBuild:Building_Your_Own_Software:Another_Example_-\_LLVM[`https://hprc.tamu.edu/wiki/SW:EasyBuild:Building_Your_Own_Software:Another_Example_-\_LLVM`]
=== Useful Links
==== Official Sites
* EasyBuild Home - https://easybuild.io/[`https://easybuild.io/`]
* Official Documentation - https://docs.easybuild.io/en/latest/index.html#[`https://docs.easybuild.io/en/latest/index.html#`]
** Using the EasyBuild Command Line - https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html[`https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html`]
*** Available easyconfig Parameters - https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html#easyconfig-params[`https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html#easyconfig-params`]
*** List of Known Toolchains - https://docs.easybuild.io/en/latest/version-specific/toolchains.html#vsd-list-toolchains[`https://docs.easybuild.io/en/latest/version-specific/toolchains.html#vsd-list-toolchains`]
*** List of easyblocks - https://docs.easybuild.io/en/latest/eb_list_easyblocks.html#basic-usage-easyblocks[`https://docs.easybuild.io/en/latest/eb_list_easyblocks.html#basic-usage-easyblocks`]
** Writing easyconfig Files - https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html[`https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html`]
** Implementing easyblocks - https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html[`https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html`]
* GitHub - https://github.com/easybuilders[`https://github.com/easybuilders`]
** easyconfig archive - https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs[`https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs`]
** easyblock archive - https://github.com/easybuilders/easybuild-easyblocks/tree/develop/easybuild/easyblocks[`https://github.com/easybuilders/easybuild-easyblocks/tree/develop/easybuild/easyblocks`]
==== Tutorials
* *How to Troubleshoot a Failed Easybuild Compilation* - https://confluence.clarkson.edu/display/OITKB/How+to+Troubleshoot+a+Failed+Easybuild+Compilation[`https://confluence.clarkson.edu/display/OITKB/How+to+Troubleshoot+a+Failed+Easybuild+Compilation`]
* *ISC Tutorial 2021* - https://easybuilders.github.io/easybuild-tutorial/2021-isc21/[`https://easybuilders.github.io/easybuild-tutorial/2021-isc21/`]
* *LUST Tutorial 2021* - https://easybuilders.github.io/easybuild-tutorial/2021-lust/[`https://easybuilders.github.io/easybuild-tutorial/2021-lust/`]
* *ISC Tutorial 2020* - https://easybuilders.github.io/easybuild-tutorial/2020-06-isc20/[`https://easybuilders.github.io/easybuild-tutorial/2020-06-isc20/`]
* *Writing/Contributing EasyConfigs* (PDF) - https://users.ugent.be/\~kehoste/EasyBuild_20190130_tutorial-easyconfigs.pdf[`https://users.ugent.be/~kehoste/EasyBuild_20190130_tutorial-easyconfigs.pdf`]
* 2020 Video Tutorial
** 0: Practical Information - https://www.youtube.com/watch?v=CiQ-bfFL2FA[`https://www.youtube.com/watch?v=CiQ-bfFL2FA`]
** 1: Introduction - https://www.youtube.com/watch?v=gdG4smrZgjc[`https://www.youtube.com/watch?v=gdG4smrZgjc`]
** 2: Installing EasyBuild - https://www.youtube.com/watch?v=qFaBB6esk9k[`https://www.youtube.com/watch?v=qFaBB6esk9k`]
** 3: Configuring EasyBuild - https://www.youtube.com/watch?v=1bxlgkQfc_g[`https://www.youtube.com/watch?v=1bxlgkQfc_g`]
** 4: Basic Usage - https://www.youtube.com/watch?v=armwW68kHHg[`https://www.youtube.com/watch?v=armwW68kHHg`]
** 5: Troubleshooting Installations - https://www.youtube.com/watch?v=5DOQm_1K4vU[`https://www.youtube.com/watch?v=5DOQm_1K4vU`]
** 6: Hierarchical Module Naming Schemes - https://www.youtube.com/watch?v=LNukZ8xceUc[`https://www.youtube.com/watch?v=LNukZ8xceUc`]
** 7: Easybuild at Julich Supercomputing Centre - https://www.youtube.com/watch?v=fOqZWRZM5PA[`https://www.youtube.com/watch?v=fOqZWRZM5PA`]
** 9: The EasyBuild Community - https://www.youtube.com/watch?v=l6_vUR56RJg[`https://www.youtube.com/watch?v=l6_vUR56RJg`]
** 8: EasyBuild at ComputeCanada - https://www.youtube.com/watch?v=sauTmalY9Hg[`https://www.youtube.com/watch?v=sauTmalY9Hg`]
** 10: Adding Support for Additional Software - https://www.youtube.com/watch?v=3D0JfbXBljY[`https://www.youtube.com/watch?v=3D0JfbXBljY`]
** 11: Contributing to EasyBuild - https://www.youtube.com/watch?v=kCQHPkEWNSw[`https://www.youtube.com/watch?v=kCQHPkEWNSw`]
** 12: Comparison with Other Tools - https://www.youtube.com/watch?v=hHqyc1_Wz3I[`https://www.youtube.com/watch?v=hHqyc1_Wz3I`]
** 13: Getting Help - https://www.youtube.com/watch?v=AQ3h0X8I4yk[`https://www.youtube.com/watch?v=AQ3h0X8I4yk`]
* 2021 LUMI User Tutorial
** 1: Introduction (2:29:36) - https://www.youtube.com/watch?v=JTRw8hqi6x0[`https://www.youtube.com/watch?v=JTRw8hqi6x0`]
** 2: Using EasyBuild (2:36:59) - https://www.youtube.com/watch?v=C3S8aCXrIMQ[`https://www.youtube.com/watch?v=C3S8aCXrIMQ`]
** 3: Advanced Topics (2:23:48) - https://www.youtube.com/watch?v=KbcvHa4uO1Y[`https://www.youtube.com/watch?v=KbcvHa4uO1Y`]
** 4: EasyBuild on Cray Systems (2:36:35) - https://www.youtube.com/watch?v=uRu7X_fJotA[`https://www.youtube.com/watch?v=uRu7X_fJotA`]
* 2016 Video Tutorial
** 1: Introduction (55:20) - https://www.youtube.com/results?search_query=easybuild+tutorial[`https://www.youtube.com/results?search_query=easybuild+tutorial`]
** 2: Getting Started (45:59) - https://www.youtube.com/watch?v=yo_KwlDtg0w&t=50s[`https://www.youtube.com/watch?v=yo_KwlDtg0w&t=50s`]
== A Hierarchy of Methods
There is a hierarchy of methods for attempting to compile and install a software package.
This hierarchy ranges from discovering an already installed package to bypassing EasyBuild
and building a package from scratch from the source code to finding and running a docker file
with Singularity. While EasyBuild also basically
builds a package from scratch, it is typically easier when one is starting out with EasyBuild
to build and install a package from scratch yourself until you're sufficiently familiar with
all the magical incantations needed to create an easyconfig file to do it for you.
=== Using an Already Installed Package
==== The Short Version
If a software package named `Armadillo` is requested, look for already installed versions via:
-----
module spider Armadillo
-----
Find out how to load a specific version - with the default being the most recent one - via:
-----
module spider Armadillo/10.7.5
-----
Load the software version and the listed dependencies via:
-----
module load GCC/10.3.0 OpenMPI/4.1.1 Armadillo/10.7.5
-----
If this doesn't work look for help in the following section.
==== The Long Version
One should first check to see if a requested software package is already locally installed.
This can be done via the command:
-----
module spider packagename
-----
A specific example is:
-----
module spider Armadillo/10.7.5
-----
which obtains:
-----
----------------------------------------------------------------------------------------------
Armadillo:
----------------------------------------------------------------------------------------------
Description:
Armadillo is an open-source C++ linear algebra library (matrix maths) aiming towards a good balance between speed and ease of use. Integer, floating point and
complex numbers are supported, as well as a subset of trigonometric and statistics functions.
Versions:
Armadillo/9.700.2
Armadillo/9.880.1
Armadillo/9.900.1
Armadillo/10.5.3
Armadillo/10.7.5
----------------------------------------------------------------------------------------------
For detailed information about a specific "Armadillo" package (including how to load the modules) use the module's full name.
Note that names that have a trailing (E) are extensions provided by other modules.
For example:
$ module spider Armadillo/10.7.5
----------------------------------------------------------------------------------------------
...
-----
There can be many available versions of a software package already installed.
If a specific version is not requested the default choice should be the most recent version.
The available versions can also be found by locating the installation directory and listing the versions contained therein.
-----
ls -l /sw/eb/sw/Armadillo
dr-xr-sr-x 6 baum staff 4096 Sep 6 2021 10.5.3-foss-2020b
dr-xr-sr-x 6 baum staff 4096 Dec 21 2021 10.7.5-foss-2021a
dr-xr-sr-x 6 j-perdue staff 4096 Jan 1 2021 9.700.2-foss-2019a
dr-xr-sr-x 6 j-perdue staff 4096 Dec 31 2020 9.880.1-foss-2020a
dr-xr-sr-x 6 j-perdue staff 4096 Dec 23 2020 9.900.1-foss-2019b
dr-xr-sr-x 6 j-perdue staff 4096 Dec 27 2020 9.900.1-foss-2020a
-----
A final check can be made by checking the easyconfig files available in the appropriate
cleaned directory.
The last two elements of the path shown below indicate the cluster on
which the installation is being performed - in this case `grace` - and
the name of the software being installation - in this case `Armadillo`.
There is a possibility that a version found in the installation
directory didn't complete the installation process, in which case it will not
be listed in the cleaned directory.
-----
ls -l /sw/eb/ebfiles_repo_cleaned/grace/Armadillo/
-rw-rw-r-- 1 francis staff 728 Sep 6 2021 Armadillo-10.5.3-foss-2020b.eb
-rw-rw-r-- 1 francis staff 721 Dec 21 2021 Armadillo-10.7.5-foss-2021a.eb
-rw-rw-r-- 1 francis staff 895 Jan 1 2021 Armadillo-9.700.2-foss-2019a.eb
-rw-rw-r-- 1 francis staff 895 Dec 31 2020 Armadillo-9.880.1-foss-2020a.eb
-rw-rw-r-- 1 francis staff 894 Dec 30 2020 Armadillo-9.900.1-foss-2019b.eb
-rw-rw-r-- 1 francis staff 894 Dec 30 2020 Armadillo-9.900.1-foss-2020a.eb
-----
After the appropriate software package and/or version has been found you can
find what other packages might also be required via:
-----
module spider Armadillo/10.7.5
...
You will need to load all module(s) on any one of the lines below before the "Armadillo/10.7.5" module is available to load.
GCC/10.3.0 OpenMPI/4.1.1
...
-----
at which point you can load your package and its dependencies via:
-----
module load GCC/10.3.0 OpenMPI/4.1.1 Armadillo/10.7.5
-----
and check to see what has been loaded via:
-----
module list
Currently Loaded Modules:
1) GCCcore/10.3.0 10) OpenSSL/1.1 19) ScaLAPACK/2.1.0-fb
2) zlib/1.2.11 11) libevent/2.1.12 20) bzip2/1.0.8
3) binutils/2.36.1 12) UCX/1.10.0 21) ICU/69.1
4) GCC/10.3.0 13) libfabric/1.12.1 22) Boost/1.76.0
5) numactl/2.0.14 14) PMIx/3.2.3 23) Eigen/3.3.9
6) XZ/5.2.5 15) OpenMPI/4.1.1 24) arpack-ng/3.8.0
7) libxml2/2.9.10 16) OpenBLAS/0.3.15 25) Armadillo/10.7.5
8) libpciaccess/0.16 17) FlexiBLAS/3.0.4
9) hwloc/2.4.1 18) FFTW/3.3.9
-----
A considerable number of packages can be loaded based on the dependencies.
If the package is not already installed we must move on to the next section wherein
we will attempt to install a package via its easyconfig file.
=== Installing from an Available Easyconfig File
==== Finding an Easyconfig File
A step up in complexity is installing a package from an available easyconfig file.
It involves searching the available easyconfig files that are archived at:
https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs[`https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs`]
to see if one is available for the desired software package.
The packages are kept in separate alphanumerical subdirectories on Github.
Click on the appropriate letter/number and scroll down or search the subdirectory for your software
package name. Note that packages whose names start with capital letters are listed before the
lower-case names.
If you can't find the package name by scrolling try searching for it since you can
never be sure what combination of upper- and lower-case letters are being used.
If you're uncertain about the name try using the Github search box on the upper left
to simultaneously search through all the subdirectories.
If you still can't find it then it's probably not there and you'll have to proceed to
further sections.
==== Compiling/Installing an Easyconfig File
=== Modifying an Existing Easyconfig File
=== Creating a New Easyconfig File Using an Existing File as a Template
Making small modifications to an existing easyconfig file works quite well when you are attempting
to install a new version of a software package that isn't yet available in the EB archives, or if
you want to update the compiler chain for an older version.
If there is not an existing easyconfig file for any version of a software package you wish to
install, you need to take the next step of using an existing easyconfig file that includes the
easyblock file you will need for your particular instance.
In the first example below, we're attempting to install a Python module for which there is no
official EB easyconfig file available. Our strategy is to find an existing easyconfig file that uses
the easyblock file we require - in this example the `PythonBundle` easyblock file - and extensively modify it
to meet our needs.
==== Installing Python Modules Via the PythonBundle Easyblock
===== Finding a Template
The PythonBundle easyblock is a starting point for installing Python modules that are not yet
available as easyconfig files. The easyconfig file for `statsmodels` will serve as a template.
-----
easyblock = 'PythonBundle'
name = 'statsmodels'
version = '0.12.1'
homepage = 'https://www.statsmodels.org/'
description = """Statsmodels is a Python module that allows users to explore data, estimate statistical models,
and perform statistical tests."""
toolchain = {'name': 'foss', 'version': '2020b'}
dependencies = [
('Python', '3.8.6'),
('SciPy-bundle', '2020.11'),
]
use_pip = True
sanity_pip_check = True
exts_list = [
('patsy', '0.5.1', {
'checksums': ['f115cec4201e1465cd58b9866b0b0e7b941caafec129869057405bfe5b5e3991'],
}),
(name, version, {
'checksums': ['a271b4ccec190148dccda25f0cbdcbf871f408fc1394a10a7dc1af4a62b91c8e'],
}),
]
moduleclass = 'math'
-----
===== Modifying the Template
A real-world example involves the installation of a Python module called *Colossus* that is a
toolkit for calculations pertaining to cosmology, the large-scale structure of the universe, and the properties of dark matter halos.
The installation page for *Colossus* at:
https://bdiemer.bitbucket.io/colossus/installation.html[`https://bdiemer.bitbucket.io/colossus/installation.html`]
tells us that it requires *Numpy*, *SciPy* and *Six*, and that the *Astropy* and *Camb* packages are optional.
The various parts of the *statsmodels* easyconfig file will now be modified to obtain an easyconfig
file for *Colossus*.
First, we change the `name` and `version` values appropriately. Version numbers are not given to this package so
the date of the most recent update is used.
We also change `homepage` and `description` to match the package.
-----
name = 'Colossus'
version = '2022-08-02'
homepage = 'https://bdiemer.bitbucket.io/colossus/index.html'
description = """Colossus is a python toolkit for calculations pertaining to cosmology,
the large-scale structure of the universe, and the properties of dark matter halos."""
-----
Next, we update the `toolchain` if a more recent one is available. In this case we've updated from `foss/2020b` to
`foss/2021b`. We may not always choose the most recent version of the toolchain, though, since the various
required modules may only be readily available as already-installed EB packages in an earlier version of the
toolchain.
-----
toolchain = {'name': 'foss', 'version': '2021b'}
-----
Keep `Python` and `SciPy-bundle` in the `dependencies` section. We upgrade the version of `Python`
from `3.8.6` to `3.9.6`, the version that corresponds to `foss/2021b`. We can find the compiler used in
`foss/2021b` by loading and then listing the loaded modules via:
-----
module purge
module load foss/2021b
module list
Currently Loaded Modules:
1) EasyBuild/4.6.0 12) OpenSSL/1.1
2) EasyBuild-faster/0 13) libevent/2.1.12
3) GCCcore/11.2.0 14) UCX/1.11.2
4) zlib/1.2.11 15) libfabric/1.13.2
5) binutils/2.37 16) PMIx/4.1.0
6) GCC/11.2.0 17) OpenMPI/4.1.1
7) numactl/2.0.14 18) OpenBLAS/0.3.18
8) XZ/5.2.5 19) FlexiBLAS/3.0.4
9) libxml2/2.9.10 20) FFTW/3.3.10
10) libpciaccess/0.16 21) ScaLAPACK/2.1.0-fb
11) hwloc/2.5.0 22) foss/2021b
-----
which shows us that `foss/2021b` corresponds to `GCC/11.2.0`. Now we need to find a Python
version
Similarly, we upgrade
`SciPy-bundle` from `2020.11` to `2021.10`.
The EB `Python` easyconfig file includes dozens of modules in additional to basic Python, and one of them is `six`
which satisfies that requirement. All the required modules are now includes.
* The optional module `astropy` is available as
-----
easyblock = 'PythonBundle'
name = 'Colossus'
version = '2022-08-02'
homepage = 'https://bdiemer.bitbucket.io/colossus/index.html'
description = """Colossus is a python toolkit for calculations pertaining to cosmology, the large-scale structure
of the universe, and the properties of dark matter halos."""
toolchain = {'name': 'foss', 'version': '2021b'}
dependencies = [
('Python', '3.8.6'),
('SciPy-bundle', '2020.11'),
]
use_pip = True
sanity_pip_check = True
exts_list = [
('patsy', '0.5.1', {
'checksums': ['f115cec4201e1465cd58b9866b0b0e7b941caafec129869057405bfe5b5e3991'],
}),
(name, version, {
'checksums': ['a271b4ccec190148dccda25f0cbdcbf871f408fc1394a10a7dc1af4a62b91c8e'],
}),
]
moduleclass = 'math'
-----
=== Compiling/Installing a Package in a Python/R Environment
==== Systemwide Installation
==== Userspace Installation
=== Compiling a Package from the Source Code With Make/CMake
==== Systemwide Installation
==== Userspace Installation
=== Finding and Running a Docker Image Using Singularity
== Using eb on the Command Line
The main tool to interact with the EasyBuild framework is the command-line program `eb`.
There are three main ways the `eb` program can be used to build and install modules.
=== General Methods for Building Modules
==== Via easyconfig Files
The most common and basic use of `eb` is to supply the name of an easyconfig file
containing parameters that specify how to build a module.
An example to build and install version 2.0 of `HPL` using version 1.4.10 of the `goolf` toolchain is:
-----
eb HPL-2.0-goolf-1.4.10.eb --robot
-----
All easyconfig filenames follow the format:
-----
<name>-<version>-<toolchain>-<versionsuffix>.eb
-----
Multiple easyconfig files can also be specified as arguments to `eb`:
-----
eb HPCG-2.1-goolf-1.4.10.eb GCC-4.8.3.eb
-----
An easyconfig file is a plain text file in Python syntax that mostly
contains easyconfig parameters defining `key-value` assignments.
Around 50 generic easyconfig parameters are supported, and can be separated into
a handful of mandatory parameters and much more than a handful of optional parameters.
Only a small subset of the optional parameters are commonly used, with the rest used to
solve vexing issues with packages that are built in uncommon ways.
Examples of ways to build packages are those using the `make` or `cmake` commands,
with more or less everything else being uncommmon.
A list of all of the available parameters can be found at:
https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html[`https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html`]
This list can also be obtained via `eb -a`.
Constructing easyconfig files is at best an easy matter of using or modifying an
already existing file, and at worst a neverending search through the official documentation,
tutorials, the archives of user groups, and the Github archive of thousands of files to find the Holy Grail
that will finally allow your file to successfully compile a module.
One good way to learn how to construct these files is via a set of example files, starting with the nearly
trivial and moving on to increasingly complex examples.
This is attempted in the <<Learning How to Build easyconfig Files Through Examples>> section below.
==== Via Command-Line Options
All of the parameters available to use within easyconfig files can also be specified on
the command line. A simple example of this is:
-----
eb --software-name=HPCG --toolchain=goolf,1.4.10
-----
==== Via Github Pull Requests
An easyconfig file that is added or modified by a Github pull request to the
EasyBuild `easybuild-easyconfigs` repository can be used via:
-----
eb --from-pr 1177
-----
which would use the GCC 4.9.2 easyconfig file contributed via pull request number 1177.
The `easybuild-easyconfigs` repository is located at:
https://github.com/easybuilders/easybuild-easyconfigs[`https://github.com/easybuilders/easybuild-easyconfigs`]
and it would be useful to become familiar with the workings of Github for this and
many other reasons.
=== Specifying Locations of Easyconfig Files
==== Present Working Directory
If `eb` is run with the name of an easyconfig file as the only argument in a directory
that contains that file, it will use that file for processing. Say we have
the easyconfig file `M4-1.4.19.eb` in the directory `/home/baum/eb/TEST` and while in
that directory we run:
-----
cd /home/baum/eb/TEST
eb M4-1.4.19.eb
== Temporary log file in case of crash /tmp/baum/logs/easybuild-MvzIxw.log
== resolving dependencies ...
== processing EasyBuild easyconfig /home/baum/eb/TEST/M4-1.4.19.eb
...
-----
and see that the `M4-1.4.19.eb` in the directory is processed.
==== EasyBuild Standard Location(s)
If the `M4-1.4.19.eb` was not in that directory we would see instead:
-----
cd /home/baum/eb
eb M4-1.4.19.eb
== Temporary log file in case of crash /tmp/baum/logs/easybuild-_fyV_p.log
== resolving dependencies ...
== processing EasyBuild easyconfig /sw/eb/ebfiles_repo_cleaned/grace/M4/M4-1.4.19.eb
...
-----
where the easyconfig file in a standard location in the EasyBuild installation
is used. This standard location is specified via the environment
variable `EASYBUILD_ROBOT`, which can be found via:
-----
env | grep EASYBUILD_ROBOT
EASYBUILD_ROBOT=/sw/eb/ebfiles_repo_cleaned/grace
-----
==== Arbitrary Directories
You can specify any directory within which `eb` will look for
easyconfig files via the `-r PATH` or `--robot=PATH` option. For example,
if you are not in the `/home/baum/eb/TEST` directory and want to install
a hypothetical easyconfig file `M4-1.4.19.eb` therein you can enter:
-----
cd /home/baum/eb
eb -r /home/baum/eb/TEST M4-1.4.19.eb
-----
and the `M4-1.4.19.eb` file will be fetched and used from that directory.
Also, if the environment variable `EASYBUILD_ROBOT` were not already set as in
the previous example, this would be used to appraise `eb` of its location via:
-----
eb -r /sw/eb/ebfiles_repo_cleaned/grace M4-1.4.19.eb
-----
=== Options for Testing Before Installing
==== Dry Runs
To print a build overview for a dry run including dependencies use the argument `-D`.
This yields little information for the `M4-1.4.19` module since it has no dependencies
other than the `SYSTEM` toolchain.
-----
eb -D M4-1.4.19.eb
== Temporary log file in case of crash /tmp/baum/logs/easybuild-xhlhlg.log
Dry run: printing build status of easyconfigs and dependencies
CFGS=/home/baum/eb/TEST
* [ ] $CFGS/M4-1.4.19.eb (module: Core | M4/1.4.19)
== Temporary log file(s) /tmp/baum/logs/easybuild-xhlhlg.log* have been removed.
== Temporary directory /tmp/baum/tmp/eb-1jEupK has been removed.
-----
A module like `GCC-11.2.0` with many dependencies will yield significantly
more information. Building `GCC-11.2.0` involves a bootstrapping process wherein
dependencies like `M4`, `zlib`, `binutils`, etc. are first built using the
`SYSTEM` toolchain to enable `GCCcore-11.2.0` to be install, and then built again
using the newly installed `GCCcore-11.2.0`
toolchain.
-----
eb -D GCC-11.2.0.eb
== Temporary log file in case of crash /tmp/baum/logs/easybuild-NCvQnY.log
Dry run: printing build status of easyconfigs and dependencies
* [ ] /sw/eb/ebfiles_repo_cleaned/grace/M4/M4-1.4.19.eb (module: Core | M4/1.4.19)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/zlib/zlib-1.2.11.eb (module: Core | zlib/1.2.11)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/help2man/help2man-1.47.4.eb (module: Core | help2man/1.47.4)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/M4/M4-1.4.18.eb (module: Core | M4/1.4.18)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/Bison/Bison-3.7.6.eb (module: Core | Bison/3.7.6)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/Bison/Bison-3.3.2.eb (module: Core | Bison/3.3.2)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/flex/flex-2.6.4.eb (module: Core | flex/2.6.4)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/binutils/binutils-2.37.eb (module: Core | binutils/2.37)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/GCCcore/GCCcore-11.2.0.eb (module: Core | GCCcore/11.2.0)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/zlib/zlib-1.2.11-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | zlib/1.2.11)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/help2man/help2man-1.48.3-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | help2man/1.48.3)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/M4/M4-1.4.19-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | M4/1.4.19)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/Bison/Bison-3.7.6-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | Bison/3.7.6)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/flex/flex-2.6.4-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | flex/2.6.4)
* [x] /sw/eb/ebfiles_repo_cleaned/grace/binutils/binutils-2.37-GCCcore-11.2.0.eb (module: Compiler/GCCcore/11.2.0 | binutils/2.37)
* [x] /home/baum/eb/TEST/GCC-11.2.0.eb (module: Core | GCC/11.2.0)
== Temporary log file(s) /tmp/baum/logs/easybuild-NCvQnY.log* have been removed.
== Temporary directory /tmp/baum/tmp/eb-vm0vO9 has been removed.
-----
==== Testing a Build Without Installing
To actually go through all the steps in the process up to the installation step
we use the `-s` argument which can stop the process after any of the steps, with
the available steps being `fetch`, `ready`, `source`, `patch`, `prepare`, `configure`,
`build`, `test`, `install`, `extensions`, `postiter`, `postproc`, `sanitycheck`,
`cleanup`, `module`, `permissions`, `package` and `testcases`.
We now do this with `M4-1.4.19` and stop before installation using `-s test` since
`test` is the step before `install`.
Note that if the module you are testing is already installed you must also
insert the argument `-f` to force it to continue even after detecting that it is
already installed.
-----
eb [-f] -s test M4-1.4.19.eb
== Temporary log file in case of crash /tmp/baum/logs/easybuild-g9Z6Lm.log
== resolving dependencies ...
== processing EasyBuild easyconfig /sw/eb/ebfiles_repo_cleaned/grace/M4/M4-1.4.19.eb
== building and installing Core/M4/1.4.19...
== fetching files...
== creating build dir, resetting environment...
== ... (took 2 secs)
== unpacking...
== patching...
== preparing...
== configuring...
== ... (took 38 secs)
== building...
== ... (took 2 secs)
== testing...
== COMPLETED: Installation STOPPED successfully (took 44 secs)
== Results of the build can be found in the log file(s) /tmp/baum/easybuild/M4/1.4.19/system-system/easybuild/easybuild-M4-1.4.19-20210915.141948.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /tmp/baum/logs/easybuild-g9Z6Lm.log* have been removed.
== Temporary directory /tmp/baum/tmp/eb-FCeCpj has been removed.
-----
=== Options for Obtaining Information
The `eb` command is also used to obtain information about available options and easyconfig and easyblock files.
Although not all of these topics have been covered, it is useful to have informational options
with examples grouped together in a single location.
The most basic information
option is:
-----
eb [-h|--help]
-----
which prints a list of all available options, informative and otherwise.
==== Show List of Parameters
A list of available easyconfig parameters is obtainable via:
-----
eb [-a|--available-easyconfig-params]
-----
which can be found online at:
https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html[`https://docs.easybuild.io/en/latest/version-specific/easyconfig_parameters.html`]
A list of parameters specific to a given easyblock, say `EB_WRF` can be found via:
-----
eb -a -e EB_WRF | grep "* "
-----
which prints:
-----
buildtype* Specify the type of build (serial, smpar (OpenMP), dmpar (MPI), dm+sm (hybrid OpenMP/MPI)). [default: None]
rewriteopts* Replace -O3 with CFLAGS/FFLAGS [default: True]
runtest* Build and run WRF tests [default: True]
-----
If you do not use `grep` to isolate the asterisk-marked easyblock specific parameters you will get a list of
all parameters within which the specific ones are buried.
==== Show List of Toolchains
A list of available toolchains can be obtained via:
-----
eb --list-toolchains
-----
This list is also available in the officials docs at:
https://docs.easybuild.io/en/latest/version-specific/toolchains.html[`https://docs.easybuild.io/en/latest/version-specific/toolchains.html`]
==== Show Toolchain Options
The options available for any of the toolchains listed as available by `eb --list-toolchains` can
be obtained via:
-----
eb --avail-toolchain-opts GCC
-----
which will provide:
-----
Available options for GCC toolchain:
32bit: Compile 32bit target (default: False)
cciscxx: Use CC as CXX (default: False)
cstd: Specify C standard (default: None)
...
vectorize: Enable compiler auto-vectorization, default except for noopt and lowopt (default: None)
verbose: Verbose output (default: False)
veryloose: Very loose precision (default: False)
-----
==== Show List of Easyblocks
A list of available easyblocks is obtained via:
-----
eb --list-easyblocks
-----
with that list also available in the official docs at:
https://docs.easybuild.io/en/latest/eb_list_easyblocks.html[`https://docs.easybuild.io/en/latest/eb_list_easyblocks.html`]
==== Search for an easyconfig File
If you're looking to see if EasyBuild has an easyconfig file for a software package of interest, you
can do so via:
-----
eb [-S|--search]
-----
For example, a search for `ArviZ` - a package for the exploratory analysis of Bayesian models
with Python - via:
-----
eb -S ArviZ
-----
will yield something like:
-----
* $CFGS1/ArviZ-0.7.0-foss-2019b-Python-3.7.4.eb
* $CFGS2/ArviZ-0.7.0-intel-2019b-Python-3.7.4.eb
* $CFGS2/ArviZ-0.11.1-intel-2020b.eb
-----
Once you have obtained this information, you can discover how to load one of these modules via:
-----
module spider ArviZ/0.7.0
-----
which will tell you something like:
-----
ArviZ: ArviZ/0.7.0-Python-3.7.4
Description:
Exploratory analysis of Bayesian models with Python
You will need to load all module(s) on any one of the lines below before the "ArviZ/0.7.0-Python-3.7.4" module is available to load.
GCC/8.3.0 OpenMPI/3.1.4
iccifort/2019.5.281 impi/2018.5.288
-----
You can also use regular expressions to narrow your search. For example,
to search for which easyconfig files are available for `GCC v4.6.x` without listing easyconfig files
that use `GCC v4.6.x` as a toolchain you would enter:
-----
eb -S ^GCC-4.6
-----
and to find all easyconfig files for Python versions 2.7.8 and 2.7.9 that use the intel toolchain:
-----
eb -S '^Python-2.7.[89].*intel'
-----
==== List Available easyconfig Constants
A list of constants that can be used in easyconfig files is found via:
-----
eb --avail-easyconfig-constants
-----
This presently (7/21) yields:
-----
ARCH: x86_64 (CPU architecture of current system (aarch64, x86_64, ppc64le, ...))
EXTERNAL_MODULE: EXTERNAL_MODULE (External module marker)
HOME: /home/baum (Home directory ($HOME))
OS_NAME: centos linux (System name (e.g. 'fedora' or 'RHEL'))
OS_PKG_IBVERBS_DEV: ('libibverbs-dev', 'libibverbs-devel', 'rdma-core-devel') (OS packages providing ibverbs/infiniband development support)
OS_PKG_OPENSSL_BIN: openssl (OS packages providing the openSSL binary)
OS_PKG_OPENSSL_DEV: ('openssl-devel', 'libssl-dev', 'libopenssl-devel') (OS packages providing openSSL developement support)
OS_PKG_OPENSSL_LIB: ('libssl', 'libopenssl') (OS packages providing openSSL libraries)
OS_PKG_PAM_DEV: ('pam-devel', 'libpam0g-dev') (OS packages providing Pluggable Authentication Module (PAM) developement support)
OS_TYPE: Linux (System type (e.g. 'Linux' or 'Darwin'))
OS_VERSION: 7.9.2009 (System version)
SYSTEM: {'version': 'system', 'name': 'system'} (System toolchain)
SYS_PYTHON_VERSION: 2.7.5 (System Python version (platform.python_version()))
-----
==== List Available easyconfig Templates
Getting a list of all template names and constants that can be usd in easyconfig files is done via:
-----
eb --available-easyconfig-templates
-----
which will obtain something like:
-----
Template names/values derived from easyconfig instance
%(module_name)s: Module name
%(nameletter)s: First letter of software name
...
SOURCELOWER_PY2_WHL: Generic (non-compiled) Python 2 wheel package with lowercase name (%(namelower)s-%(version)s-py2-none-any.whl)
SOURCE_PY3_WHL: Generic (non-compiled) Python 3 wheel package (%(name)s-%(version)s-py3-none-any.whl)
SOURCELOWER_PY3_WHL: Generic (non-compiled) Python 3 wheel package with lowercase name (%(namelower)s-%(version)s-py3-none-any.whl)
-----
==== Show the Current EasyBuild Configuration
The current EasyBuild environment configuration can be found via:
-----
eb --show-config
-----
On a local cluster named `grace`, we load a configuration via:
-----
module load EasyBuild-grace
-----
which running `eb --show-config` shows contains the following configuration parameters:
-----
#
# Current EasyBuild configuration
# (C: command line argument, D: default value, E: environment variable, F: configuration file)
#
buildpath (E) = /tmp/baum/easybuild/
containerpath (E) = /sw/eb/containers
deprecated (E) = 2.0
installpath (E) = /sw/eb
installpath-modules (E) = /sw/eb/mods
installpath-software (E) = /sw/eb/sw
minimal-toolchains (E) = True
module-naming-scheme (E) = HierarchicalMNS
packagepath (E) = /sw/eb/packages
prefix (E) = /sw/eb
read-only-installdir (E) = True
repositorypath (E) = /sw/eb/ebfiles_repo/grace
robot (E) = /sw/eb/ebfiles_repo_cleaned/grace, /sw/eb/sw/EasyBuild/4.4.0/easybuild/easyconfigs
robot-paths (E) = /sw/eb/ebfiles_repo_cleaned/grace, /sw/eb/sw/EasyBuild/4.4.0/easybuild/easyconfigs
sourcepath (E) = /sw/eb/sources
tmp-logdir (E) = /tmp/baum/logs/
tmpdir (E) = /tmp/baum/tmp/
-----
==== Show Default Module Classes
An optional parameter in easyconfig files is `moduleclass`, which specifies the
category to which the software belongs. The default is the `base` class, although it is
recommended to replace that with a more appropriate choice from the list of
available options. EasyBuild enforces that only known module classes can
be specified, so providing one not on the list will throw an error.
A list of the available classes can be obtained via:
-----
eb --show-default-moduleclasses
-----
-----
Default available module classes:
base: Default module class
astro: Astronomy, Astrophysics and Cosmology
...
tools: General purpose tools
vis: Visualization, plotting, documentation and typesetting
-----
[[building_easycofig_files]]
== Learning How to Build easyconfig Files Through Examples
EasyBuild builds software packages based on the specifications provided in
what are called `easyconfig` files. The official documentation for `easyconfig` files can
be found at:
* Writing `easyconfig` Files
https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html[`https://docs.easybuild.io/en/latest/Writing_easyconfig_files.html`]
* Implementing `easyconfig` Files
https://docs.easybuild.io/en/latest/Implementing-easyblocks.html[`https://docs.easybuild.io/en/latest/Implementing-easyblocks.html`]
An easyconfig file provides a build specification for EasyBuild.
It is a plaintext file in Python syntax that is mostly key-value assignment
statements to define easyconfig parameters.
=== Example 1 - Inspecting the Parts of an Existing File
The following is a relatively simple example `easyconfig` file for a specific version of `bzip2`
that can be found on `grace` at:
-----
/sw/eb/sw/bzip2/1.0.8-GCCcore-10.2.0/easybuild/reprod/bzip2-1.0.8-GCCcore-10.2.0.eb
-----
It consists of mandatory parameters plus other common optional parameters needed
to sufficiently supply a build specification.
This example contains comments not found in the actual file to help with
understanding what is going on.
Most of the comments suffice at this point to explain the contents of the file,
although there will be additional sections following to go into a little more
depth on the source code and dependency sections.
.An Example `easyconfig` file for `bzip2`
[source,python]
-----
# Mandatory: The name of the software package to build.
name = 'bzip2'
# Mandatory: The version of the software package to build.
version = '1.0.8'
# Mandatory: A URL of a homepage for the software.
homepage = 'https://sourceware.org/bzip2'
# Mandatory: A brief description of the software package.
description = """
bzip2 is a freely available, patent free, high-quality data compressor. It
typically compresses files to within 10% to 15% of the best available
techniques (the PPM family of statistical compressors), whilst being around
twice as fast at compression and six times faster at decompression.
"""
# Mandatory: The toolchain to be used to compile the software package.
toolchain = {'name': 'GCCcore', 'version': '10.2.0'}
# Optional: A list of extra options for the compiler.
toolchainopts = {'pic': True}
# Optional: A list of URLs from where the source code file(s) can be obtained.
source_urls = ['https://sourceware.org/pub/%(name)s/']
# Optional: A list of source file names.
sources = [SOURCE_TAR_GZ]
# Optional: A list of patches to apply.
patches = ['%(name)s-%(version_major_minor)s.6-pkgconfig.patch']
# Optional: A list of checksums for the software and patches.
checksums = [
'ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269', # %(name)s-%(version)s.tar.gz
'5a823e820b332eca3684416894f58edc125ac3dace9f46e62f98e45362aa8a6d', # %(name)s-%(version_major_minor)s.6-pkgconfig.patch
]