LinearAlgebra-Awareness-Benchmark

Prerequisites

TensorFlow and PyTorch with MKL support

TensorFlow: Installation instructions

PyTorch: Installation instructions

Running the Experiments

More details in the respective experiment folders

Performance on Intel Xeon Platinum 8160 CPU:

Number of Threads: 1
Reporting statistic: Minimum of 20 repetitions.
TensorFlow Version: 2.7.0
PyTorch Version: 1.10.0
MKL: Intel OneAPI 2022.0.0

Performance comparision: Eager mode vs Graph mode

Operands

A and B are general square matrices of size 3000

File	Expression	C	TF (Eager)	PyT (Eager)	TF (Graph)	PyT (Graph)
sgemm	ATB	0.39	0.40	0.40	0.40	0.40
cse	(ATB)T(ATB)	-	1.25	1.27	0.78	0.80

Experiment 1 : Common Subexpression Elimination

Operands

A and B are general square matrices of size 3000

File	Expression	TF	PyT
sgemm	ATB	0.40	0.40
cse_add	ATB + ATB	0.40	0.41
cse_mul_parenthesis	(ATB)T(ATB)	0.78	0.80
cse_mul_no_parenthesis	(ATB)TATB	1.17	1.15

Experiment 2: Optimization of Matrix Chains

Operands

H is a square matrix of size 3000

x,y are vectors of size 3000

Right to Left Parenthesization

File	Expression	TF (matmul)	PyT (matmul)	PyT (multi_dot)
no_parenthesis	HTHx	0.40	0.41	0.006
rtol_parenthesis	HT(Hx)	0.006	0.004	-

Left to Right Parenthesization

File	Expression	TF (matmul)	PyT (matmul)	PyT (multi_dot)
no_parenthesis	yTHTH	0.006	0.005	0.005
ltor_parenthesis	(yTH)TH	0.006	0.005	-

Mixed Parenthesization

File	Expression	TF (matmul)	PyT (matmul)	PyT (multi_dot)
no_parenthesis	HTyxTH	0.41	0.40	0.01
mixed_parenthesis	(HTy)(xTH)	0.01	0.01	-

Experiment 3: Matrix Properties

Matrix multiplication AB with matrices having special properties

A : General Matrix

L : Lower Triangular Matrix

T : Tridiagonal Matrix

D : Diagonal Matrix

Operands : A,L,T,D are square matrices of size 3000.

File	Expression	SciPy (blas)	TF (matmul)	TF (optimized)	PyT (matmul)	PyT (optimized)
sgemm	AB	0.40	0.40	-	0.41	-
trmm	LB	0.24	0.40	n.a	0.40	n.a
syrk	AAT	0.24	0.41	n.a	0.39	n.a
tridiagmm	TB	0.20	0.41	0.02	0.40	n.a
diagmm	DB	0.12	0.39	0.018	0.40	n.a

Experiment 4: Algebraic Manipulations

Operands:

Distributivity: A and B are General square matrices of size 3000.

Blocked matrices: A_B = [[A₁, 0], [0, A₁]], B_B = [[B₁, 0], [0, B₁]]

File	Expression	TF (LHS / RHS)	PyT (LHS / RHS)
distributivity_eq9	AB + AC = A(B+C)	0.78 / 0.40	0.81 / 0.41
distributivity_eq10	Ax - HT(Hx) = (A - HTH)x	0.01 / 0.42	0.01 / 0.41
blocked_matrices	ABBB = [(A1B1),(A1B1)]T	0.40 / 0.20	0.40 / 0.20

Experiment 5: Code Motion

Operands:

A and B are general square marices of size 3000

V is a 3x3000

File	Expression	TF (naive / recommended)	PyT (naive / recommended)
loop_inv	for i in range(3): AB + V[i]V[i]T	0.42 / 0.42	0.42 / 0.41
partial_op_sum	(A+B)[2,2]	0.011 / 6e-4	0.018 / 2e-3
partial_op_prod	(AB)[2,2]	0.39 / 2e-3	0.40 / 3e-3