10-pytorch-01.qmd

---
title: PyTorch 01 - Tensors
jupyter: python3
---

## ![](assets/images/pytorch/pytorch.svg){height="48px"} PyTorch Introduction

[![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/trgardos/ml-549-fa24/blob/main/10-pytorch-01.ipynb)

[PyTorch](https://pytorch.org/) is an open-source machine learning library developed by Facebook's AI Research lab. 

It is widely used for deep learning tasks, particularly in the field of natural language processing (NLP) and computer vision. 

Started at Facebook AI Research (FAIR), now _Fundamntal AI Research_, to build a consolidated tool for deep learning tasks.

* 2015 there was Theano (Montreal) Caffe (Berkeley) and Lua Torch (FAIR).
* 2015 December, Google released TensorFlow
* ~2016 Released Caffe2 for targeting mobile and edge devices (Facebook Production)
* 2015-2016 FAIR refactored Torch to separate computation backend from frontend and create a new frontend in Pythyon
    * the core library is still called `torch`
* Sep 2016 PyTorch v0.1.1 released (`torch.nn` and `torch.autograd` modules)
* Dec 2018 PyTorch v1.0 released (Replace Caffe2 for FB production)
* Mar 15 2023 PyTorch v2.0 released (dynamic shapes, distributed training, Transformers support, etc.)
* (Latest Release)Aug 2024 PyTorch 2.4.1 

## Key Features of PyTorch

1. **Dynamic Computation Graph**: PyTorch's dynamic computation graph allows for more intuitive and flexible model 
   building. This means that the graph is constructed at runtime, allowing for more dynamic and interactive model 
   development.

2. **GPU and TPU Acceleration**: PyTorch can leverage GPUs and TPUs for accelerated training, making it faster than many 
   other deep learning frameworks.

3. **Autograd**: PyTorch's autograd module provides automatic differentiation, which allows for easy computation of 
   gradients and updates to model parameters during training.

4. **High-Level API**: PyTorch has a high-level API that allows for easy model building and training. It also has a low-level 
   API that provides more control over the model building process.

## PyTorch vs TensorFlow


::: {layout-ncol="2"}

![](assets/images/pytorch/num_hf_models_2023.png)

![](assets/images/pytorch/Fraction-of-Papers-Using-PyTorch-vs.-TensorFlow.png)

![](assets/images/pytorch/percentage_repo_2023.png)

:::

From [https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2023/](https://www.assemblyai.com/blog/pytorch-vs-tensorflow-in-2023/).

## Key Differences

:::: {.columns}
::: {.column width="50%"}

### PyTorch

* Dynamic Computation Graph
* Autograd
* High-Level API
* Arguably more "pythonic"

:::
::: {.column width="50%"}

### TensorFlow

* Static Computation Graph
* Eager Execution
* Low-Level API (high-level API is Keras)

:::
::::

## Tutorial

We'll borrow heavily from [https://pytorch.org/tutorials/](https://pytorch.org/tutorials/)
and other sources that we'll cite.

## Tensors

Fundamentally, PyTorch [tensor](https://pytorch.org/docs/stable/generated/torch.tensor.html) is a data structure for
storing matrices and multi-dimensional arrays.

Similar to NumPy's ndarrays.

But it does much more.

* Manages translation to accelerator data formats and hardware memory
* Stores information needed for _automatic gradient_ (autograd) calculation for parameter updates
* etc...

``` {python}
import torch
import numpy as np
```

### Tensor Creation

We have to get data into PyTorch tensors before we can operate on them.

There are multiple ways to do that.

**Create directly from a Python list such as this one:**

``` {python}
data = [[1, 2], [2, 3]]
print(f"data: {data}")
print(f"type(data): {type(data)}")
```


**Create the tensor and explore some of its attributes.**

``` {python}
x_data = torch.tensor(data)
print(f"x_data: {x_data}")
print(f"type(x_data): {type(x_data)}")
print(f"x_data.dtype: {x_data.dtype}")
print(f"x_data.shape: {x_data.shape}")
print(f"x_data.device: {x_data.device}")
print(f"x_data.requires_grad: {x_data.requires_grad}")
print(f"x_data.is_leaf: {x_data.is_leaf}")
```

As mentioned, some of the interesting attributes are:

* `x_data.device`: The device where the tensor is stored.
* `x_data.requires_grad`: Whether the tensor requires gradient computation.
* `x_data.is_leaf`: Whether the tensor is a leaf node in the computation graph.

We'll get more into these later.

If we use decimal numbers, then a floating point number type is chosen.

``` {python}
y_data = torch.tensor([[0.1, 1.2], [2.2, 3.1]])
print(f"y_data: {y_data}")
print(f"y_data.dtype: {y_data.dtype}")
print(f"y_data.device: {y_data.device}")
```

**From a NumPy array**

``` {python}
np_array = np.array(data)
x_np = torch.from_numpy(np_array)
```

**As the same shape of another tensor**

``` {python}
x_ones = torch.ones_like(x_data) # retains the properties of x_data
print(f"Ones Tensor: \n {x_ones} \n")

x_rand = torch.rand_like(x_data, dtype=torch.float) # overrides the datatype of x_data
print(f"Random Tensor: \n {x_rand} \n")
```


**With random or constant values**

``` {python}
shape = (2,3)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")

```

::: {.callout-tip}
There are many more creation functions listed in [Creation Ops](https://pytorch.org/docs/stable/torch.html#creation-ops)
for things like sparse tensors, other random number generators, etc.
:::

### Operations on Tensors

Let's look at some interesting operations we can perform on tensors.

#### Standard numpy-like indexing and slicing

Python itself has some flexible indexing and slicing support. See this
[tutorial](https://railsware.com/blog/indexing-and-slicing-for-lists-tuples-strings-sequential-types/)
for a nice overview.

PyTorch follows the NumPy [indexing and slicing](https://numpy.org/doc/stable/user/basics.indexing.html)
conventions.

```{python}
# Define a 4x4 tensor
tensor = torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=torch.float32)
#tensor = torch.ones(4, 4)
print(f"tensor.dtype: {tensor.dtype}")

# Matrices are stored in row-major order, and like NumPy, it is a list of lists.
print(f"First row: {tensor[0]}")
print(f"Second row: {tensor[1]}")

# Like python we use `:` to index and slice
print(f"First column: {tensor[:, 0]}")
print(f"Second column: {tensor[:, 1]}")

# Slicing with `...` is a shortcut for "all remaining dimensions"
print(f"Last column: {tensor[..., -1]}")
print(f"Last column: {tensor[:, -1]}")

# There is also broadcasting support
tensor[:,1] = 0
print(tensor)
```

#### Joining tensors

You can use `torch.cat` to concatenate a sequence of
tensors along a given dimension. See also
[torch.stack](https://pytorch.org/docs/stable/generated/torch.stack.html),
another tensor joining operator that is subtly different from
`torch.cat`.

```{python}
t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)
```


```{python}
t2 = torch.cat([tensor, tensor, tensor], dim=0)
print(t2)
```

### Matrix Multiplication

Matrix multiplication is a fundamental operation in deep learning and PyTorch provides a number of ways to perform it.

#### Equivalent matrix multiplication operations

We'll use the following helper function to format the tensors as LaTeX matrices.

```{python}
#| code-fold: true
import torch
from IPython.display import display, Math

def tensor_to_latex(tensor):
    """Convert a PyTorch tensor to a LaTeX bmatrix string."""
    if tensor.dim() == 1:
        # Convert 1D tensor to a 2D column vector for display
        tensor = tensor.unsqueeze(1)
    
    latex_str = "\\begin{bmatrix}\n"
    for row in tensor:
        latex_str += " & ".join([f"{val.item():.2f}" for val in row]) + " \\\\\n"
    latex_str += "\\end{bmatrix}"
    return latex_str
```

```{python}
# Define matrix A and vector x
A = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
x = torch.tensor([5.0, 6.0])

# Compute the product A * x
Ax = torch.matmul(A, x)

# Convert tensors to LaTeX
latex_A = tensor_to_latex(A)
latex_x = tensor_to_latex(x)
latex_Ax = tensor_to_latex(Ax)

# Display the equation A x = Ax on one line
display(Math(f"A \\times x = {latex_A} \\times {latex_x} = {latex_Ax}"))
```

We'll use the tensor from before.

```{python}
#| code-fold: true
latex_tensor = tensor_to_latex(tensor)
display(Math(f"\\mathrm{{tensor}} = {latex_tensor}"))
```

There are three different ways shown below that produce the same result.

##### **Method 1:** Use the `@` matrix multiplication operator

```{python}
# ``tensor.T`` returns the transpose of a tensor
y1 = tensor @ tensor.T
# print(f"y1: {y1}")

latex_tensorT = tensor_to_latex(tensor.T)
latex_y1 = tensor_to_latex(y1)
display(Math(f"y1 = \\mathrm{{tensor}} \\times \\mathrm{{tensor.T}} = {latex_tensor} \\times {latex_tensorT} = {latex_y1}"))
```

<br>

##### **Method 2:** call `.matmul` method on the tensor

```{python}
y2 = tensor.matmul(tensor.T)
# print(f"y2: {y2}")

latex_y2 = tensor_to_latex(y2)
display(Math(f"y2 = \\mathrm{{tensor}} \\times \\mathrm{{tensor.T}} = {latex_tensor} \\times {latex_tensorT} = {latex_y2}"))
```

<br>

##### **Method 3:** Call the `torch.matmul` function with the two input operands and the output tensor.

```{python}
# This creates a new tensor with the same shape and type as y1, filled with random numbers from a uniform distribution over [0, 1)
y3 = torch.rand_like(y1)
#print(f"y3: {y3}")

torch.matmul(tensor, tensor.T, out=y3)
# print(f"y3: {y3}")

latex_y3 = tensor_to_latex(y3)
display(Math(f"y3 = \\mathrm{{torch.matmul(tensor, tensor.T)}} = {latex_tensor} \\times {latex_tensorT} = {latex_y3}"))
```

<br>

There is additional behavior depending on the dimensionality of the tensors.

#### Vector Dot Product

If both tensors are 1-D, then the **vector dot product** is performed.

```{python}
torch.manual_seed(0)

# vector x vector
tensor1 = torch.randint(low=0, high=10, size=(3,))
tensor2 = torch.randint(low=0, high=10, size=(3,))
# print(f"tensor1: {tensor1} and tensor2: {tensor2}")

latex_tensor1 = tensor_to_latex(tensor1)
latex_tensor2 = tensor_to_latex(tensor2)

display(Math(f"\\mathrm{{tensor1}} = {latex_tensor1}, \\mathrm{{tensor2}} = {latex_tensor2}"))


tensor3 = torch.matmul(tensor1, tensor2)
print(f"tensor3 = tensor1 @ tensor2: {tensor3}")
print(f"tensor3.size(): {tensor3.size()}")

```

So with 1-D tensors, `torch.matmul` and `torch.dot` are equivalent.

```{python}
tensor3 = torch.dot(tensor1, tensor2)
print(f"tensor3 = torch.dot(tensor1, tensor2): {tensor3}")
print(f"tensor3.size(): {tensor3.size()}")
```

#### Matrix-Vector Multiplication

Now let's look at the case where one operand is 1D and the other is 2D.

If the first argument is 2-dimensional and the second argument is 1-dimensional,
the matrix-vector product is returned.


```{python}
tensor1 = torch.tensor([[1, 2], [3, 4]])
print(f"Operand 1 --tensor1.size(): {tensor1.size()}")
# print(f"Operand 1 -- tensor1: {tensor1}")
latex_tensor1 = tensor_to_latex(tensor1)

tensor2 = torch.tensor([1, 2])
print(f"Operand 2 --tensor2.size(): {tensor2.size()}")
#print(f"Operand 2 --tensor2: {tensor2}")

latex_tensor2 = tensor_to_latex(tensor2)

tensor3 = torch.matmul(tensor1, tensor2)
#print(f"Output -- tensor3: {tensor3}")
print(f"Output -- tensor3.size(): {tensor3.size()}")

latex_tensor3 = tensor_to_latex(tensor3)
display(Math(f"\\mathrm{{tensor3}} = \\mathrm{{tensor1}} \\times \\mathrm{{tensor2}} = {latex_tensor1} \\times {latex_tensor2} = {latex_tensor3}"))
```

So this would be the similar to

```{python}
# Change tensor2 to be shape (2,1)
tensor2 = torch.tensor([[1], [2]])
print(f"Operand 2 -- tensor2.size(): {tensor2.size()}")
print(f"Operand 2 -- tensor2: {tensor2}")

tensor3 = torch.matmul(tensor1, tensor2)
print(f"Output -- tensor3: {tensor3}")
print(f"Output -- tensor3.size(): {tensor3.size()}")
```

But note the difference in the size of the output.

If the first argument is 1-dimensional and the second argument is 2-dimensional,
a 1 is prepended to its dimension for the purpose of the matrix multiply.
After the matrix multiply, the prepended dimension is removed.

``` {python}
tensor2 = torch.tensor([1, 2])
print(f"Operand 1 -- tensor2.size(): {tensor2.size()}")
print(f"Operand 1 -- tensor2: {tensor2}")

tensor3 = torch.matmul(tensor2, tensor1)
print(f"Output -- tensor3: {tensor3}")
print(f"Output -- tensor3.size(): {tensor3.size()}")
```

which is similar to

``` {python}
tensor2 = torch.tensor([[1, 2]])
print(f"Operand 1 -- tensor2.size(): {tensor2.size()}")
print(f"Operand 1 -- tensor2: {tensor2}")

tensor3 = torch.matmul(tensor2, tensor1)
print(f"Output -- tensor3: {tensor3}")
print(f"Output -- tensor3.size(): {tensor3.size()}")
```


#### Batched Matrix Multiplication

It is very common to have a "batch" of inputs to a neural network, where maybe
the input to the network is a 3x4 matrix, but PyTorch optimizes the computation
to process an entire batch of matrices at once.

![Batches of size 5](assets/images/pytorch/batches.png){fig-align="center"}

PyTorch follows the following broadcast conventions.

##### Batched Matrix x Broadcasted Vector
In the next example, we have batches of 5 matrices, each of size 3x4, and we
multiply each by a vector of size 4.

So $[5\times3\times4] @ [4\times1] = [5\times3\times1]$.

```{python}
# batched matrix x broadcasted vector
tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))
print(f"tensor1.size(): {tensor1.size()}")
print(f"tensor1: {tensor1}")

tensor2 = torch.randint(low=0, high=10, size=(4,))
print(f"tensor2.size(): {tensor2.size()}")
print(f"tensor2: {tensor2}")

tensor3 = torch.matmul(tensor1, tensor2)
print(f"tensor3 = torch.matmul(tensor1, tensor2): {tensor3}")
print(f"tensor3.size(): {tensor3.size()}")

latex_tensor3 = tensor_to_latex(tensor3)
display(Math(f"\\mathrm{{tensor3}} = {latex_tensor3}"))
```


##### Batched Matrix x Batched Matrix
What about if we multiply two batched matrices?

For example $[5\times3\times4] @ [5\times4\times5] = [5\times3\times5]$.

```{python}
# batched matrix x batched matrix
tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))
print(f"tensor1.size(): {tensor1.size()}")
print(f"tensor1: {tensor1}")

tensor2 = torch.randint(low=0, high=10, size=(5, 4, 5))
print(f"tensor2.size(): {tensor2.size()}")
print(f"tensor2: {tensor2}")

tensor3 = torch.matmul(tensor1, tensor2)
print(f"tensor3 = torch.matmul(tensor1, tensor2): {tensor3}")
print(f"tensor3.size(): {tensor3.size()}")
# torch.Size([10, 3, 5])
```

So we multiply the first matrix in the first batch by the first matrix in the second batch.


##### Batched Matrix x Broadcasted Matrix

And here we have $[5\times3\times4] @ [4\times5] = [5\times3\times5]$.

```{python}
# batched matrix x broadcasted matrix
tensor1 = torch.randint(low=0, high=10, size=(5, 3, 4))
tensor2 = torch.randint(low=0, high=10, size=(4, 5))
tensor3 = torch.matmul(tensor1, tensor2)
print(f"tensor3: {tensor3}")
print(f"tensor3.size(): {tensor3.size()}")
```


### Element-wise Multiplication

```{python}
# This computes the element-wise product. z1, z2, z3 will have the same value
z1 = tensor * tensor
z2 = tensor.mul(tensor)

z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)
```

**Single-element tensors** If you have a one-element tensor, for example
by aggregating all values of a tensor into one value, you can convert it
to a Python numerical value using `item()`:

```{python}
agg = tensor.sum()
agg_item = agg.item()
print(agg_item, type(agg_item))
```

**In-place operations** Operations that store the result into the
operand are called in-place. They are denoted by a `_` suffix. For
example: `x.copy_(y)`, `x.t_()`, will change `x`.

```{python}
print(f"{tensor} \n")
tensor.add_(5)
print(tensor)
```


::: {.callout-note}
In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged.
:::

## Fully Connected Layer as Matrix Multiplication

As a reminder, here's a fully connected network (also known as  dense network)
with 3 inputs, 4 hidden nodes and 2 outputs.

![Deep Fully Connected Network](assets/images/pytorch/shallow_net.png)

For example, we can express the hidden layer of this network as a matrix multiplication.

$$
\begin{bmatrix}
w_{00} & w_{01} & w_{02} \\
w_{10} & w_{11} & w_{12} \\
w_{20} & w_{21} & w_{22} \\
w_{30} & w_{31} & w_{32} \\
\end{bmatrix}
\begin{bmatrix}
x_{0} \\
x_{1} \\
x_{2} \\
\end{bmatrix} +
\begin{bmatrix}
b_{0} \\
b_{1} \\
b_{2} \\
b_{3} \\
\end{bmatrix}
=
\begin{bmatrix}
h_{0} \\
h_{1} \\
h_{2} \\
h_{3} \\
\end{bmatrix}
$$

Which we can compute directly on tensors.

```{python}
import torch

torch.manual_seed(42)

# Define the input tensor (batch size of 1 for simplicity)
input_tensor = torch.tensor([[1.0, 2.0, 3.0]])

# Initialize weights and biases for the hidden layer
weights_hidden = torch.randn(3, 4)  # 3 inputs to 4 hidden nodes
bias_hidden = torch.randn(4)        # 4 hidden nodes

# Initialize weights and biases for the output layer
weights_output = torch.randn(4, 2)  # 4 hidden nodes to 2 outputs
bias_output = torch.randn(2)        # 2 outputs

# Perform matrix multiplication and add biases for the hidden layer
hidden_layer = torch.matmul(input_tensor, weights_hidden) + bias_hidden

# Apply ReLU activation function
hidden_layer_activated = torch.relu(hidden_layer)

# Perform matrix multiplication and add biases for the output layer
output_layer = torch.matmul(hidden_layer_activated, weights_output) + bias_output

print("Output:", output_layer)
```

Let's look at the matrix equations.

Here's the calculation of the hidden layer.

```{python}
#| code-fold: true
latex_input_tensor = tensor_to_latex(input_tensor.T)
latex_weights_hidden = tensor_to_latex(weights_hidden.T)
latex_bias_hidden = tensor_to_latex(bias_hidden)
latex_weights_output = tensor_to_latex(weights_output.T)
latex_bias_output = tensor_to_latex(bias_output)
latex_hidden_layer = tensor_to_latex(hidden_layer.T)
latex_hidden_layer_activated = tensor_to_latex(hidden_layer_activated.T)
latex_output_layer = tensor_to_latex(output_layer.T)

# display(Math(f"\\text{{input tensor}} = {latex_input_tensor}"))
# display(Math(f"\\text{{weights hidden}} = {latex_weights_hidden}"))
# display(Math(f"\\text{{bias hidden}} = {latex_bias_hidden}"))
# display(Math(f"\\text{{weights output}} = {latex_weights_output}"))
# display(Math(f"\\text{{bias output}} = {latex_bias_output}"))
# display(Math(f"\\text{{hidden layer}} = {latex_hidden_layer}"))
# display(Math(f"\\text{{hidden layer activated}} = {latex_hidden_layer_activated}"))
# display(Math(f"\\text{{output layer}} = {latex_output_layer}"))

display(Math(f"{latex_weights_hidden} \\times {latex_input_tensor} + {latex_bias_hidden} = {latex_hidden_layer}"))
```

Then we apply the ReLU activation function.

```{python}
#| code-fold: true
display(Math(f"\\mathrm{{ReLU}}({latex_hidden_layer}) = {latex_hidden_layer_activated}"))
```

And finally we calculate the output linear layer.

```{python}
#| code-fold: true
display(Math(f"{latex_weights_output} \\times {latex_hidden_layer_activated} + {latex_bias_output} = {latex_output_layer}"))
```

### Fully Connected Layer with Batched Input

With no change to the previous code, we can now process a batch of inputs.

```{python}
import torch

torch.manual_seed(42)

# Define the input tensor (batch size of 5)
input_tensor = torch.tensor([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0], [10.0, 11.0, 12.0], [13.0, 14.0, 15.0]])

#vvvv The code below is the same as above ^^^^^

# Initialize weights and biases for the hidden layer
weights_hidden = torch.randn(3, 4)  # 3 inputs to 4 hidden nodes
bias_hidden = torch.randn(4)        # 4 hidden nodes

# Initialize weights and biases for the output layer
weights_output = torch.randn(4, 2)  # 4 hidden nodes to 2 outputs
bias_output = torch.randn(2)        # 2 outputs

# Perform matrix multiplication and add biases for the hidden layer
hidden_layer = torch.matmul(input_tensor, weights_hidden) + bias_hidden

# Apply ReLU activation function
hidden_layer_activated = torch.relu(hidden_layer)

# Perform matrix multiplication and add biases for the output layer
output_layer = torch.matmul(hidden_layer_activated, weights_output) + bias_output

print("Output:", output_layer)
```


------------------------------------------------------------------------

### Further Reading

-   [torch.utils.data API](https://pytorch.org/docs/stable/data.html)


## References

- [PyTorch](https://pytorch.org/)
- [PyTorch Tutorials](https://pytorch.org/tutorials/)
- [PyTorch Documentation](https://pytorch.org/docs/)
- [PyTorch GitHub](https://github.com/pytorch/pytorch)
- [PyTorch Brand Guidelines](https://pytorch.org/assets/brand-guidelines/PyTorch-Brand-Guidelines.pdf)