diff --git a/README.md b/README.md index 9e44f4e..aec8c1a 100644 --- a/README.md +++ b/README.md @@ -234,6 +234,18 @@ python examples/variational_autoencoder.py -m ``` ## Invariant MNIST (fully connected) +Some comparisons using parameter scans maxabs normalization as default +``` +python3 examples/invariant_mnist.py -m mlp.n=2,3,4,5,6 mlp.hidden.width=128 mlp.layer_type=polynomial optimizer=sophia +``` +| n | test accuracy | +|--------------|----------------------| +|2 | 0.9501000046730042 +|3 | 0.9785000085830688 +|4 | 0.9711999893188477 +|5 | 0.9653000235557556 +|6 | + Without polynomial refinement ```python python examples/invariant_mnist.py max_epochs=100 train_fraction=1 mlp.layer_type=continuous mlp.n=5 mlp.p_refine=False mlp.hidden.layers=4 @@ -335,11 +347,11 @@ to [KAN: Kolmogorov–Arnold Networks 2024](https://arxiv.org/pdf/2404.19756) was published (9 years after the original implementation of the technique in this repo), where B-splines were used on the grid. Looking at that repo, the real difference seems to be B-splines vs lagrange polynomials. -[Variations on the Chebyshev-Lagrange Activation Function](https://arxiv.org/abs/1906.10064) implements a linear extension +[Variations on the Chebyshev-Lagrange Activation Function](https://arxiv.org/abs/1906.10064) implements a linear extension to the values beyond [-1,1] which would solve the problem of polynomial growth outside that range. [KAN: Kolmogorov–Arnold Networks: A review 2024](https://vikasdhiman.info/reviews/KAN_a_review.pdf) A review of KANs with respect to other types of networks, especially spline networks -[Linear spline networks 2020](https://arxiv.org/pdf/2001.06263) +[Linear spline networks 2020](https://arxiv.org/pdf/2001.06263) -[Learning Activation Functions in Deep (Spline) Neural Networks 2020](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9264754) using B splines \ No newline at end of file +[Learning Activation Functions in Deep (Spline) Neural Networks 2020](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9264754) using B splines