feat: implement Zeromorph (#145)

* Schoolbook adapted KZG PoC

Remove toxic waste from setup parameters

Refactorings

ZeroMorph skeleton

* feat: provide a non-hiding variant of KZG

Drafts a non-hiding variant of the KZG PCS on univariate Dense polynomial representations

Details:
- Changed access levels and ordering for various modules and functions like `sumcheck`, `cpu_best_multiexp`, and `mod kzg;`.
- Upgraded `halo2curves` version and added new dependencies like `pairing`, `anyhow`, `rand`, and `group`.
- Leveraged the UniPoly in sumcheck.rs
- Created `non_hiding_kzg.rs` file, introducing new structures and functionalities like `UVUniversalKZGParam`, `UVKZGProverKey`, `UVKZGPoly`, and `UVKZGPCS` along with their implementation and tests.
- Modified the import of `Engine` in `kzg.rs` from `halo2curves::pairing::Engine` to `pairing::Engine` (reflecting the new version of halo2curves).

feat: add batch commit / open / prove

* Small subtasks: params, phi, un function, EE

Minor adjustments

* chore: setup zeromorph

refactor: Remove kzg and zeromorph provider modules

* feat: add openings for Zeromorph

- Added new data structures and their related functions including the `ZMProverKey`, `ZMVerifierKey`, `ZMCommitment`, `ZMEvaluation`, `ZMProof` and `ZMPCS` in `non_hiding_zeromorph.rs`
- Implemented new functionalities in `non_hiding_zeromorph.rs` and `spartan/polynomial.rs` to provide better handling of polynomials, including fetching commitments for a polynomial, computing the quotient polynomials of an existing polynomial.
- Enhanced the `UniPoly` struct in `spartan/sumcheck.rs` with `Index` and `IndexMut` for better coefficient access, and `AddAssign` and `MulAssign` for scalar and self types.
- Removed and replaced certain elements in `UVUniversalKZGParam` struct in `non_hiding_kzg.rs` for improved flexibility, and included import and implementation of `TranscriptReprTrait`.

draft ZM verify

set up test structure

defer test_quo

* Include commits into ZMProof and adjust RO

* feat: Refactor polynomial evaluation methods

- Temporarily disabled some assertions in both `non_hiding_zeromorph.rs` and `polynomial.rs` for debugging purposes.
- Introduced `evaluate_opt`, `fix_variables`, and `fix_one_variable_helper` functions in `polynomial.rs` to support multilinear polynomial evaluation and partial evaluation.

fix: add domain separators

- Integrated additional functionality into the ZMPCS implementation in `non_hiding_zeromorph.rs` by adding a protocol name function.
- Improved security in `non_hiding_zeromorph.rs` by integrating domain separators into transcript's `open` and `verify` functions.

refactor: Refactor code and test performance for NHZM

- Implement `AsRef` trait for `UniPoly` structure in `src/spartan/sumcheck.rs` to facilitate direct access to its coefficients.
- Refactor code in `src/provider/non_hiding_zeromorph.rs` to directly use `into_iter` in map function when creating `quotients_polys`, avoiding a large clone
- Enhance test performance in `src/provider/non_hiding_zeromorph.rs` by pre-generating a `universal_setup` for tests, introducing a `max_vars` variable and RNG in the `commit_open_verify_with` test function, and adjusting the range of num_vars used for testing.

fix: adjust APIs & Serialize impls for the Nova traits

* feat: set up serde & abomonation bounds

* add generic PCS errors

* pp

wip

* fix: adjust type parameters for nontrivial_with_zm test

fix: ignore panicking test

fix: add doctest for evaluate_opt

fix: remove obsolete comments

chore: move UniPoly methods where they should be

test: make clear current zeromorph operates in monomial basis

- Added an additional test `test_dense_evaliations()` to provide more comprehensive testing for the `evaluate()` and `evaluate_opt()` functions in `MultilinearPolynomial`.

refactor TranscriptReprTrait impl for compat with Commitments

* feat: Implement KZG commitment trait and serialization features

- Added serialization, deserialization, and Abomonation traits to UVUniversalKZGParam struct in `non_hiding_kzg.rs` file along with implementing comparison and length evaluation traits.
- Created a new file `kzg_commitment.rs` which implements KZG Commitment Engine and setup, commit functions.
- Integrated `kzg_commitment` in module `provider` and set up conversions between Commitment and UVKZGCommitment.
- Enhanced assertion in `minroot_serde.rs` file from clone comparison to dereferenced comparison in MinRoot delay case.

* Use the ZMPCS Evaluation Engine and the KZG Commitment Engine in tests.

feat: Improve `prove` and `verify` methods in `ZMPCS` struct

- Updated `prove` and `verify` methods in `ZMPCS<E>` struct within `non_hiding_zeromorph.rs` with actual logic for commitment, evaluation, and verification.
- Adjusted the construction of `ZMCommitment` and `ZMEvaluation` within `prove` and `verify` methods.
- Commented on portions of the code in `non_hiding_zeromorph.rs` that need further modification and refinement.
- Modified test value for `test_pp_digest_with` in the `test_supernova_pp_digest` test case to match the new expected output.

* fix evaluation reversal bug

fix: remove superfluous eval functions

fix: remove endianness shenanigans

test: add evaluation unit test

* fix: parallellize pp generation

* feat: Enhance KZG commitment SRS generation efficiency using parallel computation

- Introduced a new module `util` within the `provider` module, implementing fixed-base MSM,
- New trait constraint has been imposed for `E::Fr: PrimeFieldBits` within the `non_hiding_zeromorph.rs` file, and usages have been adjusted in the `test` module.
- Adding the `PrimeFieldBits` import from the `ff` crate and importing `provider::util` from the local crate.

refactor: Refactor utility functions for elliptic curve operations

- Renamed and moved `util.rs` to `fb_msm.rs` under the `provider` directory.
- documented the FB MSM

* Apply comments from code review

Co-authored-by: Adrian Hamelink <[email protected]>

* Improve documentation and testing in non_hiding_zeromorph.rs

- Enhanced the clarity and accuracy of code comments and documentation in `non_hiding_zeromorph.rs`, specifically within the `ZMPCS` struct's methods.
- Enriched the `quotients` function's documentation, detailing its mathematical underpinnings
- Implemented a new rigorous test, `test_quotients`, to ensure the `quotients` function's output satisfies the polynomial identity.

* test: refactor random ML poly generation

* Finish documenting / testing Zeromorph (#142)

* fix: remove a TODO using RefCast

* fix: remove unsightly clone

* doc: comment intermediate ZM steps

* refactor: Refactor `open` function in zeromorph provider

Extracted the computation of `q_hat` in the `open` function into a new function `batched_lifted_degree_quotient` for more modular code.

* test: test batched_lifted_degree_quotient

Addition of a new test `test_batched_lifted_degree_quotient` to validate batched degree quotient lifting for polynomials.

* refactor: Refine computations in non_hiding_zeromorph.rs

- Overhauled the 'eval_and_quotient_scalars' function with a revised return type with pair of vectors rather than scalar and a vector,
- correspondingly split scalars `degree_check_q_scalars` and `zmpoly_q_scalars` within the `open` method of `non_hiding_zeromorph.rs`.
- Adjusted the 'verify' method to accommodate the modified 'eval_and_quotient_scalars' function output changes.

* test: test partially evaluated quotient \zeta_x

- Introduced a new test case `test_partially_evaluated_quotient_zeta` to validate `zeta_x` construction.
- correct an off-by-one error in the y_k terms

* test: test partially evaluated quotient

- Created a new function `phi` for evaluating using an inefficient formula in `non_hiding_zeromorph.rs`
- Implemented a new test function `test_partially_evaluated_quotient_z` for validation of `Z_x` computation method in `non_hiding_zeromorph.rs`

* fix: refine q_hat comment

* fix: organize code more sparsely after rebase

* refactor: Improve code readability and error handling in zeromorph

- Updated and enhanced clarity and consistency of mathematical notation in comments across `non_hiding_kzg.rs` and `non_hiding_zeromorph.rs` files.
- Implemented error handling in the `ZMPCS::verify` function within the `non_hiding_zeromorph.rs` file.

* fix: compute quotient commitments in parallel

* refactor: clean up

---------

Co-authored-by: emmorais <[email protected]>
Co-authored-by: Matej Penciak <[email protected]>
Co-authored-by: Adrian Hamelink <[email protected]>

Cargo.toml

@@ -34,6 +34,7 @@ byteorder = "1.4.3"
 thiserror = "1.0"
 halo2curves = { version = "0.4.0", features = ["derive_serde"] }
 group = "0.13.0"
+pairing = "0.23.0"
 abomonation = "0.7.3"
 abomonation_derive = { version = "0.1.0", package = "abomonation_derive_ng" }
 tap = "1.0.1"
@@ -43,6 +44,8 @@ tracing-subscriber = { version = "0.3.17", features = ["env-filter"] }
 cfg-if = "1.0.0"
 once_cell = "1.18.0"
 itertools = "0.12.0"
+rand = "0.8.5"
+ref-cast = "1.0.20"
 
 [target.'cfg(any(target_arch = "x86_64", target_arch = "aarch64"))'.dependencies]
 pasta-msm = { git="https://github.com/lurk-lab/pasta-msm", branch="dev", version = "0.1.4" }
@@ -59,7 +62,6 @@ pprof = { version = "0.11" }
 sha2 = "0.10.7"
 tracing-test = "0.2.4"
 proptest = "1.2.0"
-rand = "0.8.5"
 
 [[bench]]
 name = "recursive-snark"

ThirdPartyNotices.txt

@@ -52,3 +52,22 @@ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLI
 WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
 COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
 TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+------------------------------------------------------------
+https://github.com/AztecProtocol/aztec-packages/
+
+Licensed under Apache 2.0
+
+Copyright 2022 Aztec
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

src/errors.rs

@@ -36,9 +36,9 @@ pub enum NovaError {
   /// returned if the provided number of steps is zero
   #[error("InvalidNumSteps")]
   InvalidNumSteps,
-  /// returned when an invalid inner product argument is provided
-  #[error("InvalidIPA")]
-  InvalidIPA,
+  /// returned if there is an error in the proof/verification of a PCS
+  #[error("PCSError")]
+  PCSError(#[from] PCSError),
   /// returned when an invalid sum-check proof is provided
   #[error("InvalidSumcheckProof")]
   InvalidSumcheckProof,
@@ -70,3 +70,17 @@ pub enum NovaError {
   #[error("InternalError")]
   InternalError,
 }
+
+/// Errors specific to the Polynomial commitment scheme
+#[derive(Clone, Debug, Eq, PartialEq, Error)]
+pub enum PCSError {
+  /// returned when an invalid inner product argument is provided
+  #[error("InvalidIPA")]
+  InvalidIPA,
+  /// returned when there is a Zeromorph error
+  #[error("ZMError")]
+  ZMError,
+  /// returned when a length check fails in a PCS
+  #[error("LengthError")]
+  LengthError,
+}

src/lib.rs

@@ -1000,14 +1000,15 @@ mod tests {
   use super::*;
   use crate::{
     provider::{
-      traits::DlogGroup, Bn256Engine, GrumpkinEngine, PallasEngine, Secp256k1Engine,
-      Secq256k1Engine, VestaEngine,
+      non_hiding_zeromorph::ZMPCS, traits::DlogGroup, Bn256Engine, Bn256EngineZM, GrumpkinEngine,
+      PallasEngine, Secp256k1Engine, Secq256k1Engine, VestaEngine,
     },
     traits::{evaluation::EvaluationEngineTrait, snark::default_ck_hint},
   };
   use ::bellpepper_core::{num::AllocatedNum, ConstraintSystem, SynthesisError};
   use core::{fmt::Write, marker::PhantomData};
   use ff::PrimeField;
+  use halo2curves::bn256::Bn256;
   use traits::circuit::TrivialCircuit;
 
   type EE<E> = provider::ipa_pc::EvaluationEngine<E>;
@@ -1142,6 +1143,18 @@ mod tests {
       &trivial_circuit2_grumpkin,
       "5aec6defcb0f6b2bb14aec70362419388916d7a5bc528c0b3fabb197ae57cb03",
     );
+    #[cfg(not(feature = "asm"))]
+    test_pp_digest_with::<Bn256EngineZM, GrumpkinEngine, _, _, ZMPCS<Bn256, _>, EE<_>>(
+      &trivial_circuit1_grumpkin,
+      &trivial_circuit2_grumpkin,
+      "e20ab87e395e787e272330a4c9c79916b83bf95632d9604511ad6a1448625402",
+    );
+    #[cfg(not(feature = "asm"))]
+    test_pp_digest_with::<Bn256EngineZM, GrumpkinEngine, _, _, ZMPCS<Bn256, _>, EE<_>>(
+      &cubic_circuit1_grumpkin,
+      &trivial_circuit2_grumpkin,
+      "ce304ffff7f6dfc143322bb621e9025ba9d77f1b8d0e199e6bc432aa66c98101",
+    );
 
     let trivial_circuit1_secp = TrivialCircuit::<<Secp256k1Engine as Engine>::Scalar>::default();
     let trivial_circuit2_secp = TrivialCircuit::<<Secq256k1Engine as Engine>::Scalar>::default();
@@ -1384,6 +1397,12 @@ mod tests {
     test_ivc_nontrivial_with_compression_with::<PallasEngine, VestaEngine, EE<_>, EE<_>>();
     test_ivc_nontrivial_with_compression_with::<Bn256Engine, GrumpkinEngine, EE<_>, EE<_>>();
     test_ivc_nontrivial_with_compression_with::<Secp256k1Engine, Secq256k1Engine, EE<_>, EE<_>>();
+    test_ivc_nontrivial_with_compression_with::<
+      Bn256EngineZM,
+      GrumpkinEngine,
+      ZMPCS<Bn256, _>,
+      EE<_>,
+    >();
   }
 
   fn test_ivc_nontrivial_with_spark_compression_with<E1, E2, EE1, EE2>()
@@ -1484,6 +1503,12 @@ mod tests {
     test_ivc_nontrivial_with_spark_compression_with::<Bn256Engine, GrumpkinEngine, EE<_>, EE<_>>();
     test_ivc_nontrivial_with_spark_compression_with::<Secp256k1Engine, Secq256k1Engine, EE<_>, EE<_>>(
     );
+    test_ivc_nontrivial_with_spark_compression_with::<
+      Bn256EngineZM,
+      GrumpkinEngine,
+      ZMPCS<Bn256, _>,
+      EE<_>,
+    >();
   }
 
   fn test_ivc_nondet_with_compression_with<E1, E2, EE1, EE2>()
@@ -1626,6 +1651,8 @@ mod tests {
     test_ivc_nondet_with_compression_with::<PallasEngine, VestaEngine, EE<_>, EE<_>>();
     test_ivc_nondet_with_compression_with::<Bn256Engine, GrumpkinEngine, EE<_>, EE<_>>();
     test_ivc_nondet_with_compression_with::<Secp256k1Engine, Secq256k1Engine, EE<_>, EE<_>>();
+    test_ivc_nondet_with_compression_with::<Bn256EngineZM, GrumpkinEngine, ZMPCS<Bn256, _>, EE<_>>(
+    );
   }
 
   fn test_ivc_base_with<E1, E2>()

src/provider/ipa_pc.rs

@@ -1,6 +1,6 @@
 //! This module implements `EvaluationEngine` using an IPA-based polynomial commitment scheme
 use crate::{
-  errors::NovaError,
+  errors::{NovaError, PCSError},
   provider::{pedersen::CommitmentKeyExtTrait, traits::DlogGroup},
   spartan::polys::eq::EqPolynomial,
   traits::{
@@ -406,7 +406,7 @@ where
     if P_hat == CE::<E>::commit(&ck_hat.combine(&ck_c), &[self.a_hat, self.a_hat * b_hat]) {
       Ok(())
     } else {
-      Err(NovaError::InvalidIPA)
+      Err(NovaError::PCSError(PCSError::InvalidIPA))
     }
   }
 }

src/provider/kzg_commitment.rs

@@ -0,0 +1,78 @@
+//! Commitment engine for KZG commitments
+//!
+
+use std::marker::PhantomData;
+
+use ff::PrimeFieldBits;
+use group::{prime::PrimeCurveAffine, Curve};
+use pairing::Engine;
+use rand::rngs::StdRng;
+use rand_core::SeedableRng;
+use serde::{Deserialize, Serialize};
+
+use crate::traits::{
+  commitment::{CommitmentEngineTrait, Len},
+  Engine as NovaEngine, Group,
+};
+
+use crate::provider::{
+  non_hiding_kzg::{UVKZGCommitment, UVUniversalKZGParam},
+  pedersen::Commitment,
+  traits::DlogGroup,
+};
+
+/// Provides a commitment engine
+#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
+pub struct KZGCommitmentEngine<E: Engine> {
+  _p: PhantomData<E>,
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> CommitmentEngineTrait<NE>
+  for KZGCommitmentEngine<E>
+where
+  E::G1: DlogGroup<PreprocessedGroupElement = E::G1Affine, Scalar = E::Fr>,
+  E::G1Affine: Serialize + for<'de> Deserialize<'de>,
+  E::G2Affine: Serialize + for<'de> Deserialize<'de>,
+  E::Fr: PrimeFieldBits, // TODO due to use of gen_srs_for_testing, make optional
+{
+  type CommitmentKey = UVUniversalKZGParam<E>;
+  type Commitment = Commitment<NE>;
+
+  fn setup(label: &'static [u8], n: usize) -> Self::CommitmentKey {
+    // TODO: this is just for testing, replace by grabbing from a real setup for production
+    let mut bytes = [0u8; 32];
+    let len = label.len().min(32);
+    bytes[..len].copy_from_slice(&label[..len]);
+    let rng = &mut StdRng::from_seed(bytes);
+    UVUniversalKZGParam::gen_srs_for_testing(rng, n.next_power_of_two())
+  }
+
+  fn commit(ck: &Self::CommitmentKey, v: &[<E::G1 as Group>::Scalar]) -> Self::Commitment {
+    assert!(ck.length() >= v.len());
+    Commitment {
+      comm: E::G1::vartime_multiscalar_mul(v, &ck.powers_of_g[..v.len()]),
+    }
+  }
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> From<Commitment<NE>>
+  for UVKZGCommitment<E>
+where
+  E::G1: Group,
+{
+  fn from(c: Commitment<NE>) -> Self {
+    UVKZGCommitment(c.comm.to_affine())
+  }
+}
+
+impl<E: Engine, NE: NovaEngine<GE = E::G1, Scalar = E::Fr>> From<UVKZGCommitment<E>>
+  for Commitment<NE>
+where
+  E::G1: Group,
+{
+  fn from(c: UVKZGCommitment<E>) -> Self {
+    Commitment {
+      comm: c.0.to_curve(),
+    }
+  }
+}

src/provider/mod.rs

@@ -2,6 +2,7 @@
 
 // public modules to be used as an evaluation engine with Spartan
 pub mod ipa_pc;
+pub mod non_hiding_zeromorph;
 
 // crate-public modules, made crate-public mostly for tests
 pub(crate) mod bn256_grumpkin;
@@ -10,6 +11,10 @@ pub(crate) mod pedersen;
 pub(crate) mod poseidon;
 pub(crate) mod secp_secq;
 pub(crate) mod traits;
+// a non-hiding variant of {kzg, zeromorph}
+pub(crate) mod kzg_commitment;
+pub(crate) mod non_hiding_kzg;
+mod util;
 
 // crate-private modules
 mod keccak;
@@ -25,8 +30,11 @@ use crate::{
   },
   traits::Engine,
 };
+use halo2curves::bn256::Bn256;
 use pasta_curves::{pallas, vesta};
 
+use self::kzg_commitment::KZGCommitmentEngine;
+
 /// An implementation of the Nova `Engine` trait with BN254 curve and Pedersen commitment scheme
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub struct Bn256Engine;
@@ -55,6 +63,20 @@ impl Engine for GrumpkinEngine {
   type CE = PedersenCommitmentEngine<Self>;
 }
 
+/// An implementation of the Nova `Engine` trait with BN254 curve and Zeromorph commitment scheme
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub struct Bn256EngineZM;
+
+impl Engine for Bn256EngineZM {
+  type Base = bn256::Base;
+  type Scalar = bn256::Scalar;
+  type GE = bn256::Point;
+  type RO = PoseidonRO<Self::Base, Self::Scalar>;
+  type ROCircuit = PoseidonROCircuit<Self::Base>;
+  type TE = Keccak256Transcript<Self>;
+  type CE = KZGCommitmentEngine<Bn256>;
+}
+
 /// An implementation of the Nova `Engine` trait with Secp256k1 curve and Pedersen commitment scheme
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub struct Secp256k1Engine;