Path to Native Wasm

[Stebalien]

This is a sequel to #769 mapping out potential avenues to permission less wasm actors.

Launching truly "native" Wasm actors is tricky for two reasons:

1. Gas accounting potentially malicious Wasm code is tricky. We've tuned our current Wasm gas model based on the expected execution time of our current built-in actors, not based on the worst possible program. The worst possible program could, e.g., ~80gas per instruction that could randomly access memory (20x what we usually charge for instructions) simply because we wouldn't be able to assume the CPU cache. Branch prediction, CPU pipelines, and out-of-order CPU instruction execution throw yet another wrench into the mix.
2. We rely on an optimizing compiler (wasmtime's cranelift) to compile WASM down to native and optimizing compilers have unpredictable compile times as a general rule. Unfortunately, this process is also (at the moment) platform specific so running this logic in wasm (so we can charge for it) is tricky.

Given this, we're considering a few options for launching native Wasm actors, listed below in order of increasing difficulty and performance.

Interpreter In Wasm (Substrate's solution)

Compile a wasm interpreter to wasm (e.g., Parity's wasmi) and interpret user-deployed wasm actors. The performance is expected to be pretty abysmal, but it (hopefully) will be no worse than the EVM.
- This mitigates the gas accounting issue by moving user code one step away from the hardware: by making everything slower, we're reducing the impact of things like cache misses and by interpreting wasm code, we're making it difficult for an attacker to execute arbitrary sequences of native instructions on the validator's CPU.
- This fixes the compilation issue by entirely removing that step.

This is the fastest path to native wasm actors and will likely be the first approach proposed by the FVM team. However, there are two possible approaches:

1. FEVM-like.
2. Native-like.

FEVM-Like Wasm Interpreter

We could add a Wasm interpreter to Filecoin the same way we added the EVM:

1. Set the actor's "code CID" to the wasm interpreter's code CID.
2. Record the wasm actor's actual code CID in the actors state-tree.

The main benefit of this approach is that it's "just another runtime". However, it has some significant downsides:

1. Actors will have two wasm code CIDs, which may be confusing.
2. The upgrade path to "compiled" wasm is tricker.

Native-Like Wasm Interpreter ⭐

Alternatively, we could make it look a little more native:

1. Set the actor's "code CID" to the user provided code CID.
2. When invoking an actor, check if the code CID is "builtin". If so, execute normally. If not, load the wasm module for the wasm interpreter and pass the actor's code CID into the interpreter.

This has the benefit of having a better upgrade path to future interpreter-less wasm actors.

NOTE: After discussing this at the FilDev summit in Iceland, this seems to be the popular approach.

Native Interpreter

Alternatively, we could run the interpreter natively. However, this requires improving the gas model as we'd be metering the user's wasm directly instead of metering the interpreter itself.

Single Pass Compiler (NEAR's solution)

Use a single pass compiler (wasmer has one, wasmtime is building one) and execute user-deployed wasm actors with a pessimistic gas model instead of the "expected time" gas model we use for builtin actors.

• This is tricker because we need a new gas model. Basically, we need a bunch of "slow" wasm programs to test against.
• We'll still need to find some way to charge for the single pass compilation step. But the runtime should be linear in the size of the program so we should be fine as long as we specify a large enough safety factor.
• While this version should be faster in terms of wall-clock time, it's unclear whether or not it'll be any more gas efficient due to the pessimistic gas model.

Optimizing Compiler

Find a way to compile/optimize wasm actors. Unfortunately, while both of these approaches may make the wasm bytecode execute faster, turning that enhanced performance into reduced gas costs is difficult.

Really, I don't know how to make this work, I'm just throwing out a few ideas.

CLIF

Compile to CLIF (cranelift intermediate format) inside a wasm actor (charging gas) before lowering to native outside. Unfortunately, this bakes in cranelift and wasmtime, and may not even be sufficient (the lowering step may need to, e.g., do register allocation?).

Consensus

Let the network (validators) compile wasm actors in the background and switch to some "cheaper" gas model for the newly compiled actor once some fraction of the network has successfully compiled it. There would likely need to be some form incentive system to make this happen.

Gas Spent

One option is to compile actors to native wasm after the network has "spent" some amount of gas (or Filecoin) on them. This means that "popular" actor code will get auto-optimized.

Gas Savings

To get gas savings here, we'll either need:

1. Some way to detect and discount "optimizable" sequences of wasm instructions.
2. Some form of "these actors are well behaved" consensus, either automatic (ideal but very hard) or manual (a FIP-like process where the community can agree on "well behaved" libraries/actors that can run with the cheaper gas model).

[bmann]

Some form of "these actors are well behaved" consensus, either automatic (ideal but very hard) or manual (a FIP-like process where the community can agree on "well behaved" libraries/actors that can run with the cheaper gas model).

I notice the previous comment on FIP-like process. In the Ethereum ecosystem, this has typically been the role of ERCs -- non binding, non consensus affecting common patterns, like NFT ERC721. Core devs do not decide (but often weigh in), and it's up to the authors to gather consensus, re-use, and interop.

I'd suggest the same sort of lightweight process, but maybe not directly "in" the same FIP repo (that's been problematic for the EIP process).

Sounds like lots of this stuff would "just work" with IPVM efforts, but I know @expede is looking at this more closely and will likely have some thoughts of her own.

[Stebalien]

Unfortunately, this would need to be a binding process as the gas price-list is consensus critical. I.e., the network would need to decide to "charge less" for these actors.

[expede]

turning that enhanced performance into reduced gas costs is difficult.

I think that this is an admirable goal, but not strictly nessesary for an MVP. Making things execute faster / more efficiently for runners seems like a good idea, be greener etc! You can definitely discount the optimized code — there are known techniques for this, but it's going to largely be custom code that's off the strictly critical path for Filecoin. A possible easy win on the metering side is using a specializer. Because this still runs normal Wasm, it would probably be very easy to meter, but at the cost of storage (the space/time tradeoff) — I hear that you may have a lot of storage kicking around Filecoin, though 😉

Another thing to consider is that very aggressive optimizations can paradoxically increase the running time of certain (but not all!) inputs, so you may need to be somewhat conservative in which optimizations are appropriate.

We had started down this road a bit back in my EVM days, but I ejected from that community before we wrote any EIPs for these (just gave some presentations and workshops). A bunch of these ideas ended up in IPVM, and I think that running at least time-bounded hotspot optimization after some threshold of successful runs is a good idea. Basically, don't bother optimizing most code, but if you have a cryptokitties on your hands, the providers should run an optimizer. Nodes can share optimized code (from trusted sources or ZKed), and you can get around JIT bombs by 1. running lighter optimizations first and 2. limiting how long a particular optimizer pass can run for (longer for later passes).

Filecoin could potentially lift code from IPVM to do dynamic optimization across smart contracts. There's probably lower hanging fruit from FVM, but we're going to do this work in Wasm anyways and maybe you could just lift it directly if we implement it the right way. This includes the strategies mentioned above, but also running a specializer to reuse partial results across smart contracts. We don't have an implementation for this today, but it's going to be part of the Rust implementation of IPVM (Homestar). It may be worth digging deeper into what your needs from this are and aligning our roadmaps to make sure that the stuff we build can be modularized & reused if that's interesting :)

We rely on an optimizing compiler (wasmtime's cranelift) to compile WASM down to native and optimizing compilers have unpredictable compile times as a general rule. Unfortunately, this process is also (at the moment) platform specific so running this logic in wasm (so we can charge for it) is tricky.

Just wanted to highlight my earlier comment that this can be done progressively even with Cranelift or LLVM — but that is likely custom code. This certainly isn't unheard of, and for small optimizations it's not that much work TBH, but I'd be surprised if Cranelift exposed relevant APIs, so we're talking about adding custom runtime optimization which is likley nonzero new code.

Some form of "these actors are well behaved" consensus, either automatic (ideal but very hard) or manual (a FIP-like process where the community can agree on "well behaved" libraries/actors that can run with the cheaper gas model).

As @bmann mentions above, this sounds very similar to the EIP precompile process. While it is possible to automate this at some stage, "doing things that don't scale" (i.e. community calls & human giovernance) makes sense to me for a first pass 💯

[Stebalien]

I think that this is an admirable goal, but not strictly nessesary for an MVP.

I completely agree. The MVP will likely be an interpreter. Not great, but easy to secure.

As @bmann mentions above, this sounds very similar to the EIP precompile process. While it is possible to automate this at some stage, "doing things that don't scale" (i.e. community calls & human giovernance) makes sense to me for a first pass 💯

Yeah, I was thinking EIP precompiles here. Except, with wasm, we don't have to make everyone actually implement these native functions. They just need to agree that they're "well behaved", run them through an optimizing compiler, and run them natively.

[bmann]

Yeah that was my point. Perhaps a FIP that can be updated over time and reviewed once per quarter for main net, and a much faster moving allow list for testnet.

[laudiacay]

I’ve always thought the paradigm of “assigning a gas value to ops then charging a flat rate” forces a centralized party to decide on fixed exchange rates between instructions. This seems silly, especially for something so significantly more complex than EVM bytecode as WASM.

I think doing EVM style gas might be an avoidable waste of time, I believe this work should just be done by a market.

Has anyone done any work on auction mechanisms that incentivize “accurate” relative market pricing of the compute resources required to execute a transaction? Modified Vickrey auction maybe…? If so, is there any reason we couldn’t use these solutions?

IIRC Polkadot has some rudimentary work here but I haven’t dove in deep. I’m going to send this to my microecon friends to see if they have further thoughts…

[Stebalien]

A true market may be possible but it would be tricky and might end up with slow block times. Basically, you'd need a relative weight function where block producers take the compute time of a chain into account. I say relative because the weight of some parent P would vary based on the hardware of the block producer building some block B on top of P.

That is definitely an interesting proposal. In this world, instead of packing messages based on gas, a block producer would:

1. Pack messages up to a real-world estimated "time" limit such that the network is expected to accept the block.
2. Pick messages by effectively "auctioning off" time in the block. This is similar to what we currently do except (a) I don't think the current EIP-1559 style gas burning is workable and (b) a user could, in theory, pay more to get an extra large message included (incentivizing the block producer to take a risk).

But..., this is far from trivial:

• It definitely doesn't work with tipsets or delayed execution.
• We'd definitely need a very good VDF to make catch-up possible (to delay the time between validating/executing some parent block P and creating a new block B). Otherwise, this system would incentivize blocks that take nearly an entire block time to execute.
• This would drive block producers to buy faster and faster computers. Which is good? I guess?
• This would likely lead to standardizing the network around specific CPUs.

This is definitely something we should look into (and certainly something someone else has worked on), but it's probably a longer-term project.

[BlocksOnAChain]

@Stebalien I just wanted to communicate how great it is to see that we have all our potential ideas and plans presented in this discussion. Thanks for sharing it all.
🤝
I genuinely like the Native-Like Wasm interpreter idea as the "mid-path" solution for people who are excited about FVM Wasm actors and just want to test and build things, but being aware that this is still experimental and that all Wasm Actors features will be unlocked later on.
I will do a deeper dive into the Parity and Near solutions we shared here, so we can talk more and determine some good ideas to use and to define our future roadmap.

[Stebalien]

The new rowhammer++ attack is going to be interesting: https://arstechnica.com/security/2023/10/theres-a-new-way-to-flip-bits-in-dram-and-it-works-against-the-latest-defenses/

This kind of thing can be "fixed" by making sure we interrupt and pause every so often, but it's one of the many reasons "native wasm" will be non-trivial.

[he-chien-tsai]

The main point of gas fees is resisting attack vectors. Precise, fair or cheap price is unnecessary. Users and developers are not sensitive to price in practice.

A rougher dynamic approach may be good enough. which avoids hassles of determining precise price and auctioning market. It may works with Native Interpreter.

Set a staging fee P with a reasonable large number to resist attack vectors and user only has to pay once in common use cases.
Set a threshold h for triggering further invoice.

1. User have to pay initial fee P in order to call a function.
2. Node create two threads T1 and T2 . T1 run the function call . T2 monitor T1

In thread T2 :

Set the sum of computing cost _C_ := 0 initially
Repeat until _T1_ is finished :
      Calculate the rough gas fee _g_ by  incremental instruction cycles and memory used of wasm interpreter in _T1_ 
      _C_ := _C_ + _g_
      If _C_ + _h_ > _P_ : 
          Pause _T1_ . Send an invoice of _P_ to user for further computation .
          If user doesn't pay the invoice :  seize the fee that has been paid . kill _T1_ .
          If user pay the invoice : reset _C_ := 0 . resume _T1_ . 
If _P_ > _C_ : refund _P_ - _C_  to user.

I'm very against Optimizing Compiler. Optimizing introduces unnecessary complexity and potential unexpected behaviors that should not be mandatory by network. Developers can decide whether optimizing is appropriate by themself.

It looks that undetermined things just keep growing with discussions. May you release test nets of some potential solutions and call for testing, which can settle them one by one.