Example of Providing Custom Modules

[petrbrzek]

Hello, thank you for the nice library with a great API. I have a question: how can I provide my own modules? I would like to achieve essentially what is described here: https://github.com/justjake/quickjs-emscripten?tab=readme-ov-file#async-module-loader. Thanks.

[sebastianwessel]

Hey there,

Thanks for your interest in this library!

I'm currently working on extending the documentation (see pr #5). Here’s a basic overview of the two main options available, each targeting different use cases:

Virtual File System Options

There are two separate virtual file systems: one for primitive node_modules and another for application parts. The application in the sandbox can access node_modules only through import. The application file system is currently accessible only via the node:fs function, but it will be extended to support relative imports as well.

Option One: Providing Custom Node Modules

You can provide custom node modules, which can be used like regular node modules with import {...} from '...'. This method is used for all the currently available modules like node:fs, node:path.

In the runtime options, you can set nodeModules, which is essentially a simple nested object. For example, if you want to add a package my-package that is available via import { myFunction } from 'my-package', you can do it like this:

import { quickJS } from '@sebastianwessel/quickjs'
import { readFileSync } from 'node:fs'

const { createRuntime } = await quickJS()

const { evalCode } = await createRuntime({
  nodeModules: {
    'my-package': readFileSync('host/path/file'),
  },
})

await evalCode(`
import { myFunction } from 'my-package'

export default await myFunction()
`)

Currently, handling relative imports within a package is not feasible. Therefore, you should convert the library you want to use into a single JavaScript file. You can use ESBuild or Bun for this purpose.

Personally, I use Bun. Check out the repository, where you can find vendor.ts. This setup script builds the library testRunner.ts. You only need to run bun vendor.ts to build the single file, which can then be loaded as explained above.

Option Two: Providing Custom Files for node:fs Read/Write

This option follows the same setup but uses the mountFs option.

Feel free to reach out if you have any questions or need further assistance!

[sebastianwessel]

There was a bug in custom module handling, which is fixed in version 1.1.0.

An example can be found here An example can be found here

The documentation is now available

[petrbrzek]

@sebastianwessel I have a use case where I don’t know in advance what modules the user will import. So, I need to resolve them dynamically. This can be done here https://github.com/justjake/quickjs-emscripten?tab=readme-ov-file#async-module-loader because a callback is called and I can resolve the dependencies. Do I understand correctly that this cannot be done now?

[sebastianwessel]

...but you should know which modules the sandbox can provide i general or? Why not providing them all?

What is the reason to have this dynamically during sandbox execution?
In regular runtimes, you also need the npm install step before.

I do not really understand, why there is a need to have it dynamically during sandbox execution.

...but to answer your question:
The sandbox only has access to the provided virtual file system. This includes the module loader as well.
Because of this, it is not possible to add dynamically additional packages & modules, which are not in the runtime setup.
It's also kind of unwanted and prevented by design intentionally.

[petrbrzek]

I am working on a low-code platform where users can write their own JavaScript functions and can also use npm dependencies. These dependencies will then be resolved dynamically through https://esm.sh/.

I’m basically doing exactly what’s in this README. Except instead of using fs, I will make a fetch request to esm.sh.

runtime.setModuleLoader((moduleName) => {
  const modulePath = path.join(importsPath, moduleName)
  if (!modulePath.startsWith(importsPath)) {
    throw new Error("out of bounds")
  }
  console.log("loading", moduleName, "from", modulePath)
  return fs.readFile(modulePath, "utf-8")
})

// evalCodeAsync is required when execution may suspend.
const context = runtime.newContext()
const result = await context.evalCodeAsync(`
import * as React from 'esm.sh/react@17'
import * as ReactDOMServer from 'esm.sh/react-dom@17/server'
const e = React.createElement
globalThis.html = ReactDOMServer.renderToStaticMarkup(
  e('div', null, e('strong', null, 'Hello world!'))
)
`)

So it would be sufficient if the setModuleLoader callback were exposed to me.

[sebastianwessel]

Ah, ok, I see. Now I understand what you try to achieve.

Generally, the createRuntime method also returns vm

import { quickJS } from '@sebastianwessel/quickjs'
const { createRuntime } = await quickJS()
const { evalCode, vm } = await createRuntime()

vm.context.runtime.setModuleLoader(....)

But, currently, the created virtual file system is not exposed.
I will add it to the return of createRuntime().
Until next release, you can simply manually edit the package in node_modules - the code should be readable.

In quickJs.js (in esm or commonjs folder - depending on your project settings)
from

return { vm: arena, dispose, evalCode };

to

return { vm: arena, dispose, evalCode, fs };

import { quickJS } from '@sebastianwessel/quickjs'
const { createRuntime } = await quickJS()
const { evalCode, vm, fs } = await createRuntime()

vm.context.runtime.setModuleLoader((moduleName) => {
  const modulePath = path.join(importsPath, moduleName)
  if (!modulePath.startsWith(importsPath)) {
    throw new Error("out of bounds")
  }
  console.log("loading", moduleName, "from", modulePath)
  // use the the fs returned by createRuntime, not the Node.js one for readFile
  return fs.readFile(modulePath, "utf-8")
}, modulePathNormalizer)


const modulePathNormalizer: JSModuleNormalizer = (baseName: string, requestedName: string) => {
	// relative import
	if (requestedName.startsWith('.')) {
		const parts = baseName.split('/')
		parts.pop()

		return resolve(`/${parts.join('/')}`, requestedName)
	}

	// module import
	const moduleName = requestedName.replace('node:', '')
	return join('/node_modules', moduleName)
}

Created an issue #19 to improve this

[petrbrzek]

@sebastianwessel Cool, thanks 🙏.

[sebastianwessel]

@petrbrzek version 1.3 is out. It returns moutedFs and the package also exports modulePathNormalizer.

So, it should make it a bit easier for you.