Lacking Support for Node.js 'process' Variable

[arvindavoudi]

🚨 Problem with Node Modules Requiring Process Identification

📝 Description

The "process" variable is fundamental in Node.js environments, suggesting that pythonmonkey might not fully emulate or provide a Node.js-like environment for certain modules. Understanding how to properly bridge this gap or if there's a workaround would be crucial for progressing with my project.

🔄 Steps to Reproduce

1. Install pythonmonkey using the command pip install pythonmonkey.
2. Create a project directory with the following structure:

my_project/
├── main.py
└── package.json

3. Add the following contents to my_project/package.json:

{
    "dependencies": {
        "react": "^18.2.0",
        "react-dom": "^18.2.0"
    }
}

4. Run npm install in the project directory to install the Node.js dependencies listed in my_project/package.json, specifically 'react' and 'react-dom'.

5. Add the following Python code to my_project/main.py directory to use the "react-dom/server" module:

import pythonmonkey as pm
reactdom_server = pm.require('./node_modules/react-dom/server')

6. Finally execute my_project/main.py using the command python3 main.py.

🌟 Expected Behavior

I expected to successfully import and use "react-dom/server" in my Python script for server-side rendering of React components, leveraging pythonmonkey for integration.

🛑 Actual Behavior

Encountering an error when executing my_project/main.py, indicating a problem with recognizing the process variable, which is essential for some Node.js modules:

Traceback (most recent call last):
  File "/my_project/main.py", line 2, in <module>
    reactdom_server = pm.require('./node_modules/react-dom/server')
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/pythonmonkey/require.py", line 353, in require
    return createRequire(filename)(moduleIdentifier)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pythonmonkey.SpiderMonkeyError: Error in file evaluate, on line 4:
ReferenceError: process is not defined

ℹ️ Additional Information

PythonMonkey Installation Method:

Installed from pip

OS Platform and Distribution:

MacOS 14.2.1

Python Version (python --version):

3.11.7

PythonMonkey Version (pip show pythonmonkey):

0.3.0

[wesgarland]

Hi, @arvindavoudi!

PythonMonkey is a library which links JavaScript to Python under the hood, using a bare-minimum JS host environment (imagine the browser without the DOM). It is not a plug-in replacement for NodeJS, although it could be used to write one (we plan to do that some day).

You have at your disposal in PythonMonkey all of the JS language, plus a few extra globals that are modelled after what is available in the browser, excluding the DOM for obvious reasons

• globalThis
• XMLHttpRequest
• Event
• EventTarget
• console
• atob
• btoa

We have also implemented a CommonJS require function, since loading code from disk can be very helpful. :) You can access this require via the createRequire export from the pythonmonkey module. That library also has an eval method for evaluating JS code, and a globalThis export; these should make it possible to inject symbols into the global scope of your JS context. This require function can load either JavaScript or Python modules in CommonJS format.

PythonMonkey does not support ES modules at this time. We are hoping to forge some kind of Python and ES module equivalence in the future.

The JS context also has access to Python via the globalThis.python object. It has methods eval and exec which evaluate Python expressions or execute Python code.

It would be relatively easy to create a process module in Python. Call it process.py, and start writing code like

import sys
import os
exports['argv0'] = sys.executable
exports['pid'] = os.getpid()
exports['env'] = os.environ

Or perhaps you could find it easier to write process.js instead?

    python.exec(`
import pythonmonkey
import sys
import os
import inspect
`);

  process.argv0 = python.eval('sys.executable');
  process.argv = [ process.argv0 ].concat(python.eval('sys.argv'));
  process.exit = function pm$$processExit(exitCode) {
    if (arguments.length === 0)
      exitCode = process.exitCode;
    if (typeof exitCode !== 'number')
      exitCode = !exitCode ? 0 : parseInt(exitCode) || 1;
    python.eval('lambda x: exit(int(x))')(exitCode & 255);
  };

Note - I suspect you will need a lot more "node" to run react-dom/server than just the process module.

[arvindavoudi]

Thanks for your help, wesgarland. Your guidance was invaluable since there was no rich official documentation available.

Upon a thorough examination of React and ReactDOM's source code, I identified the prevalent use of process.env.ENV which was the only cause of problem. So I managed to integrate process.env.NODE_ENV as below. This is the content of my_project/main.py file:

import pythonmonkey as pm
pm.globalThis['process']  =  {'env':  {'NODE_ENV':  None}}

react = pm.require('react') # No Issue
reactdom_server = pm.require('react-dom/server') # Where we encountered errors

Note: The None value was selected after testing its value in a Node.js application.

But wait, we have more critical issues in PythonMonkey!!

First Thing First, React-DOM Patching Process

After using npm install to install those two packages, we have to take several simple steps to make react-dom package work in pythonMonkey.

After resolving the issue for the undefined process variable, we encounter an error like below:

pythonmonkey.SpiderMonkeyError: Error in file /opt/homebrew/lib/python3.11/site-packages/pythonmonkey/node_modules/ctx-module/ctx-module.js, on line 236:
Error: module not found -- require('stream') from /my_project/node_modules/react-dom/cjs/react-dom-server-legacy.node.development.js

It shows we need a standard library implemented in Node js named stream. But we can omit this need since simple SSR can be done without it and we don't have any choice until PythonMonkey gain these Node JS standard libraries over time.

Note

There is a package named stream in npm. This package presents itself as:
Ported straight from the [Node.js core] and adapted to [component/emitter]'s api.
During my attempts to use npm version of stream instead of the Node.js' standard library version, I encountered errors that couldn't be resolved even after requiring window and setting it to Null. After investigating the issue, I realized it was a duplicate of #issue243. In the end, I decided that it would be best to abandon the stream implementation, as it didn't seem likely to work out in a short time.

ReactDOM Patching Has These Steps:

1- In /my_project directory, run command npm install text-encoding.

2- Then open /my_project/node_modules/react-dom/cjs/react-dom-server-legacy.node.development.js and make these changes to drop the need of stream module:

// Delete line 18 which is
var stream = require('stream');

// Change line 7032 from
}(stream.Readable);

// To
};  // Remove "(stream.Readable);"

3- Do the following changes in /my_project/node_modules/react-dom/cjs/react-dom-server.node.development.js to also drop the requirement of having Node's standard library module named util:

// Replace line 18 which is
var util = require('util');

// With
const TextEncodingPolyfill = require('text-encoding');
const util = {
	TextEncoder: TextEncodingPolyfill.TextEncoder,
	TextDecoder: TextEncodingPolyfill.TextDecoder,
};

These two patches resolve the issue of not being able to use the react-dom package in PythonMonkey due to the unavailability of Node JS standard library.

Testing Usability After Patches, But Is It Really Usable?

Now that importing react-dom package does not lead to any errors, there are four different ways for applying SSR:
1- Define function component in JS. Render component to string in JS. Print it out in JS.
2- Define function component in JS. Render component to string in JS. Print it out in Python.
3- Define function component in JS. Render component to string in Python. Print it out in Python.
4- Define function component in Python. Render component to string in Python. Print it out in Python.

1- The First Method:

# /my_project/main.py
import pythonmonkey as pm
pm.globalThis['process']  =  {'env':  {'NODE_ENV':  None}}

pm.require('./main_') # Leads to running the JS script written below

// /my_project/main_.js
const React = require('react'); // Never had any issues
const ReactDOMServer = require('react-dom/server'); // No Issues in importing Anymore

function App() {
	console.log("Component function is being run.")
	return React.createElement("h1",  {},  "Hello");
}

console.log(ReactDOMServer.renderToString(React.createElement(App)));

Output of python3 main.py command runs without errors as expected.

Component function is being run.
<h1>Hello</h1>

2- The Second Method:

# /my_project/main.py
import pythonmonkey as pm
pm.globalThis['process']  =  {'env':  {'NODE_ENV':  None}}

rendered_component = pm.require('./main_') # Leads to running the JS script written below
print(rendered_component['html_component'])

// /my_project/main_.js
const React = require('react'); // Never had any issues
const ReactDOMServer = require('react-dom/server'); // No Issues in importing Anymore

function App() {
	console.log("Component function is being run.")
	return React.createElement("h1",  {},  "Hello");
}

exports['html_component'] = ReactDOMServer.renderToString(React.createElement(App));

The output generated by this code is also perfectly fine:

Component function is being run.
<h1>Hello</h1>

3- The Third Method:

# /my_project/main.py
import pythonmonkey as pm
pm.globalThis['process']  =  {'env':  {'NODE_ENV':  None}}

react = pm.require('react')  # Never had any issues
reactdom_server = pm.require('react-dom/server') # No Issues in importing Anymore

app_component = pm.require('./main_')
print(
	reactdom_server.renderToString(react.createElement(app_component))
)

// /my_project/main_.js
const React = require('react');

module.exports = function App() {
	console.log("Component function is being run.")
	return React.createElement("h1",  {},  "Hello");
}

In this method, I have encountered some critical issues shown by the printed message below:

zsh: segmentation fault  python3 main.py

4- The Fourth Method:

# /my_project/main.py
import pythonmonkey as pm
pm.globalThis['process']  =  {'env':  {'NODE_ENV':  None}}

react = pm.require('react')  # Never had any issues
reactdom_server = pm.require('react-dom/server') # No Issues in importing Anymore

def app_component():
	return react.createElement("h1", {}, "Hello")

print(
	reactdom_server.renderToString(react.createElement(app_component))
)

Similar to the previous method, this approach can also lead to critical errors. I have encountered two different types of errors, which I will list below. They don't follow a certain pattern and I randomly see both of them:

zsh: segmentation fault  python3 main.py

zsh: bus error  python3 main.py

I suspect that the issue may be caused by calling a JavaScript function inside a Python function and then returning that Pythonic function to a JavaScript function. Although I couldn't find any other evidence of this happening, this seems to be the most likely logic behind the problem.

I would appreciate it if you could check it out. Additionally, if you want, I can upload the entire my_project directory to a GitHub repo to make it easier for you to replicate the fault/error.

[philippedistributive]

@arvindavoudi Hi, can you try again with our latest release?
If you still experience crashes, can you provide the stack trace?