Development setup failing with module not found errors

[zxy994]

Bug Report

Steps to Reproduce:

###Fork and clone repository locally.

1. Followed the Developer guide as written at =>https://perspective.finos.org/docs/development/

2. Installed necessary dependencies
   => Running on WSL Ubuntu
   No LSB modules are available.
   Distributor ID: Ubuntu
   Description: Ubuntu 22.04.2 LTS
   Release: 22.04
   Codename: jammy

=> cmake version 3.22.1
=> #define BOOST_LIB_VERSION "1_71"
=> wasm-pack 0.10.3

yarn/stable,stable,now 1.22.19-1 all [installed]
cmake-data/jammy-updates,now 3.22.1-1ubuntu1.22.04.1 all [installed,automatic]
cmake/jammy-updates,now 3.22.1-1ubuntu1.22.04.1 amd64 [installed]
libboost-iostreams1.74.0/jammy,now 1.74.0-14ubuntu3 amd64 [installed,automatic]
libboost-regex1.74.0/jammy,now 1.74.0-14ubuntu3 amd64 [installed,automatic]
libbpf0/jammy-updates,jammy-security,now 1:0.5.0-1ubuntu22.04.1 amd64 [installed,automatic]
libbrotli1/jammy,now 1.0.9-2build6 amd64 [installed,automatic]
libbsd0/jammy,now 0.11.5-1 amd64 [installed,automatic]
libbz2-1.0/jammy,now 1.0.8-5build1 amd64 [installed,automatic]
libc-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
libc-dev-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
libc-devtools/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
g++-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu122.04 amd64 [installed,automatic]
g++/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed]
gawk/jammy,now 1:5.1.0-1build3 amd64 [installed,automatic]
gcc-11-base/jammy-updates,jammy-security,now 11.3.0-1ubuntu122.04 amd64 [installed,automatic]
gcc-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu122.04 amd64 [installed,automatic]
gcc-12-base/jammy-updates,jammy-security,now 12.1.0-2ubuntu122.04 amd64 [installed,automatic]
gcc/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed,automatic]
gdb/jammy-updates,now 12.1-0ubuntu1~22.04 amd64 [installed]

3. run yarn install

Result:

yarn install
yarn install v1.22.19
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ node scripts/install_emsdk.js
Setting up EMSDK environment (suppress these messages with EMSDK_QUIET=1)
Setting environment variables:
PATH = /p/perspective/.emsdk:/p/perspective/.emsdk/upstream/emscripten:/p/perspective/.emsdk/node/14.18.2_64bit/bin:/tmp/yarn--1678699404956-0.23660272833921225:/p/perspective/node_modules/.bin:/.config/yarn/link/node_modules/.bin:/.yarn/bin:/.nvm/versions/node/v18.15.0/libexec/lib/node_modules/npm/bin/node-gyp-bin:/.nvm/versions/node/v18.15.0/lib/node_modules/npm/bin/node-gyp-bin:/.nvm/versions/node/v18.15.0/bin/node_modules/npm/bin/node-gyp-bin:/.nvm/versions/node/v18.15.0/bin:/miniconda3/bin:/miniconda3/condabin:~/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 3.1.14 (4343cbec72b7db283ea3bda1adc6cb1811ae9a73)
Copyright (C) 2014 the Emscripten authors (see AUTHORS.txt)
This is free and open source software under the MIT license.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ husky install
husky - Git hooks installed
Done in 3.06s.

4. run yarn build

Result:

yarn build
yarn run v1.22.19
$ node scripts/build.js
Python 3.10.9
running build
running build_py
running egg_info
writing perspective_python.egg-info/PKG-INFO
writing dependency_links to perspective_python.egg-info/dependency_links.txt
writing requirements to perspective_python.egg-info/requires.txt
writing top-level names to perspective_python.egg-info/top_level.txt
reading manifest file 'perspective_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.bumpversion.cfg'
warning: no files found matching 'Makefile'
warning: manifest_maker: MANIFEST.in, line 10: 'recursive-include' expects

...

warning: no previously-included files found matching 'docs/#'
no previously-included directories found matching 'docs/build'
no previously-included directories found matching 'docs/gh-pages'
no previously-included directories found matching 'docs/dist'
no previously-included directories found matching 'bench'
warning: no directories found matching 'dist/test'
warning: no previously-included files matching '~' found anywhere in distribution
warning: no previously-included files matching '.pyc' found anywhere in distribution
warning: no previously-included files matching '.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'perspective_python.egg-info/SOURCES.txt'
~l/miniconda3/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'perspective.include.perspective' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################
Python recognizes 'perspective.include.perspective' as an importable package,
but it is not listed in the `packages` configuration of setuptools.

'perspective.include.perspective' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).

Please make sure that 'perspective.include.perspective' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).

You can read more about "package discovery" and "data files" on setuptools
documentation page.

!!

check.warn(importable)
~/miniconda3/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'perspective.include.perspective.python' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################
Python recognizes 'perspective.include.perspective.python' as an importable package,
but it is not listed in the `packages` configuration of setuptools.

'perspective.include.perspective.python' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).

Please make sure that 'perspective.include.perspective.python' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).

You can read more about "package discovery" and "data files" on setuptools
documentation page.

!!

check.warn(importable)
/home/pearl/miniconda3/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'perspective.src' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################
Python recognizes 'perspective.src' as an importable package,
but it is not listed in the `packages` configuration of setuptools.

'perspective.src' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).

Please make sure that 'perspective.src' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).

You can read more about "package discovery" and "data files" on setuptools
documentation page.

!!

check.warn(importable)
/home/pearl/miniconda3/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'perspective.tests.single_threaded' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################
Python recognizes 'perspective.tests.single_threaded' as an importable package,
but it is not listed in the `packages` configuration of setuptools.

'perspective.tests.single_threaded' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).

Please make sure that 'perspective.tests.single_threaded' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).

You can read more about "package discovery" and "data files" on setuptools
documentation page.

!!

check.warn(importable)
/home/pearl/miniconda3/lib/python3.10/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning: Installing 'perspective.tests.table.arrow' as data is deprecated, please list it in packages.
!!

############################
# Package would be ignored #
############################
Python recognizes 'perspective.tests.table.arrow' as an importable package,
but it is not listed in the `packages` configuration of setuptools.

'perspective.tests.table.arrow' has been automatically added to the distribution only
because it may contain data files, but this behavior is likely to change
in future versions of setuptools (and therefore is considered deprecated).

Please make sure that 'perspective.tests.table.arrow' is included as a package by using
the `packages` configuration field or the proper discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).

You can read more about "package discovery" and "data files" on setuptools
documentation page.

!!

check.warn(importable)
running build_ext
-- Found PythonInterp: ~/miniconda3/bin/python3.10 (found suitable exact version "3.10.9")
-- pybind11 v2.8.1
Building minimal Apache Arrow
CMake Deprecation Warning at ~/p/perspective/python/perspective/build/temp.linux-x86_64-cpython-310/rapidjson-src/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- No Doxygen found. Documentation won't be built
CMake Deprecation Warning at ~/p/perspective/python/perspective/build/temp.linux-x86_64-cpython-310/rapidjson-src/example/CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

CMake Deprecation Warning at ~/p/perspective/python/perspective/build/temp.linux-x86_64-cpython-310/rapidjson-src/thirdparty/gtest/googletest/CMakeLists.txt:48 (cmake_minimum_required):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

Update the VERSION argument value or use a ... suffix to tell
CMake that the project does not need compatibility with older versions.

-- Found PythonInterp: ~/miniconda3/bin/python3.10 (found version "3.10.9")
-- Proceeding with version: 2.0.5.v2.0.5
-- CMAKE_CXX_FLAGS: -std=c++0x -Wall -pedantic -Werror -Wextra -Werror=shadow -faligned-new -Werror=implicit-fallthrough=2 -Wunused-result -Werror=unused-result -Wunused-parameter -Werror=unused-parameter -fsigned-char
-- Found PythonInterp: ~/miniconda3/bin/python3.10 (found suitable version "3.10.9", minimum required is "3")
-- tests/monster_test.fbs: add generation of C++ code with '--no-includes;--gen-compare'
-- tests/monster_test.fbs: add generation of binary (.bfbs) schema
-- tests/namespace_test/namespace_test1.fbs: add generation of C++ code with '--no-includes;--gen-compare;--gen-name-strings'
-- tests/namespace_test/namespace_test2.fbs: add generation of C++ code with '--no-includes;--gen-compare;--gen-name-strings'
-- tests/union_vector/union_vector.fbs: add generation of C++ code with '--no-includes;--gen-compare;--gen-name-strings'
-- tests/optional_scalars.fbs: add generation of C++ code with '--no-includes;--gen-compare'
-- tests/native_type_test.fbs: add generation of C++ code with ''
-- tests/arrays_test.fbs: add generation of C++ code with '--scoped-enums;--gen-compare'
-- tests/arrays_test.fbs: add generation of binary (.bfbs) schema
-- tests/monster_test.fbs: add generation of C++ embedded binary schema code with '--no-includes;--gen-compare'
-- tests/monster_extra.fbs: add generation of C++ code with '--no-includes;--gen-compare'
-- samples/monster.fbs: add generation of C++ code with '--no-includes;--gen-compare'
-- samples/monster.fbs: add generation of binary (.bfbs) schema

Skipping WASM binding
Building C++ binding
Building Python 3.10 binding
Building C++ without strict warnings
Building RELEASE

-- Configuring done
-- Generating done
-- Build files have been written to: ~/p/perspective/python/perspective/build/temp.linux-x86_64-cpython-310
[ 0%] Built target arrow_fb_files
Consolidate compiler generated dependencies of target re2
[ 10%] Built target re2
Consolidate compiler generated dependencies of target arrow
[ 62%] Built target arrow
Consolidate compiler generated dependencies of target psp
[ 96%] Built target psp
Consolidate compiler generated dependencies of target binding
[100%] Built target binding

Done in 16.87s.

5. yarn start blocks

RESULT:

yarn start blocks
yarn run v1.22.19
$ npm run start --workspace blocks

[email protected] start
mkdirp dist && node server.js

node:internal/modules/cjs/loader:571
throw e;
^

Error: Cannot find module '/p/perspective/node_modules/@finos/perspective/dist/cjs/perspective.node.js'
at createEsmNotFoundErr (node:internal/modules/cjs/loader:1096:15)
at finalizeEsmResolution (node:internal/modules/cjs/loader:1089:15)
at resolveExports (node:internal/modules/cjs/loader:565:14)
at Module._findPath (node:internal/modules/cjs/loader:634:31)
at Module._resolveFilename (node:internal/modules/cjs/loader:1061:27)
at Module._load (node:internal/modules/cjs/loader:920:27)
at Module.require (node:internal/modules/cjs/loader:1141:19)
at require (node:internal/modules/cjs/helpers:110:18)
at Object. (/p/perspective/examples/blocks/server.js:11:29)
at Module._compile (node:internal/modules/cjs/loader:1254:14) {
code: 'MODULE_NOT_FOUND',
path: '~/p/perspective/node_modules/@finos/perspective/package.json'
}

Node.js v18.15.0
npm ERR! Lifecycle script start failed with error:
npm ERR! Error: command failed
npm ERR! in workspace: [email protected]
npm ERR! at location: /home/pearl/p/perspective/examples/blocks
error Command failed with exit code 1.

6. run jupyter labextension list

RESULT:

JupyterLab v3.6.1
~/miniconda3/share/jupyter/labextensions
@jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
app dir: ~/miniconda3/share/jupyter/lab
@finos/perspective v1.9.4 enabled OK
@finos/perspective-jupyterlab v1.9.4 enabled OK
@finos/perspective-viewer v1.9.4 enabled OK
@finos/perspective-viewer-d3fc v1.9.4 enabled OK
@finos/perspective-viewer-datagrid v1.9.4 enabled OK
@finos/perspective-viewer-openlayers v1.9.4 enabled OK

linked packages:
@finos/perspective: ~/p/perspective/packages/perspective
@finos/perspective-jupyterlab: ~/p/perspective/packages/perspective-jupyterlab
@finos/perspective-viewer: ~/p/perspective/rust/perspective-viewer
@finos/perspective-viewer-d3fc: ~/p/perspective/packages/perspective-viewer-d3fc
@finos/perspective-viewer-datagrid: ~/p/perspective/packages/perspective-viewer-datagrid
@finos/perspective-viewer-openlayers: ~/p/perspective/packages/perspective-viewer-openlayers

The following source extensions are overshadowed by older prebuilt extensions:
@jupyter-widgets/jupyterlab-manager

Both yarn start with examples package, and having perspective run in jupyter lab despite being linked to local repository not working. Any help in resolving this?

[timkpaine]

There is no bug here, you only built the python side of the project but you're attempting to use the JavaScript side. If you want to use the JavaScript assets, you also need to build the JavaScript side of the project. You can update your perspectiverc file locally, or rely on yarn build_python and yarn build_js

[zxy994]

Right! Thanks, this answer works and I am able to run "yarn start blocks"
###RESULT:

###In regards to the second issue, Jupyter lab not finding perspective module persists.

1) I run jupyter lab clean
###RESULT:
jupyter lab clean
[LabCleanApp] Cleaning /home/pearl/miniconda3/share/jupyter/lab...
[LabCleanApp] Removing staging...
[LabCleanApp] Success!

2) I run yarn jlab_link
###RESULT:
yarn jlab_link
yarn run v1.22.19
$ node scripts/jlab_link.js
Installing /home/pearl/p/perspective/packages/perspective as a linked package because it does not have extension metadata:
No jupyterlab key
Installing /home/pearl/p/perspective/rust/perspective-viewer as a linked package because it does not have extension metadata:
No jupyterlab key
Installing /home/pearl/p/perspective/packages/perspective-viewer-datagrid as a linked package because it does not have extension metadata:
No jupyterlab key
Installing /home/pearl/p/perspective/packages/perspective-viewer-d3fc as a linked package because it does not have extension metadata:
No jupyterlab key
Installing /home/pearl/p/perspective/packages/perspective-viewer-openlayers as a linked package because it does not have extension metadata:
No jupyterlab key
Installing /home/pearl/p/perspective/packages/perspective-jupyterlab as a linked package because it does not have extension metadata:
Missing extension module "dist/umd/perspective-jupyterlab.js"
Building jupyterlab assets (development)
Jupyterlab should now have all changes from the current working directory. To pick up new changes, run yarn build and jupyter lab build.
Done in 321.95s.

3. I run yarn build and jupyter lab build

4)jupyter labextension list
###RESULT:
JupyterLab v3.6.1
/home/pearl/miniconda3/share/jupyter/labextensions
@jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
app dir: /home/pearl/miniconda3/share/jupyter/lab
@finos/perspective v1.9.4 enabled OK
@finos/perspective-jupyterlab v1.9.4 enabled OK
@finos/perspective-viewer v1.9.4 enabled OK
@finos/perspective-viewer-d3fc v1.9.4 enabled OK
@finos/perspective-viewer-datagrid v1.9.4 enabled OK
@finos/perspective-viewer-openlayers v1.9.4 enabled OK

linked packages:
@finos/perspective: /home/pearl/p/perspective/packages/perspective
@finos/perspective-jupyterlab: /home/pearl/p/perspective/packages/perspective-jupyterlab
@finos/perspective-viewer: /home/pearl/p/perspective/rust/perspective-viewer
@finos/perspective-viewer-d3fc: /home/pearl/p/perspective/packages/perspective-viewer-d3fc
@finos/perspective-viewer-datagrid: /home/pearl/p/perspective/packages/perspective-viewer-datagrid
@finos/perspective-viewer-openlayers: /home/pearl/p/perspective/packages/perspective-viewer-openlayers

The following source extensions are overshadowed by older prebuilt extensions:
@jupyter-widgets/jupyterlab-manager

Yet this module not found persists. Any advise? Is the result for "yarn jlab_link" as expected?

[texodus]

Don't run jupyter lab build after running yarn jlab_link, it does the equivalent of npm install in Jupyter's staging directory removing the linked packages.

[zxy994]

Thanks! Based on your answer I tried the following

1) Rerun yarn jlab_link , and did not run jupyter lab build after
###RESULT:
import perspective

ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 1
----> 1 import perspective

ModuleNotFoundError: No module named 'perspective'

2)Run yarn unlink in all perspective packages where yarn link was registered.

3)Uninstalled extensions using jupyter labextension uninstall

4)Update conda

5) Rebuild jupyter lab

6) Use command in doc , jupyter labextension install ./packages/perspective-jupyterlab
###RESULT:
(Deprecated) Installing extensions with the jupyter labextension install command is now deprecated and will be removed in a future major version of JupyterLab.

Users should manage prebuilt extensions with package managers like pip and conda, and extension authors are encouraged to distribute their extensions as prebuilt packages
An error occurred.
ValueError: "/home/pearl/p/perspective/packages/perspective-jupyterlab" is not a valid extension:
Missing extension module "dist/umd/perspective-jupyterlab.js"

7) Run jupyter labextension install @finos/perspective-jupyterlab
###RESULT:
Other labextensions (built into JupyterLab)
app dir: /home/pearl/miniconda3/share/jupyter/lab
@finos/perspective v1.9.4 enabled OK
@finos/perspective-jupyterlab v1.9.4 enabled OK

8) Run yarn link in all perspective packages to register links

9) run yarn jlab_link
###RESULT: Same as before
import perspective

ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 1
----> 1 import perspective

ModuleNotFoundError: No module named 'perspective'

[texodus]

You didn't install perspective-python or run add the local development area to your PYTHONPATH, you just built it locally and told Jupyter to install the NPM plugin. You actually need the locally build perspective-python available to Jupyter's python interpreter. This is pretty basic Python development stuff tho, not related to Perspective - see the Python docs on how imports work for information on how to load a local library in Jupyter Lab, and/or vritualenv for example of how to set up an environment for Python library development.

@zxy994 I'm down to help you get started but please try to do some googling before posting a wall-of-text here. If you have further questions on getting started, please start a new discussion with the specifics.