Can't run lightgbm with config file

[JohnM5]

Hello,
Thank you for providing an example with the zillow competition. I tried running the example but came with exactly the same problem as here: #16

I can't run lgbm and I have exactly the same output error:

Starting cross validation
Fitting model : 0
Exception in thread "Thread-1" java.lang.IllegalStateException: failed to create LIGHTgbm subprocess with config name /Users/hadoop/StackNet/models/ucsbmggugdanr6qc19a64a7sv10.conf
at ml.lightgbm.LightgbmRegressor.create_light_suprocess(LightgbmRegressor.java:426)
at ml.lightgbm.LightgbmRegressor.fit(LightgbmRegressor.java:1885)
at ml.lightgbm.LightgbmRegressor.run(LightgbmRegressor.java:516)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.IllegalStateException: Tree is not fitted
at ml.Bagging.scoringhelperbagv2.(scoringhelperbagv2.java:109)
at ml.Bagging.BaggingRegressor.predict2d(BaggingRegressor.java:669)
at ml.Bagging.BaggingRegressor.predict_proba(BaggingRegressor.java:1875)
at ml.stacknet.StackNetRegressor.fit(StackNetRegressor.java:3065)
at stacknetrun.runstacknet.main(runstacknet.java:522)
... 5 more

I tried to run lightgbm by itself by running it with the config file:

./lightgbm config=~ ./models/ucsbmggugdanr6qc19a64a7sv10.conf task=train

But I receive a 'Permission denied' error. I tried running the same command with sudo but I get

sudo: ./lightgbm: command not found

I checked and the jar file is in the same folder as the lib folder. Here is a screenshot of how my Stacknet folder is organized:

Thank you for your help.

[kaz-Anova]

Hey @JohnM5

If you run in linux, you may need to

cd lib/
cd linux/
cd lightgbm/
chmod +x lightgbm

the lightgbm files, please read this :

https://github.com/kaz-Anova/StackNet#install-lightgbm

sudo won't work

Let me know if this solves the issue.

Same applies to xgboost and fast_rgf

[JohnM5]

Thank you for your answer but I'm running it in Mac Osx, does it still apply to me?

[kaz-Anova]

I think yes.

[JohnM5]

Thank you, the previous problems are solved but now I have another exception at the 4th tree:

Fitting model : 4
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.lang.IllegalStateException:  Tree is not fitted
	at ml.Bagging.scoringhelperbagv2.<init>(scoringhelperbagv2.java:109)
	at ml.Bagging.BaggingRegressor.predict2d(BaggingRegressor.java:669)
	at ml.Bagging.BaggingRegressor.predict_proba(BaggingRegressor.java:1875)
	at ml.stacknet.StackNetRegressor.fit(StackNetRegressor.java:3065)
	at stacknetrun.runstacknet.main(runstacknet.java:522)
	... 5 more

This seems to be the same issue as here #11 , but I checked and the test file is specified in my command.

Here is my command:

java -Xmx12048m -jar StackNet.jar train task=regression sparse=true has_head=false output_name=datasettwo model=model2 pred_file=pred2.csv train_file=dataset2_train.txt test_file=dataset2_test.txt test_target=false params=dataset2_params.txt verbose=true threads=1 metric=mae stackdata=false seed=1 folds=4 bins=3

[kaz-Anova]

This is the xgboost model. You need to repeat the changing of permissions for both xgboost and fast_rgf as explained above. You need to chmod +x the executables. Ensure they run from the command line properly.

if it still fails. open the dataset2_params.txt and change the model in row 5 from verbose:false to verbose:true to see what is going on. and print what it says.

[JohnM5]

Yes I repeated the changing of permissions for all models.
After putting verbose to true here is the addition error information that I get:

dyld: Library not loaded: /usr/local/opt/gcc/lib/gcc/6/libgomp.1.dylib
Referenced from: /Users/hadoop/StackNet/lib/mac/xg/xgboost
Reason: image not found

Did you see this ever before? Thank you

[kaz-Anova]

Yes I have. I am afraid you need to install xgboost manually (as it will take care of the gcc stuff) and drop the executables there. Alternatively , you can remove the xgboost models.

You may find this helpful for installing xgboost in mac.

[JohnM5]

Ok thank you I'll try this ! The weird thing is that xgboost works perfectly when I use an Ipython notebook...

[kaz-Anova]

if you find your executables (search by name) and you drop them there it should work.

[JohnM5]

Sorry for the late reply, first I could copy and paste one file in an other folder but after I had to paste one file in a /usr/lib folder in which you cant paste file unless you reboot your mac I'm new to mac os x and Im afraid that I will lose my files., so I'm not sure if I'll do it. Does xgboost add a lot of accuracy?

Apparently you can edit some folders in mac os x since version 10.11

I would be open to any suggestion :)

So in the end I disabled xgboost and the whole thing took about 1H40 mins.

Thank you for your help!

[kaz-Anova]

I am also not a mac expert and actually a friend of mine compiles the mac files for me ... Hopefully even without xgboost it stills scores OK.

[Firenze11]

Hi @JohnM5 , I had the same error. I realized that my gcc version is 7 and I only had /usr/local/opt/gcc/lib/gcc/7/libgomp.1.dylib. So I created a directory "6" under /usr/local/opt/gcc/lib/gcc/ , and copied everything under "7" to "6". No permission issue happened. Now things are working fine.

[JohnM5]

Hi @Firenze11 , thank you for your answer, I did this in the beginning but then I was given this error:

dyld: Symbol not found: __ZNSt11regex_errorD1Ev
  Referenced from: /Users/hadoop/StackNet/lib/mac/xg/xgboost
  Expected in: /usr/lib/libstdc++.6.dylib

I know where libstdc++.6.dylib is, I can copy it but I can't paste it to /usr/lib

[goldentom42]

Hi @JohnM5, why can't you copy the file to /usr/lib ?
If this is a permission issue then you need to use
sudo cp libstdc++.6.dylib /usr/lib/
and enter the root password

[JohnM5]

Hi @goldentom42 , I tried sudo but you cant since a recent os x update, now some folders are protected, https://apple.stackexchange.com/questions/208478/how-do-i-disable-system-integrity-protection-sip-aka-rootless-on-os-x-10-11 If I want to disable this feature, I need to reboot my laptop and since I'm new to os x i'm not sure If this will erase all of my data.

[YangWenfeng]

@JohnM5 $ cd /usr/local/opt/gcc/lib/gcc; ln -s 7/ 6