Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Train and testing quickstart Indian model fails in interactive shell #218

Open
lewismc opened this issue Oct 2, 2015 · 2 comments
Open

Train and testing quickstart Indian model fails in interactive shell #218

lewismc opened this issue Oct 2, 2015 · 2 comments

Comments

@lewismc
Copy link
Contributor

lewismc commented Oct 2, 2015

I am unable to follow the quickstart when attempting to verify that Joshua is installed and operating correctly from within the brew Formula.
Logging output below details the commands and output I am getting.
Note that this is after a successful ant build with all dependencies installed.

bash-3.2$ ln -s indian-parallel-corpora-1.0 input
bash-3.2$ $JOSHUA/bin/pipeline.pl --source bn --target en --no-prepare --aligner berkeley --type hiero --corpus input/bn-en/tok/training.bn-en --tune input/bn-en/tok/dev.bn-en --test input/bn-en/tok/devtest.bn-en
[source-numlines] rebuilding...
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn [CHANGED]
  cmd=cat /private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn | wc -l
  took 0 seconds (0s)
[source-numlines] retrieved cached result =>    20788
[berkeley-aligner-chunk-0] rebuilding...
  dep=alignments/0/word-align.conf [CHANGED]
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/data/train/splits/corpus.bn.0 [CHANGED]
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/data/train/splits/corpus.en.0 [CHANGED]
  dep=alignments/0/training.align [NOT FOUND]
  cmd=java -d64 -Xmx10g -jar /private/tmp/joshua-v20151002-2032-qnt9ko/lib/berkeleyaligner.jar ++alignments/0/word-align.conf
  took 270 seconds (4m30s)
[aligner-combine] rebuilding...
  dep=alignments/0/training.align [CHANGED]
  dep=alignments/training.align [NOT FOUND]
  cmd=cat alignments/0/training.align > alignments/training.align
  took 0 seconds (0s)
[thrax-input-file] rebuilding...
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn [CHANGED]
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en [CHANGED]
  dep=alignments/training.align [CHANGED]
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/data/train/thrax-input-file [NOT FOUND]
  cmd=/private/tmp/joshua-v20151002-2032-qnt9ko/scripts/training/paste /private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.bn /private/tmp/joshua-v20151002-2032-qnt9ko/indian-parallel-corpora-1.0/bn-en/tok/training.bn-en.en alignments/training.align | perl -pe 's/\t/ ||| /g' | grep -v '()' | grep -v '||| \+$' > /private/tmp/joshua-v20151002-2032-qnt9ko/data/train/thrax-input-file
  took 0 seconds (0s)
[thrax-prep] rebuilding...
  dep=/private/tmp/joshua-v20151002-2032-qnt9ko/data/train/thrax-input-file [CHANGED]
  dep=grammar.gz [NOT FOUND]
  cmd=hadoop/bin/hadoop fs -rm -r pipeline-bn-en-hiero-_private_tmp_joshua-v20151002-2032-qnt9ko; hadoop/bin/hadoop fs -mkdir pipeline-bn-en-hiero-_private_tmp_joshua-v20151002-2032-qnt9ko; hadoop/bin/hadoop fs -put /private/tmp/joshua-v20151002-2032-qnt9ko/data/train/thrax-input-file pipeline-bn-en-hiero-_private_tmp_joshua-v20151002-2032-qnt9ko/input-file
  JOB FAILED (return code 1)
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
15/10/02 14:39:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
rm: Call From localhost/127.0.0.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
15/10/02 14:39:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
mkdir: Call From localhost/127.0.0.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
log4j:WARN Failed to set property [port] to value "".
log4j:ERROR Could not connect to remote log4j server at [localhost]. We will try again later.
15/10/02 14:39:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: Call From localhost/127.0.0.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
@mjpost
Copy link
Member

mjpost commented Oct 3, 2015

It looks like a problem with your Hadoop cluster. is $HADOOP set? If so, it should point to your Hadoop installation. If not, Joshua will roll out its own for you.

@lewismc
Copy link
Contributor Author

lewismc commented Oct 5, 2015

I am assuming that the user does not have a cluster of Hadoop machines.

is $HADOOP set?

No, I am assuming that the brew Formula should work regardless of whether we have a $HADOOP env set.

If so, it should point to your Hadoop installation. If not, Joshua will roll out its own for you.

It appears Joshua is not doing this. I'm working on this right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mjpost @lewismc