Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to start on Windows #18

Open
cgbystrom opened this issue Sep 10, 2013 · 5 comments
Open

Unable to start on Windows #18

cgbystrom opened this issue Sep 10, 2013 · 5 comments

Comments

@cgbystrom
Copy link

Looks like Windows like to append a colon to the file path, which blow things up.
Google seem to suggest this is a known Hadoop issue.

took 1 seconds to read name_index.hfile
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file://2013-06-08%5Cprefix_indexdata
        at org.apache.hadoop.fs.Path.initialize(Path.java:157)
        at org.apache.hadoop.fs.Path.<init>(Path.java:80)
        at org.apache.hadoop.fs.Path.<init>(Path.java:58)
        at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:422)
        at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:396)
        at com.foursquare.twofishes.MemoryMappedMapFileReader.<init>(MapFileUtils.scala:37)
        at com.foursquare.twofishes.MapFileUtils$.readerAndInfoFromLocalPath(MapFileUtils.scala:59)
        at com.foursquare.twofishes.MapFileInput$$anonfun$6.apply(HFileStorageService.scala:185)
        at com.foursquare.twofishes.MapFileInput$$anonfun$6.apply(HFileStorageService.scala:185)
        at com.twitter.util.Duration$.inMilliseconds(Time.scala:346)
        at com.foursquare.twofishes.MapFileInput.<init>(HFileStorageService.scala:184)
        at com.foursquare.twofishes.PrefixIndexMapFileInput.<init>(HFileStorageService.scala:231)
        at com.foursquare.twofishes.PrefixIndexMapFileInput$.readInput(HFileStorageService.scala:223)
        at com.foursquare.twofishes.NameIndexHFileInput.<init>(HFileStorageService.scala:203)
        at com.foursquare.twofishes.HFileStorageService.<init>(HFileStorageService.scala:23)
        at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:310)
        at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:306)
        at com.foursquare.twofishes.GeocodeFinagleServer$.main(GeocodeServer.scala:364)
        at com.foursquare.twofishes.GeocodeFinagleServer.main(GeocodeServer.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file://2013-06-08%5Cprefix_indexdata
        at java.net.URI.checkPath(Unknown Source)
        at java.net.URI.<init>(Unknown Source)
        at org.apache.hadoop.fs.Path.initialize(Path.java:154)
        ... 18 more
@blackmad
Copy link
Contributor

do you have a link to someone else talking about this issue? don't have a windows box to test it on.

@gustavo-momente
Copy link

Hello, any followup on this issue ? I can reproduce the problem in Windows 7
I'm using Java:

java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)

And I by running
java -jar server-assembly-0.84.9.jar --hfile_basepath 2015-03-05-20-05-30.753698

I get the following output:

12:57:49.241 [main] INFO  c.f.twofishes.GeocodeFinagleServer$ - starting version 0.84.9
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
12:57:58.005 [main] INFO  com.foursquare.twofishes.HFileInput - took 2 seconds to read name_index.hfile
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5Cgmom%5CDesktop%5CTwoFishes%5C2015-03-05-20-05-30.753698%5Cprefix_indexdata
    at org.apache.hadoop.fs.Path.initialize(Path.java:157)
    at org.apache.hadoop.fs.Path.<init>(Path.java:80)
    at org.apache.hadoop.fs.Path.<init>(Path.java:58)
    at org.apache.hadoop.io.MapFileConcurrentReader.open(MapFileConcurrentReader.java:95)
    at org.apache.hadoop.io.MapFileConcurrentReader.<init>(MapFileConcurrentReader.java:87)
    at com.foursquare.twofishes.MemoryMappedMapFileReader.<init>(MapFileUtils.scala:38)
    at com.foursquare.twofishes.MapFileUtils$.readerAndInfoFromLocalPath(MapFileUtils.scala:61)
    at com.foursquare.twofishes.MapFileInput$$anonfun$5.apply(HFileStorageService.scala:179)
    at com.foursquare.twofishes.MapFileInput$$anonfun$5.apply(HFileStorageService.scala:179)
    at com.twitter.util.Duration$.inMilliseconds(Duration.scala:160)
    at com.foursquare.twofishes.MapFileInput.<init>(HFileStorageService.scala:178)
    at com.foursquare.twofishes.PrefixIndexMapFileInput.<init>(HFileStorageService.scala:243)
    at com.foursquare.twofishes.PrefixIndexMapFileInput$.readInput(HFileStorageService.scala:235)
    at com.foursquare.twofishes.NameIndexHFileInput.<init>(HFileStorageService.scala:211)
    at com.foursquare.twofishes.HFileStorageService.<init>(HFileStorageService.scala:24)
    at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:468)
    at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:464)
    at com.foursquare.twofishes.GeocodeFinagleServer$.main(GeocodeServer.scala:494)
    at com.foursquare.twofishes.GeocodeFinagleServer.main(GeocodeServer.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5Cgmom%5CDesktop%5CTwoFishes%5C2015-03-05-20-05-30.753698%5Cprefix_indexdata
    at java.net.URI.checkPath(Unknown Source)
    at java.net.URI.<init>(Unknown Source)
    at org.apache.hadoop.fs.Path.initialize(Path.java:154)
    ... 18 more

Any ideas of what may be causing the bug, or any debug I should be doing ? Also, could you confirm which Hadoop version is being used ?

@rahulpratapm
Copy link
Contributor

According to
http://stackoverflow.com/questions/25334604/hadoop-java-net-urisyntaxexception-relative-path-in-absolute-uri-rsrchbase-co,
the problem might be that Hadoop's Path doesn't like the : in the file name.

Not sure how you'd workaround it but it must be possible since I imagine
most Windows users would run into this.

Rahul.

On Mon, Jul 27, 2015 at 7:20 AM, gustavo-momente [email protected]
wrote:

Hello, any followup on this issue ? I can reproduce the problem in Windows
7
I'm using Java:

java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)

And I by running
java -jar server-assembly-0.84.9.jar --hfile_basepath
2015-03-05-20-05-30.753698

I get the following output:

12:57:49.241 [main] INFO c.f.twofishes.GeocodeFinagleServer$ - starting version 0.84.9
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
12:57:58.005 [main] INFO com.foursquare.twofishes.HFileInput - took 2 seconds to read name_index.hfile
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5Cgmom%5CDesktop%5CTwoFishes%5C2015-03-05-20-05-30.753698%5Cprefix_indexdata
at org.apache.hadoop.fs.Path.initialize(Path.java:157)
at org.apache.hadoop.fs.Path.(Path.java:80)
at org.apache.hadoop.fs.Path.(Path.java:58)
at org.apache.hadoop.io.MapFileConcurrentReader.open(MapFileConcurrentReader.java:95)
at org.apache.hadoop.io.MapFileConcurrentReader.(MapFileConcurrentReader.java:87)
at com.foursquare.twofishes.MemoryMappedMapFileReader.(MapFileUtils.scala:38)
at com.foursquare.twofishes.MapFileUtils$.readerAndInfoFromLocalPath(MapFileUtils.scala:61)
at com.foursquare.twofishes.MapFileInput$$anonfun$5.apply(HFileStorageService.scala:179)
at com.foursquare.twofishes.MapFileInput$$anonfun$5.apply(HFileStorageService.scala:179)
at com.twitter.util.Duration$.inMilliseconds(Duration.scala:160)
at com.foursquare.twofishes.MapFileInput.(HFileStorageService.scala:178)
at com.foursquare.twofishes.PrefixIndexMapFileInput.(HFileStorageService.scala:243)
at com.foursquare.twofishes.PrefixIndexMapFileInput$.readInput(HFileStorageService.scala:235)
at com.foursquare.twofishes.NameIndexHFileInput.(HFileStorageService.scala:211)
at com.foursquare.twofishes.HFileStorageService.(HFileStorageService.scala:24)
at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:468)
at com.foursquare.twofishes.ServerStore$.getStore(GeocodeServer.scala:464)
at com.foursquare.twofishes.GeocodeFinagleServer$.main(GeocodeServer.scala:494)
at com.foursquare.twofishes.GeocodeFinagleServer.main(GeocodeServer.scala)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file://C:%5CUsers%5Cgmom%5CDesktop%5CTwoFishes%5C2015-03-05-20-05-30.753698%5Cprefix_indexdata
at java.net.URI.checkPath(Unknown Source)
at java.net.URI.(Unknown Source)
at org.apache.hadoop.fs.Path.initialize(Path.java:154)
... 18 more

Any ideas of what may be causing the bug, or any debug I should be doing ?
Also, could you confirm which Hadoop version is being used ?

—
Reply to this email directly or view it on GitHub
#18 (comment)
.

@gustavo-momente
Copy link

Thanks for the answer Rahul, It all seems to point to that issue, nevertheless I can't find anything about how to correctly escape the colon. Could you point me to the file that handles the loading in twofishes ?

@rahulpratapm
Copy link
Contributor

I believe the call to getCanonicalPath in server/src/main/scala/HFileStorageService.scala:21 is the problem:

val basepath = new File(originalBasepath).getCanonicalPath()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@cgbystrom @blackmad @rahulpratapm @gustavo-momente