You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have been having the following issue while running Crail-Spark-TeraSort. It looks like an access permission problem in cachepath and datapath that occurs inside MappedBufferCache's constructor. This is probably because I am using Cloudera Hadoop, and it uses other usernames such as "yarn", "hdfs" for YARN and HDFS.
...
WARN scheduler.DAGScheduler: Creating new stage failed due to exception - job: 0 java.lang.NullPointerException
at com.ibm.crail.memory.MappedBufferCache.(MappedBufferCache.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
...
I was running Crail-TeraSort with my own username. But "yarn" is also creating some cache file as below. So essentially the files are created by a different user than the one trying to read them.
ls -lth /memory/cache/
drwxrwxrwx 2 yarn hadoop 4.0K Dec 6 17:06 1512608812456
drwxrwxrwx 2 yarn hadoop 4.0K Dec 5 11:43 1512503023326
I was wondering if there is a way to permanently fix it? For example, does it help to change the access permission when creating RandomAccessFile for MappedBufferCache in the datapath?
RandomAccessFile dataFile = new RandomAccessFile(dataFilePath, "rw"); // (MappedBufferCache.java:89)
Thanks,
Kevin
The text was updated successfully, but these errors were encountered:
I have been having the following issue while running Crail-Spark-TeraSort. It looks like an access permission problem in cachepath and datapath that occurs inside MappedBufferCache's constructor. This is probably because I am using Cloudera Hadoop, and it uses other usernames such as "yarn", "hdfs" for YARN and HDFS.
...
WARN scheduler.DAGScheduler: Creating new stage failed due to exception - job: 0
java.lang.NullPointerException
at com.ibm.crail.memory.MappedBufferCache.(MappedBufferCache.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
...
I was running Crail-TeraSort with my own username. But "yarn" is also creating some cache file as below. So essentially the files are created by a different user than the one trying to read them.
ls -lth /memory/cache/
drwxrwxrwx 2 yarn hadoop 4.0K Dec 6 17:06 1512608812456
drwxrwxrwx 2 yarn hadoop 4.0K Dec 5 11:43 1512503023326
I was wondering if there is a way to permanently fix it? For example, does it help to change the access permission when creating RandomAccessFile for MappedBufferCache in the datapath?
RandomAccessFile dataFile = new RandomAccessFile(dataFilePath, "rw"); // (MappedBufferCache.java:89)
Thanks,
Kevin
The text was updated successfully, but these errors were encountered: