Typically, the default memory allowance for the Java Virtual Machine (JVM) is too low. The memory requirements for Web Apollo will depend on many variables, but in general, we recommend at least 512g for the maximum memory and 512m for the minimum, though a 2 GB maximum seems to be optimal for most server configurations.
export CATALINA_OPTS="-Xms512m -Xmx2g \
-XX:+CMSClassUnloadingEnabled \
-XX:+CMSPermGenSweepingEnabled \
-XX:+UseConcMarkSweepGC"
In cases where the assembled genome is highly fragmented, additional tuning of memory requirements and garbage collection will be necessary to maintain the system stable. Below is an example from a research group that maintains over 40 Apollo instances with assemblies that range from 1,000 to 150,000 scaffolds (reference sequences) and over one hundred users:
export CATALINA_OPTS="-Xmx12288m -Xms8192m \
-XX:ReservedCodeCacheSize=64m \
-XX:+UseG1GC \
-XX:+CMSClassUnloadingEnabled \
-Xloggc:$CATALINA_HOME/logs/gc.log \
-XX:+PrintHeapAtGC \
-XX:+PrintGCDetails \
-XX:+PrintGCTimeStamps"
To change your settings, you can usually edit the setenv.sh script in $TOMCAT_BIN_DIR/setenv.sh where
$TOMCAT_BIN_DIR is the directory where the Tomcat binaries reside. It is possible that this file doesn't exist by default, but it will be picked up when Tomcat restarts. Make sure that tomcat can read the file.
In most cases, creating the setenv.sh should be sufficient but you may have to edit a catalina.sh or another file directly depending on your system and tomcat setup. For example, on Ubuntu, the file /etc/default/tomcat7 often contains these settings.
Your CATALINA_OPTS settings from setenv.sh can be confirmed with a tool like jvisualvm or via the command line with the
ps tool. e.g. ps -ef | grep java should yield something like the following allowing you to confirm that your memory settings have been picked up.
root 9848 1 0 Oct22 ? 00:36:44 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/current/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1g -Xmx2g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -Dj
If you start seeing memory leaks (java.lang.OutOfMemoryError: Java heap space) after doing an update, you might try
re-installing, as the live re-deploy itself can cause memory leaks or an inconsistent software state.
If you have named your web application named Apollo.war then you can remove all of these files from your webapps
directory and re-deploy.
- Run
apollo deploy - Undeploy any existing Apollo instances
- Stop tomcat
- Copy the war file to the webapps folder
- Start tomcat
Preferably, when running Apollo or any webserver, you should not run Tomcat as root. Therefore, when deploying your war file to tomcat or another web application server, you may need to tune your file permissions to make sure Tomcat is able to access your files.
On many production systems, tomcat will typically belong to a user and group called something like 'tomcat'. Make sure that the 'tomcat' user can read your "webapps" directory (where you placed your war file) and write into the annotations and any other relevant directory (e.g. tomcat/logs). As such, it is sometimes helpful to add the user you logged-in as to the same group as your tomcat user and set group write permissions for both.
Consider using a package manager to install Tomcat so that proper security settings are installed, or to use the jsvc http://tomcat.apache.org/tomcat-7.0-doc/security-howto.html#Non-Tomcat_settings
JBrowse tools don't show up in bin directory (or install at all) after install or typing install_jbrowse.sh
If the bin directory with JBrowse tools doesn't show up after calling install_jbrowse.sh JBrowse is having trouble installing itself for a few possible reasons. If these do not work, please observe the JBrowse troubleshooting and JBrowse install pages, as well and the setup.log file created during the installation process.
Make sure the appropriate JBrowse libraries are installed on your system.
If you see chmod: cannot access `web-app/jbrowse/bin/cpanm': No such file or directory make sure to install cpanm.
Git expects to clone a single branch which is supported in git 1.7.10 and greater. The output when that fails looks something like this:
Buildfile: build.xml
copy.apollo.plugin.webapp:
setup-jbrowse:
git.clone:
[exec] Result: 129
The solution is to upgrade git to 1.7.10 or greater or remove the line with the --single-branch option in build.xml.
If you are behind a firewall, checking out code using the git:// protocol may not be allowed, but that is the default. The output will look something like this:
setup-jbrowse:
git.clone:
[exec] Submodule 'src/FileSaver' (git://github.com/dkasenberg/FileSaver.js.git) registered for path 'src/FileSaver'
[exec] Submodule 'src/dbind' (git://github.com/rbuels/dbind.git) registered for path 'src/dbind'
. . . .
[exec] Submodule 'src/xstyle' (git://github.com/kriszyp/xstyle.git) registered for path 'src/xstyle'
[exec] Result: 1
with possibly more output below.
Type:
git config --global url."https://".insteadOf git://
in the command-line and then re-install using ./apollo clean-all ./apollo run-local (or deploy).
If you are trying to run the jbrowse binaries but get these sorts of errors, try running install_jbrowse.sh which will
initialize as many pre-requisites as possible including JBLibs and other JBrowse dependencies.
You can manually clear jbrowse files from web-app/jbrowse and re-run apollo deploy to rebuild JBrowse.
Apollo2 does fairly strict JSON validation so make sure your trackList.json file is valid JSON
If you still get this error after validating please forward the issue to our github issue tracker.
Please check that you don't already have a tomcat running netstat -tan | grep 8080. Sometimes tomcat does not exit
properly. ps -ef | grep java and then kill -9 the offending processing.
Note that you can also configure tomcat to run on different ports, or you can launch a temporary instance of apollo with
apollo run-local 8085 for example to avoid the port conflict.
If you receive an error similar to this:
SEVERE: Unable to create initial connections of pool.
org.h2.jdbc.JdbcSQLException: Error opening database:
"Could not save properties /var/lib/tomcat7/prodDb.lock.db" [8000-176]
Then this is due to the production server trying to write an h2 instance in an area it doesn't have permissions to. If you use H2 (which is great for testing or single-user user, but not for full-blown production) make sure that:
You can modify the specified data directory for the H2 database in the apollo-config.groovy. For example, using the /tmp/ directory, or some other directory:
url = "jdbc:h2:/tmp/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"
This will write a H2 db file to /tmp/prodDB.db. If you don't specify an absolute path it will try to write in the
same directory that tomcat is running in e.g., /var/lib/tomcat7/ which can have permission issues.
More detail on database configuration when specifying the apollo-config.groovy file is available in the
setup guide.
In some instances you can't write to the default cache location on disk. Part of an example config log:
2015-07-03 14:37:39,675 [main] ERROR context.GrailsContextLoaderListener - Error initializing the application: null
java.lang.NullPointerException
at grails.plugin.cache.ehcache.GrailsEhCacheManagerFactoryBean$ReloadableCacheManager.rebuild(GrailsEhCacheManagerFactoryBean.java:171)
at grails.plugin.cache.ehcache.EhcacheConfigLoader.reload(EhcacheConfigLoader.groovy:63)
at grails.plugin.cache.ConfigLoader.reload(ConfigLoader.groovy:42)
There are several solutions to this, but all involve updating the apollo-config.groovy file to override the caching
defined in the Config.groovy.
grails.cache.config = {
cache {
enabled = false
name 'globalcache'
}
}
This can also be done by removing the plugin. In grails-app/conf/BuildConfig remove / comment out the line and re-building:
compile ':cache-ehcache:1.0.5'
Can be used for small instances
grails.cache.config = {
// avoid ehcache naming conflict to run multiple WA instances
provider {
name "ehcache-apollo-"+(new Date().format("yyyyMMddHHmmss"))
}
cache {
enabled = true
name 'globalcache'
eternal false
overflowToDisk false // THIS IS THE IMPORTANT LINE
maxElementsInMemory 100000
}
}
Best for high load servers, which will need the cache. Make sure your tomcat / web-server user can write to that directory:
// copy from Config.groovy except where noted
grails.cache.config = {
...
cache {
...
maxElementsOnDisk 10000000
// this is the important part, below!
diskStore{
path '/opt/apollo/cache-directory'
}
}
...
}
When JSON is added to the URL string (e.g., addStores and addTracks) you may get this error with newer patched versions of Tomcat 7.0.73, 8.0.39, 8.5.7:
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
To fix these, the best solution we've come up with (and there may be many) is to explicitly allow these characters, which you can do starting with Tomcat versions: 7.0.76, 8.0.42, 8.5.12.
This is done by adding the following line to $CATALINA_HOME/conf/catalina.properties:
tomcat.util.http.parser.HttpParser.requestTargetAllow=|{}
Information on the grails ehcache plugin (see "Overriding values") and ehcache itself.
If you get an Unsupported major.minor error or similar, please confirm that the version of java that tomcat is
running ps -ef | grep java is the same as the one you used to build. Setting JAVA_HOME to the Java 8 JDK should fix most problems.
For certain version of MySQL we might get errors of this nature:
SQLException occurred when processing request: [GET] /apollo/annotator/getAppState Value '0000-00-00 00:00:00' can not be represented as java.sql.Timestamp. Stacktrace follows: java.sql.SQLException: Value '0000-00-00 00:00:00' can not be represented as java.sql.Timestamp
The fix is to set the zeroDateTimeBehavior=convertToNull to the url connect screen. Originally identified here. Here is an example URL:
jdbc:mysql://localhost/apollo_production?zeroDateTimeBehavior=convertToNull&autoReconnect=true&characterEncoding=UTF-8&characterSetResults=UTF-8