Skip to content

AppScale/hawkeye

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

hawkeye

API fidelity test suite for AppScale

prerequisites

The geckodriver executable must be available on your current path.

Start up appscale as usual:

appscale up

Get Hawkeye from git:

git clone [email protected]:AppScale/hawkeye.git

compile the Java version(s) as indicated in java-app/README.md and deploy the Java, Java 8, and Python 2.7 versions:

appscale deploy hawkeye/java-app/module-main/war
appscale deploy hawkeye/java8-app/module-main/war
appscale deploy hawkeye/python27-app/module-main

if you're going to test modules suite you also need to deploy additional python module:

appscale deploy hawkeye/java-app/module-a/war
appscale deploy hawkeye/java8-app/module-a/war
appscale deploy hawkeye/python27-app/module-a

for warmup tests you should also deploy:

appscale deploy hawkeye/java-app/module-warmup/war
appscale deploy hawkeye/java8-app/module-warmup/war
appscale deploy hawkeye/python27-app/module-warmup

To determine ports for deployed applications use:

appscale status

run hawkeye

Go into the test-suite directory:

cd test-suite

Create csv files containing app modules and versions info:

cat > versions-java.csv << APP_VERSIONS
MODULE,VERSION,HTTP-URL,HTTPS-URL,IS-DEFAULT
default,1,http://192.168.33.10:8080,https://192.168.33.10:4380,yes
module-a,1,http://192.168.33.10:8081,https://192.168.33.10:4381,yes
warmup,1,http://192.168.33.10:8082,https://192.168.33.10:4382,yes
APP_VERSIONS

Using versions-java.csv, versions-java8.csv, and versions-python27.csv for the different runtimes.

and run it:

python hawkeye.py --app APP_ID --versions-csv FILE --lang LANG --baseline

For this example, running Hawkeye Java would be:

python hawkeye.py --app hawkeyejava --versions-csv versions-java.csv --lang java --baseline

and Java 8 would be:

python hawkeye.py --app hawkeyejava8 --versions-csv versions-java8.csv --lang java --baseline --baseline-file hawkeye_baseline_java8.csv

and Python 2.7 would be:

python hawkeye.py --app hawkeyepython27 --versions-csv versions-python27.csv --lang python --baseline

hawkeye output

You'll see output like the following:

[user@host test-suite]$ python hawkeye.py --app myapp --versions-csv ./versions-example.csv --lang python  --baseline

User API Test Suite
===================
.FFE
----------------------------------------------------------------------
Ran 4 tests in 0.625s

FAILED (failures=2, errors=1)

XMPP Test Suite
===============
.
----------------------------------------------------------------------
Ran 1 test in 1.343s

OK

Blobstore Test Suite
====================
..........
----------------------------------------------------------------------
Ran 10 tests in 9.346s

OK

Cron Test Suite
===============
F
----------------------------------------------------------------------
Ran 1 test in 123.424s

FAILED (failures=1)

Modules API Test Suite
======================
.....
----------------------------------------------------------------------
Ran 5 tests in 6.431s

OK

Memcache Test Suite
===================
............
----------------------------------------------------------------------
Ran 12 tests in 13.017s

OK

Images Test Suite
=================
.........
----------------------------------------------------------------------
Ran 9 tests in 6.478s

OK

Datastore Test Suite
====================
.....F......FF.....F.FF.....F..FF.
----------------------------------------------------------------------
Ran 34 tests in 90.639s

FAILED (failures=9)

Asynchronous Datastore Test Suite
=================================
......F......FF.....F
----------------------------------------------------------------------
Ran 21 tests in 13.949s

FAILED (failures=4)

Secure URL Test Suite
=====================
FFFF
----------------------------------------------------------------------
Ran 4 tests in 1.497s

FAILED (failures=4)

Logservice Test Suite
=====================
F
----------------------------------------------------------------------
Ran 1 test in 0.525s

FAILED (failures=1)

NDB Test Suite
==============
........FFF....
----------------------------------------------------------------------
Ran 15 tests in 8.251s

FAILED (failures=3)

Task Queue Test Suite
=====================
.....F...F...
----------------------------------------------------------------------
Ran 13 tests in 82.349s

FAILED (failures=2)

Config Environment Variable Test Suite
======================================
..
----------------------------------------------------------------------
Ran 2 tests in 1.319s

OK

Comparison to baseline:
 97  tests matched baseline result
 23  tests did not match baseline result
    tests.async_datastore_tests.CompositeQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.GQLProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.OrderedKindAncestorQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.ProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.cron_tests.CronTest.runTest ... FAIL (ok was expected)
    tests.datastore_tests.CompositeMultiple.runTest ... FAIL (ok was expected)
    tests.datastore_tests.CompositeProjection.runTest ... FAIL (ok was expected)
    tests.datastore_tests.CompositeQueryTest.runTest ... FAIL (ok was expected)
    tests.datastore_tests.CursorQueries.runTest ... FAIL (ok was expected)
    tests.datastore_tests.GQLProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.datastore_tests.MultipleEqualityFilters.runTest ... FAIL (ok was expected)
    tests.datastore_tests.OrderedKindAncestorQueryTest.runTest ... FAIL (ok was expected)
    tests.datastore_tests.ProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.logservice_tests.FetchLogTest.runTest ... FAIL (ok was expected)
    tests.ndb_tests.NDBCompositeQueryTest.runTest ... FAIL (ok was expected)
    tests.ndb_tests.NDBGQLTest.runTest ... FAIL (ok was expected)
    tests.ndb_tests.NDBProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.secure_url_tests.AlwaysSecureRegexTest.runTest ... FAIL (ok was expected)
    tests.secure_url_tests.AlwaysSecureTest.runTest ... FAIL (ok was expected)
    tests.secure_url_tests.NeverSecureRegexTest.runTest ... FAIL (ok was expected)
    tests.secure_url_tests.NeverSecureTest.runTest ... FAIL (ok was expected)
    tests.taskqueue_tests.LeaseModificationTest.runTest ... FAIL (ok was expected)
    tests.taskqueue_tests.RESTPullQueueTest.runTest ... FAIL (ok was expected)
 11  tests ran, but not found in baseline
    tests.datastore_tests.IndexVersatility.runTest ... FAIL
    tests.datastore_tests.LongTxRead.runTest ... ok
    tests.modules_tests.TestCreatingAndGettingEntity.test_default_module ... ok
    tests.modules_tests.TestCreatingAndGettingEntity.test_module_a ... ok
    tests.modules_tests.TestTaskTargets.test_task_targets ... ok
    tests.modules_tests.TestVersionDetails.test_default_module ... ok
    tests.modules_tests.TestVersionDetails.test_module_a ... ok
    tests.taskqueue_tests.TaskEtaTest.runTest ... ok
    tests.user_tests.AdminLoginTest.runTest ... FAIL
    tests.user_tests.LogoutURLTest.runTest ... ERROR
    tests.user_tests.UserLoginTest.runTest ... FAIL
 1   tests in baseline, but not ran
    tests.images_tests.ImageUploadTest.runTest ... ok

The only important part here is following:

 97  tests matched baseline result
 23  tests did not match baseline result
    tests.async_datastore_tests.CompositeQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.GQLProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.OrderedKindAncestorQueryTest.runTest ... FAIL (ok was expected)
    tests.async_datastore_tests.ProjectionQueryTest.runTest ... FAIL (ok was expected)
    tests.cron_tests.CronTest.runTest ... FAIL (ok was expected)
    ...

If you run into any of the baseline tests failing, you should definitely dig into these logs:

  • AppScale logs (run "appscale logs somedirectory" to grab AppScale logs and put them in somedirectory).
  • Hawkeye logs, located in "hawkeye/test-suite/hawkeye-logs"