Skip to content

Performance testing

Jump to bottom
petrs edited this page Mar 19, 2015 · 27 revisions

Performance testing

Goals:

  • Provide precise measures of operations executed on-card (user can select card according to his performance needs)
  • Provide easy way how to upload your measured performance profile to our server and get similarity metric to other profiles already in database (user can identify origin of his "blank" unknown card)
  • Track changes in time of underlying hardware for the same type of card

General principles:

  • Operations are performed over RAM-allocated arrays (unless specifically targeted to EEPROM)
  • Time measured on PC side - therefore communication costs also included. Can be mitigated either by enough repetitions of measured operation (so communication cost is negligible) or two runs with separate number of repetitions are used and subtracted for time computations
  • Allocation of new instance of measured algorithm is NOT included in measurement (as good practice is to pre-allocate all objects during applet's install time)

What is measured

  • Time necessary to execute separate method on given length of data (e.g., AESKey.setKey()) [SINGLE_METHOD]

  • Time necessary to perform whole logical cryptographic operation on-card (e.g., transmit X bytes of data to card, encrypt it by AES and transmit back result) [FULL_OP].

    • Test scenario 1: (short packets - 5 to 32B?, key unchanged)
    • Test scenario 2: (longer packets - 128 to 256B?, key unchanged)
    • Test scenario 3: Test scenario 1 + key changed every call
    • Test scenario 4: Test scenario 2 + key changed every call
    • Test scenario 5: Custom software implementation of AES / SHA-2 (test of bytecode algorithmic performance)

  • Some operations can be measured without additional preparation of target object (e.g., AESKey.setKey()). Target method call is then iterated on-card according to TEST_SETTINGS.numRepeatWholeOperation to suppress of significance of overhead (apdu transmission etc.) on measurement

  • Some operations requires some pre-processing (e.g., Cipher.update() requires init() with proper key etc.). This preparation is done in separate command and is not included into measurement.

Type of results produced

  • Note: information below is always for fixed combination of algorithm, key length etc....
  • Some operations are independent of input data length (e.g., AESKey.setKey()) - presented result is average, min and max value [AMM].
    • Example (for given key length): AESKey.setKey() takes AVG: 16ms, MIN: 15ms, MAX: 17ms
  • Alternatively, histogram of measured values is produced for operations with high deviations of values from the average (e.g., KeyPair.genKeyPair() for RSA) [HIST].
    • Example: ALG_RSA - KeyPair.genKeyPair()
  • Some operations are dependent on input data length (e.g., Cipher.doFinal()) - presented result is graph with X-axis being data length and Y being time required (on single operation) [GDD]. For operations with short underlying blocks (e.g., AES), data lengths tested are 0-256 bytes [GDD256], for operations with bigger underlying blocks 0-512B [GDD512].
  • There is overhead associated with making call to measured method (e.g., calling update() once over 256B will take shorter then calling update() four times over 64B parts - common situation when computing MAC over data stored in different arrays). This overhead can significantly impact speed of whole operation. Graph figure is produced with varying number of method calls necessary to process given input data length (for this test, length is fixed at 256B). T_S.numRepeatSubOperation defines number of measured method invocations necessary to process this fixed amount of data. X-axis shows number of invocations. Y-axis show time required to process 256B (length of data for single operation is 256B / T_S.numRepeatSubOperation). [MULTI_INVOC]
    • Example: MessageDigest.update() with 1-10 invocations TODO: graph

Key classes

  • AESKey, DESKey...
  • Tested methods: setKey(), getKey(), clearKey()
  • Tests: SINGLE_METHOD
  • Result: AMM, HIST

Cipher class

  • ALG_DES_CBC_NOPAD, ALG_RSA_PKCS1...
  • Tested methods: init(), update(), doFinal()
  • Tests: SINGLE_METHOD, FULL_OP (encrypt packet with specified size)
  • Result: GDD256, GDD512

KeyPair class

  • ALG_RSA_CRT&LENGTH_RSA_1024...
  • Tested methods: genKeyPair(), getPublic(), getPrivate()
  • Tests: SINGLE_METHOD
  • Result: HIST

TODO: add all remaining classes with scope at least similar to http://www.fi.muni.cz/~xsvenda/jcsupport.html

Clone this wiki locally