-
Notifications
You must be signed in to change notification settings - Fork 24
configuration 4.3 investigation service startup performance
The eucalyptus-cloud service is restarted during development when deploying updated software (services). The service is perceived as slow to restart so we should examine performance and make improvements accordingly.
The arbitrary startup time target is 20 seconds.
The primary concern is speed of service restart during development. The main focus should therefore be the topologies that are most commonly used for development.
Performance issues due to resources allocated for services (e.g. EC2 instances) can be ignored as the main focus is development speed and such clouds will likely not have many resources.
On a cloud with (UFS)-(CLC WS)-(SC CC)-(NC) topology times from service start to all services being reported as enabled were:
| Host components | Core time | Time (Minutes) | Notes |
|---|---|---|---|
| UFS | 50s | 4 | delay appeared to be due to startup of services with dependencies |
| CLC WS | 1 | 1.5 | |
| SC CC | 1 | 1.5 |
Core time reflects that startup prior to services.
Log extract showing where time is spent during startup (CLC):
Mon Nov 16 13:14:11 2015 Populating binding cache.
Mon Nov 16 13:14:32 2015 Bootstrap stage completed: UnprivilegedConfiguration
Mon Nov 16 13:14:32 2015 Bootstrap stage completed: SystemCredentialsInit
Mon Nov 16 13:14:33 2015 Bootstrap stage completed: RemoteConfiguration
Mon Nov 16 13:14:36 2015 Bootstrap stage completed: DatabaseInit
Mon Nov 16 13:14:51 2015 Bootstrap stage completed: UpgradeDatabase
Mon Nov 16 13:14:51 2015 Bootstrap stage completed: PoolInit
Mon Nov 16 13:15:16 2015 Bootstrap stage completed: PersistenceInit
Mon Nov 16 13:15:17 2015 Bootstrap stage completed: RemoteDbPoolInit
Mon Nov 16 13:15:17 2015 Bootstrap stage completed: SystemAccountsInit
Mon Nov 16 13:15:18 2015 Bootstrap stage completed: RemoteServicesInit
Mon Nov 16 13:15:18 2015 Bootstrap stage completed: UserCredentialsInit
Mon Nov 16 13:15:18 2015 Bootstrap stage completed: CloudServiceInit
Mon Nov 16 13:15:18 2015 Bootstrap stage completed: Final
Some components are expected to be problematic in terms of startup performance, these components were examined for issues.
| Component | Stage | Time (seconds) | Reason |
|---|---|---|---|
| Mule | Final (concurrent) | 30 | Mule initialization includes spring/DI overhead |
| Message binding generator |
_UnprivilegedConfiguration_ (pre)
| 45 (+5*) | Relevant on initial startup and when redeploying code with updated messages |
* binding generation takes 5 seconds when there is no work to do.
For service restart we should also consider shutdown performance. Initial testing showed relatively good performance on shutdown when compared to service startup times.
The most significant time is currently spent starting services that depend on the availability of other services, this should be the initial focus.
Other areas requiring startup improvement are:
- up to and including the UnprivilegedConfiguration stage
- the UpgradeDatabase stage (when not upgrading)
- the PersistenceInit stage