Spectator is slow as compared to the ng-bullet

[sozakir]

I'm submitting a...

[ * ] Performance issue

Current behavior

I want to incorporate spectator in my test suit of around ~1500 unit tests and are increasing.

I have earlier used ng-bullet, with karma parallel in order to speed up my tests.

I have recently upgraded my app to Angular 9. I found out this issue topnotch48/ng-bullet-workspace#8. I am worried about the performance of my tests suit in the long run. Therefore I want to switch to spectator.

But performance is my concern here. I have used playground code and converted it to ng-bullet. I found the code with ng-bullet is faster as compared to spectator. My assumption is that if I completely switched by test suit to spectator it would slow down my current test suite.

Am I doing something wrong here or spectator main concern is to reduce the boiler plate rather than reducing the time?
I clearly sees spectator is taking more time.

Expected behavior

Minimal reproduction of the problem with instructions

Here sample stackblitz code with ng-bullet
https://stackblitz.com/edit/angular-testing-spectator-enm3yo?file=app/hello.component.spec.ts

Code using spectator
https://stackblitz.com/edit/angular-testing-spectator

I have also tested this on my proprietary company code, I have replaced the ng-bullet code with spectator and the time increased for both the component and the service. I can create a sample project and replicate it but I am busy with my own work. As the author of ng-bullet pointed out spectator is using the same method for speeding up the test and provide additional benefits. But after finding the time results are otherwise.

Any comments @NetanelBasal ??

Environment

Angular version: 9

Browser:
- [ *] Chrome (desktop) Version 85.0.4183.121 (Official Build) (64-bit)

 
For Tooling issues:
- Node version: 12.14.1
- Platform:  Windows 10

Others:

[ntvsx193]

same time, see:

[sozakir]

Hi @ntvsx193 , I will create a demo app and show you my finding by the weekend.

[benelliott]

Am I doing something wrong here or spectator main concern is to reduce the boiler plate rather than reducing the time?

You're right, Spectator isn't designed to speed up unit tests like ng-bullet is. It's just meant to provide a more approachable testing API than what you get out of the box with Angular.

That said, there have been some attempts to integrate ng-bullet: #89 and #100

The PR was eventually closed because it didn't seem to speed things up sufficiently, and Ivy was around the corner.

So I'll close this as a duplicate of #89 unless you believe the situation has changed considerably since then?

[sozakir]

@benelliott I am busy last week and i have seen in my app using Ivy, I am able to decrease the angular test execution time when I am using ng-bullet. Can you re-open this issue?.

I think of creating a demo app by weekend and try to use ng-bullet code in the spectator on my local and check if it fixes my issue.

[johncrim]

You might try switching to Jest (from karma) - it's generally regarded as more performant than launching a browser and running your tests in a browser (we use Jest and spectator). I don't have benchmarks, but it worked well for this guy:
https://dev.to/dylanwatsonsoftware/make-your-angular-tests-1000-faster-by-switching-from-karma-to-jest-1n33

The downside of Jest is it uses jsdom, which doesn't have all the functionality of a real browser. So another option is making browser dependent tests use karma, and more generic tests use Jest.

The biggest issues with using Jest right now (IMO) are:

• Initial setup is kind of a hassle
• jest-preset-angular doesn't yet support using the Ivy viewengine for tests, so the old compiler is used and jest can't yet benefit from the ivy perf improvements

[sozakir]

@johncrim can we use jest and jasmine together in the same project?. Could shared some info regarding your number of tests and their total execution time?

[johncrim]

Jest has a very jasmine like test library, though I'm not sure if that's what you're asking. I use jest and (karma+jasmine) in the same project, just using different file extensions (I use .karma.spec for karma tests, and .spec for Jest tests).

I have 365 tests across 75 test suites in 8 projects, which takes 74s (just now). That's Jest only, we have some karma tests on top of that.

Like I said, that should speed up further once jest-preset-angular has ivy support.

[NetanelBasal]

It'll be great if someone can check this optimization - angular/angular#31834.

[ssuperczynski]

@NetanelBasal

MacBook Pro
Proc: 2,2 GHz 6-Core Intel Core i7
Mem: 16 GB 2400 MHz DDR4

Angular: 10.0.8
Test runner: [email protected]

Results (40 components fully use Spectator):
Test Suites: 59 total
Tests: 1369 total

Without fix:
Time: 191.889 s

With fix: getTestBed().inject(ɵDomSharedStylesHost).ngOnDestroy();
Time: 192.708 s

With fix: window.document.querySelectorAll("style").forEach((style: HTMLStyleElement) => style.remove());
Time: 185.946 s

And small heads up, without fix but by adding:

import { cacheTestingModule } from 'ng-cache-testing-module';
describe() => {
  ...
  cacheTestingModule();
  ...
}

Time: 55.272 s

[NetanelBasal]

@ssuperczynski, based on your information, it doesn't have much effect. I'm not familiar with ng-cache-testing-module, and I don't know if it can bring side effects to existing code.

[ntvsx193]

@NetanelBasal

MacBook Pro
Proc: 2,2 GHz 6-Core Intel Core i7
Mem: 16 GB 2400 MHz DDR4

Angular: 10.0.8
Test runner: [email protected]

Results (40 components fully use Spectator):
Test Suites: 59 total
Tests: 1369 total

Without fix:
Time: 191.889 s

With fix: getTestBed().inject(ɵDomSharedStylesHost).ngOnDestroy();
Time: 192.708 s

With fix: window.document.querySelectorAll("style").forEach((style: HTMLStyleElement) => style.remove());
Time: 185.946 s

And small heads up, without fix but by adding:

import { cacheTestingModule } from 'ng-cache-testing-module';
describe() => {
  ...
  cacheTestingModule();
  ...
}

Time: 55.272 s

Why are u use jest it performance with fix remove styles?
Jest run workaround in clearable always.
This fix relevant only for karma + jasmine where it has shared workaround.

[Lonli-Lokli]

I have this test (pretty simple) and it executes for 27seconds. Why?

describe('ApiInfoComponent', () => {
  let spectator: Spectator<ApiInfoComponent>;
  const createComponent = createComponentFactory({
    component: ApiInfoComponent,
    providers: [API_PROVIDERS],
    imports: [HttpClientTestingModule],
    mocks: [...commonRootServicesToMock],
    declareComponent: false,
    detectChanges: false
  });

  beforeEach(() => (spectator = createComponent()));

  test('should create', () => {
    expect(spectator).toBeTruthy();
  });
});

component itself is standalone one

@Component({
  selector: 'about-api-info',
  standalone: true,
  templateUrl: './api-info.component.html',
  styleUrls: ['./api-info.component.scss'],
  imports: [
    RemoteDataComponent,
    IfSuccessDirective,
    LetDirective,
    AfterViewInitDirective,
    JsonPipe,
    AsyncPipe,
    NgFor,
    KeyValuePipe
  ],
  providers: [ApiInfoService, InfoService],
  changeDetection: ChangeDetectionStrategy.OnPush
})
export class ApiInfoComponent extends BaseComponent {
  private svc: ApiInfoService = inject(ApiInfoService);
  public info$ = this.svc.$info;
  public buildInfo = inject(BuildSettings);

  public load() {
    this.svc.loadInfo();
  }
}