Create new types of tests

[mmaryo]

Hello,

Currently, the web-framework benchmark tests an empty endpoint.

However, web frameworks are about more than just request delay; they are mainly about processing complex tasks like JSON serialization, object composition, DB requests, etc.

Have you ever considered enriching the benchmark with new types of tests? It could provide a more comprehensive evaluation of web frameworks.

Imagine the following endpoints:
/ - the current behavior, we check the response time
/external-service - we call external services (a function that does a sleep)
/serialization - we send and receive a large JSON body in POST request with few query parameters
/http-client - we test the standard HTTP client and load a large file hosted in the server itself
/web - we do complete web scenarios with JSON in requests, DB requests, data transformations, and large JSON in response

We also deprecated the endpoints :
/user/{id}
/user
Because they do not bring value compared to the / endpoint

The current benchmark does not need to be updated. But each new piece of code that is not a framework update must implement that logic.

[waghanza]

Hey,

Thanks for this proposal.

Seems to go in a direction that I'm confortable with

I suggest slight enhancement.

• /user/0 -> instead of /
  • the idea is to test routing condition
  • and the response
• the database used for the /web endpoint should be in-memory and local and the same across all framework (standard like sqlite)
  • better to avoid network latency and else that could mess the results
  • better to avoid persistency that could mess the reuslts

[cyrusmsk]

Can you make please a message with description of new endpoints and the plan when they will be enabled in testing?

[waghanza]

Not sure to get you @cyrusmsk

[cyrusmsk]

As I understood you want to introduce new endpoints for testing? With sleep with json maybe with SQLite.
I’m wondering about more detailed description and the timeline of incorporation of these new tests

[waghanza]

The idea is to make new tests, so as this project could be meaningul. But we plan to do this incrementally, no timeline defined at this time

[cyrusmsk]

The idea is to make new tests, so as this project could be meaningul. But we plan to do this incrementally, no timeline defined at this time

In that case maybe new filter field will be helpful. Something like "Old tests" - "New tests". So in the result will be possible to see different implementations without any breaking changes.

[waghanza]

You mean add this on UI

If you make somes PRs why not ... Just a joke

I'm more on the line of having a new UI when all tests are ready, and in the meantime we can just make the backend code step by step

[waghanza]

We are going to write new test (add some endpoints)

• /, check the routing part of a framework
• /external-service, check how frameworks call external services (database, authentication api ...)
• /serialization, do some computation with few query parameters analysis
• /http-client, how the framework loads a big file
• /web - all scenarios with a real db (and json response)

I plan to use a real sqlite database

• convient for me to create a database as a file
• without the burden of setting up a database
• no noise about network latency during database communication
• but their is IO operation (due to file database)
• however all frameworks will be implemented like that, not sure if this is so terrible do use a local database

any thought @whiplash @cyrusmsk @ahopkins @appleboy @dalisoft @sy-records @System-Glitch @pimbrouwers @panesofglass @0xTim @leocavalcante @lukaszlenart ?

[waghanza]

cc @mario-huang

[System-Glitch]

Many frameworks don't have anything related to the features you want to test. It would essentially be testing the standard library. I'm not sure those would be very valuable. I thinking mainly of:

• how the framework loads a big file
• do some computation with few query parameters analysis

[dalisoft]

@waghanza

I think /external-service is not necessary as this may different at different cases. And maybe even out-of-scope for this project. Database performance is not web-server task (i could be wrong, fix me if it is)
But these kind of tests are interesting for me:

• /static - Static endpoint like {URL}/health
• /foo/:bar - Dynamic endpoint to parse parameter
• /foo/query?bar=1 - Static endpoint with query
• /upload/json - JSON deserialize/parse performance of server or it's middleware
• /serialization - JSON serialize performance of server or it's middleware
• /upload/BIG_FILE.mp4 - File upload performance

At best, if those tests could be run at same LAN/network but over 10GbE network not at same machine as running server

Here another benchmark alternative to TechEmpower and this repo - https://github.com/SaltyAom/bun-http-framework-benchmark

[waghanza]

so for you @System-Glitch

• /serialization, do some computation with few query parameters analysis
• /http-client, how the framework loads a big file
  are not necessary since not in all frameworks

I'll argue that this is what we have mainly in server-side API, thus necessary to implement.
Also, if certain frameworks could not have those feature (which I doubt, since we could implement it it any language), we should just drop them of

---

for you @dalisoft /external-service is not necessary, but I can argue that some API could use any external service (the first one is a distant db)

[cyrusmsk]

Do all frameworks will require to implement all tests? or they can skip some of them to be presented in the table?
Will the frontend allow to see separate results of the tests or only combined result of all tests?

[waghanza]

Do all frameworks will require to implement all tests ?

Yes, the new ones

Will the frontend allow to see separate results of the tests or only combined result of all tests?

Not thought about it. But could be intersting to have an customer choice to pick 1 endpoint or all

[cyrusmsk]

@dalisoft @System-Glitch I think the project should decide on the scope..
I don't have heavy knowledge in this, but what I found when I've made some simple research it is possible to say that there are web-servers and web-frameworks. And web-frameworks could be micro and full-stack.
It seems there is no established terminology - like when we can say it is just a web-server or it is already a micro web-framework or when we can say that the solution is not micro framework anymore but already full stack (sometimes general purpose) web-framework.

The name of the repo is "web-frameworks" so probably it should be more than web-server functionality, but maybe not necessary "full stack" frameworks. Especially most of the presented solutions in the repo are web-servers/micro frameworks.

So there could be 2 types of tests for micro and full stack frameworks and 2 tables.

@waghanza also recently I watched the video (https://www.youtube.com/watch?v=qB422eeOFr8) and we can leverage on the ideas from this test. One of the nice idea is the size of the headers in request. Sending small request for Hello world - not really relevant to real use cases. Maybe only in some micro-service architecture (but probably they will use something like gPRC in that case).

[waghanza]

Hi @cyrusmsk ,

Thanks for your ideas.

If I summarize their is web server things and web framework things, and you suggest to split in two tables

• I can not disagree, but with above endpoints we can test both, just a question of how to present results
• We can test web servers (just routing parts for example) and web framework (with serialization)
• The idea is, but in the future, to have some kind of UI displaying that, probably display results by endpoints

Finally, I think that in order not to be only for tech plumers, we need to use the generic term of web framework. I agree that their is micro/full (and even nano, for some kind), but this is more for experts.

The ultimate goal of this projects is to help choosing your stack (here server-side) when migrating an existing tool / crafting a new product.

Having this distingtion, even if it exists, will lead to present complex results to analyze by a non tech-expert. The idea is to show the average of all endpoints by default, and allow fine customization (bu this is for the UI part, in step2).

The question is : in a context of a server-side api, will it be relevant to use a real local sqlite database in here ?

PS : I'll start to implement this in a few weeks if relevant

[waghanza]

Also, with the simplest scenario we have, we can see that go is not automatically better (in terms of performance) than bun
https://web-frameworks-benchmark.netlify.app/result?f=0http-bun,hono-bun,bunicorn,blaze-bun,vixeny-bun&l=go

but the idea to have those endpoints is to have more reallistic workload, and I think this order will change 😛

[cyrusmsk]

Hi @cyrusmsk ,

Thanks for your ideas.

The question is : in a context of a server-side api, will it be relevant to use a real local sqlite database in here ?

PS : I'll start to implement this in a few weeks if relevant

I think local sqlite could be fine. Or maybe even NoSQL solution that could be used to store some cache data.

I think another questions is to send not only 1 request many times, but also have an ability to send requests from the list of available one. Something line '/users/user_data={1,2,3,4}' or even send some data in JSON format as a payload. Which is quite common scenario for APIs.

[waghanza]

Good idea for multiple, but maybe in a second time (to huge to change so many things right now)

Go local sqlite then (or any other tool)