Sometimes malformed json with constrained decoding

[nivibilla]

When using a regex to constrain the output to a particular json format, sometimes after the comma it will randomly fail and do new lines.

Like

{\n
    "tool" : "search",\n
    "query" : "something",\n
\n
\n
\n
...
}

[keepitsane]

I am also running into a similar issue when using the pydantic method for json decoding.

[nivibilla]

dottxt-ai/outlines#691

This issue seems to be talked about here too, they suggest

In the build_json_from_object try adding whitespace_pattern="" or whitespace_pattern="[ \n\t]?"

Haven't tested though

[comaniac]

It's likely due to the regex issue. I'd suggest the following ways:

• Use a regex string directly instead of pydantic. If there are issues, you could post the regex here for help.
• If you really need to use pydantic/outlines, manually print the regex string and try to understand what it does. As my personal experience, the regex string generated by pydantic/outlines is not always correct or robust.

[nivibilla]

@comaniac thanks for your insight. It's interesting that the pydantic doesn't give perfect regex. Will keep that in mind. But I think also the solution mentioned in that other issue might be useful to test as well.

[nivibilla]

Could you convert this to a discussion since it's probably not an sglang issue