Warning! phrase-based model vocabularies are now different from lexical model vocabularies.

[johnml1135]

It appears to happen when there are a lot of get word graphs and a few trains:

2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677122153Z"}
2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677113643Z"}
2024-10-01 23:43:30	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:43:30.677064168Z"}
2024-10-01 23:42:48	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:46.065731712Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:46	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:26.782587856Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:27	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:08.251130903Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:09	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:07.784591941Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:08	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:04.68842351Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:05	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:42:04.441732675Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:42:05	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:48.257841234Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:49	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:40.446429763Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:41	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:41:30.164171281Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:41:30	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:40:32.77032131Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:40:33	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:39:39.581977747Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:39:40	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:57.585376561Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:58	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:46.285474743Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:47	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Trained segment pair for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:45.443995883Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:46	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:23.597854677Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:45	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:38:22.645678887Z"}
2024-10-01 23:38:24	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Trained segment pair for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:38:22.673247098Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:23	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:38:10.071637977Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:23	
{"log":"info: Serval.Machine.Shared.Services.ClearMLAuthenticationService[0]\n      ClearML Authentication Token Refresh Successful.\n","stream":"stdout","time":"2024-10-02T03:38:01.354824092Z","type":"info","source":"Serval.Machine.Shared.Services.ClearMLAuthenticationService"}
2024-10-01 23:38:22	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:38:22.645633643Z"}
2024-10-01 23:38:10	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:37:56.767955145Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:38:01	
{"log":"Warning! phrase-based model vocabularies are now different from lexical model vocabularies.\n","stream":"stderr","time":"2024-10-02T03:37:06.37522718Z"}
2024-10-01 23:37:57	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bc2c59ec21e6eb9f9ae91c\n","stream":"stdout","time":"2024-10-02T03:37:54.862996674Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:55	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:50.677922355Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:51	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:42.061922543Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}
2024-10-01 23:37:42	
{"log":"info: Serval.Translation.Controllers.TranslationEnginesController[0]\n      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7\n","stream":"stdout","time":"2024-10-02T03:37:32.483666598Z","type":"info","source":"Serval.Translation.Controllers.TranslationEnginesController"}

[ddaspit]

This makes me concerned that access to the model isn't being synchronized properly. Are we getting any expired locks errors?

[johnml1135]

Here is the context for the first error:

2024-09-29 21:59:14	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:58:42	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:58:16	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:57:39	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:57:29	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Trained segment pair for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:57:28	
machine-engine <> Warning! phrase-based model vocabularies are now different from lexical model vocabularies.
2024-09-29 21:50:00	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7
2024-09-29 21:48:32	
serval-api <info> info: Serval.Translation.Controllers.TranslationEnginesController[0]
      Got word graph for engine 66bcbbd6ec21e6eb9f9ae9e7

[johnml1135]

Let's kick the can unless it shows back up again.