Garnet crash on load data

[rnz]

Describe the bug

    _________
   /_||___||_\      Garnet 1.0.35 64 bit; standalone mode
   '. \   / .'      Port: 6379
     '.\ /.'        https://aka.ms/GetGarnet
       '.'

03::27::19 info: GarnetServer[0] Garnet 1.0.35 64 bit; standalone mode; Port: 6379
03::27::19 info: ArgParser[0] Configuration import from embedded resource succeeded. Path: defaults.conf.
03::27::19 info: ArgParser[0] Configuration import from local machine succeeded. Path: /etc/garnet/garnet.conf.
03::27::19 info: Options[0] [Store] Using page size of 32m
03::27::19 info: Options[0] [Store] Using log memory size of 128g, with 896 empty pages, for effective size of -0.109252929688p
03::27::19 info: Options[0] [Store] There are 4k log pages in memory
03::27::19 info: Options[0] [Store] Using disk segment size of 1g
03::27::19 info: Options[0] [Store] Using hash index size of 128m (2m cache lines)
03::27::19 info: Options[0] [Store] Hash index size is optimized for up to ~8m distinct keys
03::27::19 info: Options[0] [Store] Using log mutable percentage of 90%
03::27::19 info: Options[0] [Store] Not using Revivification
03::27::19 trce: TsavoriteKV [main][0] KV Initialize size:2097152, sizeBytes:134217728 sectorSize:512 alignedSizeBytes:134218240
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Using page size of 1m
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Each page can hold ~43690 key-value pairs of objects
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Using log memory size of 32m
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] This can hold ~1398101 key-value pairs of objects in memory total
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] There are 32 log pages in memory
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Using disk segment size of 32m
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Using hash index size of 16m (256k cache lines)
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Hash index size is optimized for up to ~1m distinct keys
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Using log mutable percentage of 90%
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Total memory size including heap objects is unlimited
03::27::19 info: TsavoriteKV  [obj][0] [Object Store] Not using Revivification
03::27::19 trce: GarnetServer[0] TLS is disabled
03::27::19 info: GarnetServer[0] Total configured memory limit: 107558731776
03::27::19 info: TsavoriteKV [main][0] ********* Primary Recovery Information ********
03::27::19 info: StoreWrapper[0] Error during recovery of store; storeVersion = -1; objectStoreVersion = -1 Tsavorite.core.TsavoriteException: Unable to find valid HybridLog token    at Tsavorite.core.TsavoriteKV`4.FindRecoveryInfo(Int64 requestedVersion, HybridLogCheckpointInfo& recoveredHlcInfo, IndexCheckpointInfo& recoveredICInfo) in /src/libs/storage/Tsavorite/cs/src/core/Index/Recovery/Recovery.cs:line 327    at Tsavorite.core.TsavoriteKV`4.Recover(Int32 numPagesToPreload, Boolean undoNextVersion, Int64 recoverTo) in /src/libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Tsavorite.cs:line 349    at Garnet.server.StoreWrapper.RecoverCheckpoint(Boolean recoverMainStoreFromToken, Boolean recoverObjectStoreFromToken, Guid storeIndexToken, Guid storeHlogToken, Guid objectStoreIndexToken, Guid objectStoreHlogToken) in /src/libs/server/StoreWrapper.cs:line 246
* Ready to accept connections
03::27::19 info: StoreWrapper[0] NOTE: Take a checkpoint (SAVE/BGSAVE) in order to actually delete the older data segments (files) from disk
03::27::23 dbug: GarnetServer[0] Accepted TCP connection from 10.200.3.201:50768
03::27::23 dbug: Session[0] [10.200.3.201:50768] [005C39D4] Starting RespServerSession Id=1

Here redis-cli --pipe < db0.resp

03::28::02 dbug: GarnetServer[0] Accepted TCP connection from 172.16.16.1:40488
03::28::02 dbug: Session[0] [172.16.16.1:40488] [005818D1] Starting RespServerSession Id=2
Unhandled exception. System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at Garnet.server.GarnetObjectSerializer.DeserializeInternal(BinaryReader binaryReader) in /src/libs/server/Objects/Types/GarnetObjectSerializer.cs:line 46
   at Garnet.server.GarnetObjectSerializer.Deserialize(IGarnetObject& obj) in /src/libs/server/Objects/Types/GarnetObjectSerializer.cs:line 29
   at Tsavorite.core.GenericAllocatorImpl`3.RetrievedFullRecord(Byte* record, AsyncIOContext`2& ctx) in /src/libs/storage/Tsavorite/cs/src/core/Allocator/GenericAllocatorImpl.cs:line 909
   at Tsavorite.core.AllocatorBase`4.AsyncGetFromDiskCallback(UInt32 errorCode, UInt32 numBytes, Object context) in /src/libs/storage/Tsavorite/cs/src/core/Allocator/AllocatorBase.cs:line 1719
   at Tsavorite.core.NativeStorageDevice._callback(IntPtr context, Int32 errorCode, UInt64 numBytes) in /src/libs/storage/Tsavorite/cs/src/core/Device/NativeStorageDevice.cs:line 112

Steps to reproduce the bug

1. dump many data from redis/keydb to Resp file (https://github.com/yannh/redis-dump-go):

# podman run ghcr.io/yannh/redis-dump-go:latest -insecure -host source-host -db 0 -n 1000 > db0.resp
Database 0: 61267252 element dumped

3. load data from resp-file redis-cli --pipe < db0.resp
4. wait

Expected behavior

Full loaded data from resp-file by redis-cli to garnet

Screenshots

No response

Release version

v1.0.35

IDE

No response

OS version

debian 11

Additional context

# cat /opt/garnet/etc/garnet.conf 
{
	"Port" : 6379,
	"Address" : null, 
	"MemorySize" : "100g",
	"PageSize" : "32m",
	"SegmentSize" : "1g",
	"IndexSize" : "128m",
	"IndexMaxSize": "",
	"MutablePercent" : 90,
	"ObjectStoreHeapMemorySize" : "",
	"ObjectStoreLogMemorySize" : "32m",
	"ObjectStorePageSize" : "1m",
	"ObjectStoreSegmentSize" : "32m",
	"ObjectStoreIndexSize" : "16m",
	"ObjectStoreIndexMaxSize": "",
	"ObjectStoreMutablePercent" : 90,
	"EnableStorageTier" : true,
	"CopyReadsToTail" : false,
	"ObjectStoreCopyReadsToTail" : false,
	"LogDir" : "/data",
	"CheckpointDir" : null,
	"Recover" : true,
	"DisablePubSub" : false,
	"EnableIncrementalSnapshots" : false,
	"PubSubPageSize" : "4k",
	"DisableObjects" : false,
	"EnableCluster" : false,
	"CleanClusterConfig" : false,
	"AuthenticationMode" : "NoAuth",
	"Password" : null,
	"ClusterUsername" : null,
	"ClusterPassword" : null,
	"AclFile" : null,
	"AadAuthority" : "https://login.host.com",
	"AadAudiences" : null,
	"AadIssuers" : null,
	"AuthorizedAadApplicationIds" : null,
	"AadValidateUsername": false,
	"EnableAOF" : false,
	"AofMemorySize" : "64m",
	"AofPageSize" : "4m",
	"AofReplicationRefreshFrequencyMs": 10,
	"SubscriberRefreshFrequencyMs": 0,
	"CommitFrequencyMs" : 0,
	"WaitForCommit" : false,
	"AofSizeLimit" : "",
	"CompactionFrequencySecs" : 300,
	"CompactionType" : "Lookup",
	"CompactionForceDelete": false,
	"CompactionMaxSegments" : 32,
	"ObjectStoreCompactionMaxSegments" : 32,
	"EnableLua" : false,
	"LuaTransactionMode" : false,
	"GossipSamplePercent" : 100,
	"GossipDelay" : 5,
	"ClusterTimeout" : 60,
	"ClusterTlsClientTargetHost" : "GarnetTest",
	"EnableTLS" : false,
	"CertFileName" : null,
	"CertPassword" : null,
	"CertSubjectName" : null,
	"CertificateRefreshFrequency" : 0,
	"ClientCertificateRequired" : true,
	"ServerCertificateRequired" : true,
	"CertificateRevocationCheckMode" : "NoCheck",
	"IssuerCertificatePath" : "",
	"LatencyMonitor" : true,
	"MetricsSamplingFrequency" : 1,
	"QuietMode" : false,
	"LogLevel" : "Trace",
	"LoggingFrequency" : "10",
	"DisableConsoleLogger" : false,
	"FileLogger" : null,
	"ThreadPoolMinThreads" : 0,
	"ThreadPoolMaxThreads" : 0,
	"UseAzureStorage" : false,
	"AzureStorageConnectionString" : null,
	"CheckpointThrottleFlushDelayMs" : 0,
	"EnableFastCommit" : true,
	"FastCommitThrottleFreq" : 1000,
	"NetworkSendThrottleMax" : 8,
	"EnableScatterGatherGet" : false,
	"ReplicaSyncDelayMs" : 5,
	"MainMemoryReplication" : false,
	"OnDemandCheckpoint" : false,
	"UseAofNullDevice" : false,
	"UseNativeDeviceLinux" : true,
	"RevivBinRecordSizes" : null,
	"RevivBinRecordCounts" : null,
	"RevivifiableFraction" : 1.0,
	"EnableRevivification" : false,
	"RevivNumberOfBinsToSearch" : 0,
	"RevivBinBestFitScanLimit" : 0,
	"RevivInChainOnly" : false,
	"RevivObjBinRecordCount" : 256,
	"ObjectScanCountLimit" : 1000,
	"ExtensionBinPaths": null,
	"ExtensionAllowUnsignedAssemblies": false,
	"IndexResizeFrequencySecs": 60,
	"IndexResizeThreshold": 50,
	"LoadModuleCS": null
}

FYI: dragonfly is load same resp file.