LiteDRAM USPDDRPHY unable to meet DDR4 timing requirements? #303
Comments
|
Hi @jaccharrison, LiteDRAM will indeed not allow you to achieve maximum DDR4 speed but can do up to 1400MT/s with the right FPGA speedgrade (The current limitation is related to the max BUFG freq). The IOs are used in component mode and going higher in frequency would probably require switching to native mode which could require quite a bit of work and hasn't been planned yet. I'm not directly involved in AntMicro's Rowhammer tester, but I think it would be interesting to get prebuilt bitstream from Antmicro first and try them on your system. This would rules out potential LiteX change issue/Vivado version issue. It seems you also have memtest issues. I would be intersted to have the LIteX DDR4 calibration log to see how it behaves. |
|
Hi @enjoy-digital, Thanks for the reply! Could you explain how the design is able to operate at up to 1400 MT/s? I may be missing something here, but it seems to me that if the maximum clock period on the IOSERDESE3 components is 1.6 ns (at least for the UltraScale Plus), the maximum clock frequency should be 1/1.6e-9 = 625 MHz. Wouldn't 1400 MT/s operation require a 700 MHz clock? A validated bitstream is certainly something we could try, though the bitstreams are coupled to the timing parameters of a particular DRAM module and it would be difficult to produce a validated bitstream unless I have the same DIMM as @kgugala. If this is something we'd like to try, I could supply my module geometry/timings class, which I believe is correct as values from SPD and the data sheet agree. Yes, I do have memtest issues. They're intermittent, though, which is one of the reasons that I was wondering whether I'm violating timing parameters. When you say DDR4 calibration log, do you just mean the output that is printed when I run calibration in the BIOS? Or is there a different log that you're referring to? Thanks again! I'll look into native-mode primitives. Jacob |
|
When I was testing this on a 2018.2 version of Vivado and on Virtex Ultrascale(+), the main limitation reported by the tools was that max BUFG frequency was reached. When looking at current datasheet, in indeed seems that ISERDESE3 are software limited to 625MHz clk, so would mean a 1250MT/s operation. I would need to do a test on a recent version of Vivado to compare. For the log, the calibration that is done by the BIOS would be useful yes. |
|
Sorry for the delay. Below are two representative terminal transcripts on one of my DDR4 DIMMs. What you can see from output 1 is the following:
In output 2:
Usually I see only one error, if I see any. Rarely, I'll see two errors. Sometimes, I run the tests many times in a row without seeing any errors. So my error is intermittent. I'd add that while I initially thought I was getting similar behavior on several different DIMMs, as I've gone back and documented my results more rigorously, I've only been able to reproduce the errors on one DIMM. I only have one copy of this particular DIMM unfortunately, and I don't have any other DIMMs with the same die as this DIMM. I'm performing more tests and trying to isolate any cause for these failures. In particular, I'm about to write some scripts that repeatedly run memtests on my other DIMMs to try to see if I can get any memory test failures on other DIMMs. I'll update here if I find anything interesting from those other tests. In the meantime, please let me know if you see anything notable in the following BIOS outputs! Here's a second run, captured after rebooting the board for the same module as above: |
|
Thanks @jaccharrison. The calibration logs look fine. The issue could be related to electrical settings that would need to be adjusted. You can use ddr4_mr_gen tool for this. The tool will generate commands that you can copy/past in the BIOS to adjust settings and you can then do a |
|
@jaccharrison what is the exact DIMM module you are experiencing these intermittent failures with? We have done our tests with a |
|
Thanks @enjoy-digital for the tip -- I'll try that out and let you know what I find. @tmichalak the DIMM I am using is from ATP, model number |
|
Wanted to give a quick update -- tinkering with different termination resistances did not solve the problem. Apparently others may be experiencing this same issue. I'm continuing to troubleshoot and see if I can identify a cause and solution. I'll post here if I do. |
Hi there,
I've been working with AntMicro's Rowhammer tester on a ZCU104 development board. This framework instantiates a DDR4 PHY with a clock frequency of 500 MHz. I was not seeing the outputs I expected from this framework, so I've done some troubleshooting. I have not ruled out the possibility that my unexpected behavior comes from the DDR clock being too slow. I've tried synthesizing with a higher clock frequency but it fails timing. One of the major reasons for the timing failure appears to be the use of the ISERDESE3 block in the LiteDRAM core.
According to the UltraScale Plus User Guide, ISERDESE3 blocks have a minimum clock period of 1.6ns, or 625 MHz. Thus, a DDR4 PHY that uses these blocks cannot even operate at the 666 MHz that is nominally required for the slowest DDR4 speed mode (1333 MT/s). The DDR4 ICs I'm using specify a maximum clock period of 1.8 ns for the 1333 MT/s mode, but I'd really like them to run at 1600 MT/s, where tck(max) is 1.5 ns (1.25 ns/800 MHz would be nominal).
The UltraScale Plus does have different I/O SERDES blocks that seem to be capable of running at higher frequencies. I could potentially try to help port the USP DDR4 PHY to use these higher-frequency I/O SERDES components, but before I start to seriously investigate this, I was wondering if others have feedback on this idea? Am I crazy? Am I missing something (I can't find anybody else talking about this, so I'm suspicious that I might be)? I suspect that the reason for a max clock period comes from a DLL/PLL on the DRAM -- can anybody confirm or refute this?
Any help or expertise would be greatly appreciated!
The text was updated successfully, but these errors were encountered: