Looking at https://forums.developer.nvidia.com/c/gpu-graphics/linux/148
Today, there was a reply to an issue reported by someone else, pointing to
https://github.com/NVIDIA/open-gpu-kernel-modules/issues/472

That issue has plenty of comments and was opened Mar 11, 2023 for 525.85.05.

Glancing through that issue:
* I didn't see my assert
* but 'MMUTest BAR0 window offset 0x70e000 returned garbage 0x0' showed up in one comment; the variant with 'f' instead of 'e' in the hex address showed up in another more recent comment.

Plus:

* May 21, 2024 a comment was:
> We missed calling it out in the changelog explicitly (oops), but this
> should be fixed with 555.42.02. Please test. I'll leave this
> bug open while 555.xx is still in beta.

A bit later, some user reported:
> I am also getting a crash on suspend once in a blue moon.
> Here's the logs  whenever the crash happens: [...]

With the reply (on June 4, 2024):
> Acknowledged the crash on suspend issue, we have filed a bug 4683310
> internally for tracking purpose.

I've enabled this NVreg_PreserveVideoMemoryAllocations kernel option and the these suspend/hibernate services some time ago.

-------------------------------------------------------------------
Mon Nov 21 12:30:46 UTC 2022 - Stefan Dirsch <sndirsch@suse.com>

- NVreg_PreserveVideoMemoryAllocations kernel option and enabled
  services nvidia-suspend, nvidia-resume and nvidia-hibernate now
  needed for GNOME Wayland (gdm) since
    commit 51181871e9db716546e9593216220389de0d8b03
    Author: Ray Strode <rstrode@redhat.com>
    Date:   Fri Mar 4 14:11:03 2022 -0500

      data: Disable wayland on nvidia if suspend is broken

If you don't·use GNOME Wayland you may get rid of the kernel option and disable the services again.

For this you need to edit 

  /usr/lib/modprobe.d/50-nvidia-default.conf

and run

  dracut --force
  systemctl disable nvidia-suspend.service
  systemctl disable nvidia-hibernate.service
  systemctl disable nvidia-resume.service

and reboot your system. Does that help?

> For this you need to edit 
> /usr/lib/modprobe.d/50-nvidia-default.conf

I guess you mean: /usr/lib/modprobe.d/59-nvidia-default.conf
of nvidia-open-driver-G06-signed-kmp-default-550.90.07_k6.9.3_1-1.1.x86_64

I can confirm that with /proc/driver/nvidia/params
  PreserveVideoMemoryAllocations: 0
suspending + unsuspending/waking up works and I also do not see any glitches, but I have not tried much.

nvidia-smi also works.

* * *

Initially, I forgot to run:
  dracut --force
  systemctl disable nvidia-suspend.service
  systemctl disable nvidia-hibernate.service
  systemctl disable nvidia-resume.service

and at least 'status' for suspend/resume showed that they were enabled, but also with them enabled it did work as described above.

(In reply to Tobias Burnus from comment #4)
> > For this you need to edit 
> > /usr/lib/modprobe.d/50-nvidia-default.conf
> 
> I guess you mean: /usr/lib/modprobe.d/59-nvidia-default.conf
> of nvidia-open-driver-G06-signed-kmp-default-550.90.07_k6.9.3_1-1.1.x86_64

Yes, you're right. I apologize.

> I can confirm that with /proc/driver/nvidia/params
>   PreserveVideoMemoryAllocations: 0
> suspending + unsuspending/waking up works and I also do not see any
> glitches, but I have not tried much.
> 
> nvidia-smi also works.
> 
> * * *
> 
> Initially, I forgot to run:
>   dracut --force
>   systemctl disable nvidia-suspend.service
>   systemctl disable nvidia-hibernate.service
>   systemctl disable nvidia-resume.service
> 
> and at least 'status' for suspend/resume showed that they were enabled, but
> also with them enabled it did work as described above.

That's interesting, but I'm afraid it's needed when using GNOME Wayland. May I ask which desktop you're using?

> May I ask which desktop you're using?

KDE Plasma 6 (Wayland)

I'm adding my contact at nvidia here. Could be interesting for them I believe.

21F. KNOWN ISSUES AND WORKAROUNDS

   o On some systems, where the default suspend mode is '"s2idle"', the system
     may not resume properly due to a known timing issue in the kernel. The
     suspend mode can be verified by reading the contents of the file
     '/sys/power/mem_sleep'. The following upstream kernel changes have been
     proposed to fix the issue:

           https://lore.kernel.org/linux-pci/20190927090202.1468-1-drake@endlessm.com/     

           https://lore.kernel.org/linux-pci/20190821124519.71594-1-mika.westerberg@linux.intel.com/     

     In the interim, the default suspend mode on the affected systems should
     be set to '"deep"' using the kernel command line parameter
     '"mem_sleep_default"' -

      'mem_sleep_default=deep'

I guess it's worth a try. Instead of removing PreserveVideoMemoryAllocations and therefore switching back from /proc/driver/nvidia/suspend to Kernel driver callback. 

So could you give this kernel option a try, please and readd PreserveVideoMemoryAllocations to nvidia options in modprobe.d snippet?

Also I double checked that PreserveVideoMemoryAllocations is needed when you want to make use of 

nvidia-suspend.service
nvidia-hibernate.service
nvidia-resume.service

This is checked by gdm. If these services are not enabled when using nvidia driver Wayland session isn't offered. Only Wayland.

> This is checked by gdm. If these services are not enabled when using nvidia driver 
> Wayland session isn't offered. Only Wayland.

Only X11 of course!

Currently:
$ cat /sys/power/mem_sleep 
[s2idle]

I will check the mem_sleep_default=deep kernel parameter later.

* * *

Let's link the GNOME gdm commit, mentioned in this bug, for easier access:
https://gitlab.gnome.org/GNOME/gdm/-/commit/51181871e9db716546e9593216220389de0d8b03

"deep" does not seem to be supported on this system (which has an i7-13800H, Raptor Lake-P 6p+8e cores Host Bridge/DRAM Controller, ...).

dmesg confirms:

Kernel command line: ... mem_sleep_default=deep

But:
tux:~ # cat /sys/power/mem_sleep 
s2idle

https://www.kernel.org/doc/Documentation/power/states.txt states:

"These modes are "s2idle" (Suspend-To-Idle), "shallow" (Power-On Suspend) and "deep" (Suspend-To-RAM). The "s2idle" mode is always available, while the other ones are only available if supported by the platform (if not supported, the strings representing them are not present in /sys/power/mem_sleep).  The string representing the suspend mode to be used subsequently is enclosed in square brackets."

Which shows implies that I only have one. Albeit currently it is "s2idle" while before it was "[s2idle]".

Especially given that it is a laptop, I wonder why only
  Suspend-To-Idle / S0 / "s2idle" ("freeze")
and not
  Standby / Power-On Suspend / S1 / "shallow" ("standby")
  Suspend-to-RAM / S3 / "deep"
seem to be available.

Thanks a lot for testing. Yeah, I think on newer systems like yours meanwhile only "s2idle" is left.