Bugzilla – Bug 1217619
grub2 reproducible builds
Last modified: 2023-12-06 06:00:07 UTC
While working on reproducible builds for openSUSE, I found that our grub2 package's /usr/share/grub2/x86_64-xen/grub.xen varies across builds. I identified 2 issues: The tar contains changing (octal) mtime values. The tar stores files in random filesystem readdir order. The first issue produces such diffs: --- old /usr/share/grub2/x86_64-xen/grub.xen (objdump) +++ new /usr/share/grub2/x86_64-xen/grub.xen (objdump) @@ -4896,7 +4896,7 @@ 01319f 00000000 30303030 36303000 30303031 ....0000600.0001 0131af 37353000 30303031 37353000 30303030 750.0001750.0000 0131bf 30303033 34353000 31343533 31323035 0003450.14531205 - 0131cf 35303300 30303135 30313320 30000000 503.0015013 0... + 0131cf 36313200 30303135 30313420 30000000 612.0015014 0... 0131df 00000000 00000000 00000000 00000000 ................ 0131ef 00000000 00000000 00000000 00000000 ................ and the second issue produced this diff: 01311f 00000000 00000000 01000000 00000000 ................ 01312f 00000000 00000000 01000000 08b42600 ..............&. 01313f 626f6f74 2f677275 622f7838 365f3634 boot/grub/x86_64 - 01314f 2d78656e 2f646973 6b2e6d6f 64000000 -xen/disk.mod... + 01314f 2d78656e 2f6c7378 656e2e6d 6f640000 -xen/lsxen.mod.. 01315f 00000000 00000000 00000000 00000000 ................ Here is a PoC patch for the first issue: --- a/util/grub-mkstandalone.c +++ b/util/grub-mkstandalone.c @@ -264,6 +264,7 @@ add_tar_file (const char *from, memcpy (hd.gid, "0001750", 7); set_tar_value (hd.size, size, 12); + mtime = 42; set_tar_value (hd.mtime, mtime, 12); hd.typeflag = '0'; memcpy (hd.magic, MAGIC, sizeof (hd.magic)); The second issue probably comes from grub_util_fd_readdir that would need to collect+sort entries before further processing.
Hi Bernhard, Thanks for opening the issue and providing the information. You are absolutely right, the varying timestamp and the nondeterministic readdir() result is the culprit. For that these two patches have been worked out to address them separately. https://build.opensuse.org/package/rdiff/home:michael-chang:branches:Base:System/grub2?opackage=grub2&oproject=Base%3ASystem&rev=3 Would you mind to check if the preliminary version works for you ? I'd like to have your signed-off-by or test-by before doing the submission to factory and upstream. It is all up to you and if you are not available, I can just go ahead and do the submit.
I tested it successfully. It produces bit-identical results now. The patches also look good to me. The only thing I wondered why the free (tfp); was not already there before. That should have leaked some (insignificant) memory.
(In reply to Bernhard Wiedemann from comment #2) > I tested it successfully. It produces bit-identical results now. > > The patches also look good to me. Thanks for your feedback. With the result I think it is ready to submit to factory and also upstream (with your signed-off-by). > The only thing I wondered why the free (tfp); > was not already there before. > That should have leaked some (insignificant) memory. I came across the memory leak issue while working on the patch, given it is trivial one to fix along I'm not bothered to create a separate patch for it.