Thank you.

I am not even sure if it makes sense to do that memory profiling even during firstboot: We started doing it to avoid running out of memory in the inst-sys where we have very limited resources, in particular no swap space, and the RAM disk we need to have any writable files also eats away from the available RAM.

But during firstboot that's not an issue anymore. And it might also not be desired to have awk in the target system. So maybe we should limit the use of awk / that entire memory profiling to the inst-sys.

Martin, what do you think?

The whole awk package has an installed size of 3.1 MB, of which 1.2 MB are consumed by its docs and info pages; see "qdirstat pkg:/gawk", and another 1.1 MB by its translations.

Looks like awk is used in some other places as well, albeit always only to extract one of many whitespace-separated fields in a command's output.


> [sh @ balrog] ...yast/yast-installation/startup 24 % grep -R awk .  | sed -e 's/^/> /'
> ./YaST2.ssh:		SecondStageRequired=$(awk ' /^SecondStageRequired:/ { print $2 }' < /etc/install.inf)
> ./common/logging.sh:		Loghost=$(awk ' /^Loghost:/ { print $2 }' < /etc/install.inf)
> ./YaST2.call:		SecondStageRequired=$(awk ' /^SecondStageRequired:/ { print $2 }' < /etc/install.inf)
> ./YaST2.call:MEM_TOTAL=`awk '/^MemTotal:/{ print $2 }' /proc/meminfo`
> ./First-Stage/F08-logging:    USE=$(awk '/^MemFree:/{ n=2 ; printf "%d\n", $n/4 }' /proc/meminfo)
> ./First-Stage/F10-cleanup:awk '/ \/mnt/{ print $2 }' /proc/mounts | sort -r | xargs --no-run-if-empty umount

busybox or busybox-static should also be a viable alternative since we don't need any fancy features from the awk command. Which begs the question how minimalistic an installation we support. If there is a firstboot workflow defined in a product, there needs to be a reasonable set of standard tools installed.

Sure, we can explicitly require many things like the gawk package, but that means that a minimal installation won't be so minimal anymore, and it will be difficult to use lighterweight alternatives like busybox instead.

Scott, so you don't even have busybox installed on that system? Or is the package installed, but the links for all the individual subcommands like awk, grep etc. were not created?

> [sh @ balrog] ~/tmp 28 % rpm -qpl ./busybox-1.35.0-150500.8.2.x86_64.rpm 
> /etc/man.conf
> /usr/bin/busybox
> /usr/bin/busybox.install
> /usr/share/busybox
> /usr/share/busybox/busybox.links
> /usr/share/doc/packages/busybox
> /usr/share/doc/packages/busybox/mdev.txt
> /usr/share/licenses/busybox
> /usr/share/licenses/busybox/LICENSE
> /usr/share/man/man1/busybox.1.gz
> [sh @ balrog] ~/tmp 29 % rpm -qp --scripts ./busybox-1.35.0-150500.8.2.x86_64.rpm
> [sh @ balrog] ~/tmp 30 % rpm -qp --provides ./busybox-1.35.0-150500.8.2.x86_64.rpm
> busybox = 1.35.0-150500.8.2
> busybox(x86-64) = 1.35.0-150500.8.2
> config(busybox) = 1.35.0-150500.8.2
> useradd_or_adduser_dep

It doesn't look like our busybox package comes with the links for its subcommands packaged or creates them in a post-install script.

It's not only 'awk', it's also other common Linux commands like 'grep', 'sed', 'tr', 'cut' (and probably more) that we are using in those startup scripts.

We can't and don't want to do weird workarounds for every one of them: That would only obscure the intent of the code and make it much less readable and maintainable.

(In reply to Stefan Hundhammer from comment #1)

> I am not even sure if it makes sense to do that memory profiling even during
> firstboot: We started doing it to avoid running out of memory in the
> inst-sys where we have very limited resources, in particular no swap space,
> and the RAM disk we need to have any writable files also eats away from the
> available RAM.
> [...]
> Martin, what do you think?

Actually these things are unrelated.

In yast-installation*, MEM_TOTAL and MEM_NEEDED, computed via awk, are related to y2logging and disabling the Qt GUI in favor of TUI.
The memsample memory profiling is not using awk.

* https://github.com/search?q=repo%3Ayast%2Fyast-installation+MEM_&type=code

On my Leap 15.4 desktop, awk is required by 5 packages, and the more basic ones by:
coreutils 657 (including yast2-installation)
grep      108 (including y-i)
sed        78 (missing y-i)

Which leads me to think that we should declare our dependencies in the spec file, and possibly replace our awk usage.

Indeed.

[sh @ balrog] ~ 8 % rpm -q --whatrequires gawk

fonts-config-20200609+git0.42e2b1b-150000.4.10.1.noarch
tuned-2.10.0-150400.19.10.noarch
tcsh-6.20.00-4.15.1.x86_64
rpm-build-4.14.3-150300.55.1.x86_64

OK, since 'gawk' also provides 'awk', and some more packages also require 'awk', and YaST with all its dependencies such as Ruby isn't exactly lightweight anyway, I think requiring 'awk' in yast2-installation is a good compromise.

If a lighterweight package like 'busybox' also provides 'awk' (which it currently does not), it will be an alternative.

Fix:

  https://github.com/yast/yast-installation/pull/1096

This will arrive with yast2-installation-4.6.7.

SR to OBS:  https://build.opensuse.org/request/show/1104012
SR to IBS:  https://build.suse.de/request/show/305523

See also https://bugzilla.opensuse.org/show_bug.cgi?id=1214365 about a CI failure on libnbd caused by a lack of default awk.  GNU Coding Standards have a minimum set of tools that are assumed to be present on all systems:

https://www.gnu.org/prep/standards/standards.html#Utilities-in-Makefiles
awk cat cmp cp diff echo egrep expr false grep install-info ln ls
mkdir mv printf pwd rm rmdir sed sleep sort tar test touch tr true

and other applications should not have to call out any of these as explicit dependencies, because they should already be part of the basic install.

The coreutils already covers a lot of those. See

  rpm -ql coreutils | grep bin

And to reiterate: I am with you on this. See also https://bugzilla.suse.com/show_bug.cgi?id=1214365#c1 + c2.