Bugzilla – Bug 1198453
Make %python_build and %pytest_arch parallel
Last modified: 2023-03-21 13:27:30 UTC
There are some packages in OBS which take ages. One example can be: python-scikit-learn It builds in OBS for ~2 hours and running mostly in one thread: https://build.opensuse.org/package/live_build_log/openSUSE:Factory/python-scikit-learn/standard/x86_64 I've just made a local experiment and it can definitely benefit from: 1) %python_build -j $num_threads Apparently, the package compilation can utilize ~8 jobs on average 2) BuildRequires: %{python_module pytest-xdist} And using pytest -n 16 can run tests in 1:58 while the serial takes at least 20 minutes. Can we please utilize that for bigger python packages?
Matěj, should we create a Trello card for this bug?
(In reply to Martin Liška from comment #0) > There are some packages in OBS which take ages. One example can be: > python-scikit-learn This seems like severely underdefined: what exactly do you ask for? Change of macros in python-rpm-macros (I am 100% against it), change in python-scikit-learn (please, provide SR), change in some packages (which ones)?
(In reply to Matej Cepl from comment #2) > (In reply to Martin Liška from comment #0) > > There are some packages in OBS which take ages. One example can be: > > python-scikit-learn > > This seems like severely underdefined: what exactly do you ask for? I would like to see faster builds of python packages. Similarly to make, where we currently use %make_build (which expands to make -j #cpu_cores) that leads to much faster builds. > Change > of macros in python-rpm-macros (I am 100% against it), change in > python-scikit-learn (please, provide SR), change in some packages (which > ones)? So I'm asking for a change to python-rpm-macros so that it will be applied in all packages. So please, use thread in %python_build as there are Python packages that compile C/C++/Fortran code. And similarly for test suite, utilize as many cores as possible.
(In reply to Martin Liška from comment #3) > I would like to see faster builds of python packages. Similarly to make, > where we currently use %make_build (which expands to make -j #cpu_cores) > that leads to much faster builds. > > So I'm asking for a change to python-rpm-macros so that it will be applied > in all packages. I am quite strongly against doing this globally (i.e., by changes in python-rpm-macros) for these reasons: - make has parallel building built-in itself, so it has zero additional build requirements. It is by far not so with python-pytest-xdist (see stitny~$ osc buildinfo openSUSE:Factory python-pytest-xdist standard x86_64). We just cannot dump all these dependencies into ring 0 and ring 1. - parallel building in make is apparently (at least, as far as I know) thoroughly debugged and doesn’t break tests, it is by far not so with python-pytest-xdist. There are many broken tests runs because of bugs in -xdist and debugging them is as pleasant as debugging any parallel programming problem. So, I am quite open to any SRs with changes where python-pytest-xdist is used, but I don't like the change to be made globally. Steve, any comments?
You have covered the arguments pretty well. If you want to use pytest-xdist, please feel free to do so, SRs are very welcome.