Bug 1227207

Summary: Failure to fetch ext_pillar git
Product: [openSUSE] openSUSE Distribution Reporter: Olav Reinert <seroton10>
Component: SaltAssignee: E-Mail List <salt-maintainers>
Status: CONFIRMED --- QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P2 - High CC: artem.shiliaev, pablo.suarezhernandez, seroton10
Version: Leap 15.6   
Target Milestone: Leap 15.6   
Hardware: x86-64   
OS: Other   
URL: https://github.com/SUSE/spacewalk/issues/24799
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Olav Reinert 2024-06-30 10:26:22 UTC
I have a salt-master configured to have an external pillar, as follows:


ext_pillar:
  - git:
    - master pki@git:pki-pillar.git

git_pillar_update_interval: 3600

git_pillar_privkey: /var/lib/salt/.ssh/id_ed25519
git_pillar_pubkey: /var/lib/salt/.ssh/id_ed25519.pub


When the salt-master starts, it is unable to load the pillar configuration, failing with these log entries:


2024-06-30 12:15:06,712 [salt.utils.gitfs :295 ][DEBUG   ][24319] Set update lock for git_pillar remote 'master pki@tresor.plexor.net:pki-pillar.git'
2024-06-30 12:15:06,712 [salt.utils.gitfs :295 ][DEBUG   ][24319] Fetching git_pillar remote 'master pki@tresor.plexor.net:pki-pillar.git'
2024-06-30 12:15:06,725 [salt.utils.gitfs :295 ][ERROR   ][24319] Error occurred fetching git_pillar remote 'master pki@tresor.plexor.net:pki-pillar.git': error loading known_hosts: 
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/salt/utils/gitfs.py", line 1989, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/pygit2/remotes.py", line 160, in fetch
    payload.check_error(err)
  File "/usr/lib64/python3.11/site-packages/pygit2/callbacks.py", line 99, in check_error
    check_error(error_code)
  File "/usr/lib64/python3.11/site-packages/pygit2/errors.py", line 66, in check_error
    raise GitError(message)
_pygit2.GitError: error loading known_hosts: 
2024-06-30 12:15:06,726 [salt.utils.gitfs :295 ][DEBUG   ][24319] Removed update lock for git_pillar remote 'master pki@tresor.plexor.net:pki-pillar.git'


This issue is described in the following bug report:

    https://github.com/saltstack/salt/issues/64121

As suggested in the bug report, adding the following override to the salt-master.service unit file solves the problem for me:

    [Service]
    Environment=HOME=/var/lib/salt

Perhaps consider amending the unit file provided by the salt-master RPM accordingly?
Comment 1 Olav Reinert 2024-06-30 10:35:16 UTC
NAME="openSUSE Tumbleweed"
VERSION_ID="20240628"
Comment 2 Olav Reinert 2024-07-10 16:07:17 UTC
Modified bug to apply to Leap 15.6, where I have also found it.
Comment 3 Pablo Suárez Hernández 2024-07-11 09:32:25 UTC
Hmmm, this is strange. This issue should be already fixed in our Salt package by: https://github.com/openSUSE/salt/pull/588

I just tested this in a Leap 15.6 container and it works fine to me, without any errors. Using an "ext_pillar" configuration as you did (with ssh).

Are you using the "salt-master" and "python3-salt" packages coming from the official Leap repositories?

Could you please run:

rpm -q python3-salt --changes | grep 1210994

to double check the package contains the fix?

Thanks in advance!
Comment 4 Olav Reinert 2024-07-11 13:04:04 UTC
(In reply to Pablo Suárez Hernández from comment #3)
> Hmmm, this is strange. This issue should be already fixed in our Salt
> package by: https://github.com/openSUSE/salt/pull/588
> 
> I just tested this in a Leap 15.6 container and it works fine to me, without
> any errors. Using an "ext_pillar" configuration as you did (with ssh).

FYI: My master runs in a KVM (managed by libvirt) running a default install of Leap 15.6 using the "Server" system role.

> Are you using the "salt-master" and "python3-salt" packages coming from the
> official Leap repositories?

Yes - and I have not added any repositories.

> Could you please run:
> 
> rpm -q python3-salt --changes | grep 1210994
> 
> to double check the package contains the fix?

master:~ # rpm -q python3-salt --changes | grep 1210994
- Make sure configured user is properly set by Salt (bsc#1210994)
- Prevent _pygit2.GitError: error loading known_hosts when $HOME is not set (bsc#1210994)


Just retested - when I remove my override for salt-master.service, the error occurs again; and it disappears after I reinstate the override.