Bugzilla – Bug 1225691
[Bronco] Fail to boot into sles15sp6 via iSCSI boot multi path when mutual chap enabled.
Last modified: 2024-07-18 15:30:17 UTC
Created attachment 875238 [details] Ibft_config information [Problem descriptions] Fail to boot into sles15sp6 via iSCSI boot from SAN multi path. The 2nd iSCSI target is unable to connect during installation, even issuing commands "iscsiadm -m fw -l" manually. Error message show up before boot into OS. =============================================================== [Steps for Reproducing] 1. Insert 3820c to SY480. 2. Connected to Potash ICM. 3. Setup iSCSI initiator, target, multi path and CHAP info. 4. GRUB add command: iomem=relaxed before install sles15sp6. 5. Full installation of sles15sp6(SLE-15-SP6-Full-x86_64-PublicRC-202404-Media1). The second iSCSI target is unable to connect, even issuing "iscsiadm -m fw -l". 6. After OS installation and reboot, fail to boot into os and show error message. ============================================================= [Hardware Configuration] Server Type: Synergy 480 Gen10 Build of Server (VP, Production): VP ROM Family & Version: I42 v3.20 (03/14/2024) Operating System: sles15sp6 Boot Mode (Legacy BIOS or UEFI): UEFI OS version installed (Local HD or BFS): BFS iLO version: iLO 5
Created attachment 875239 [details] Preboot configuration
setup with CHAP multipath boot from SAN: ---------------------------------------- 1. Test is for sles15sp6 boot from SAN configuration over bnx2i. 2. CHAP authentication is configured in preboot for both initiator port and one target controller. (Refer img preboot-config) 3. Each initiator port has its own iqnname, ipaddress and chap credentials. Result: ------- 1. Only one port login to the target port. Second port does not login to the target port. 2. Login to the second port fails even after issuing iscsiadm command manually, #iscsiadm -m fw -l Finding and observation: ------------------------ 1. This is a boot from SAN installation over bnx2i. The boot information required is populated using kernel module iscsi_ibft.ko, which "iscsiadm -m fw" use for login to target. 2. Refer ibft_config for "iscsiadm -m fw" output. 3. iBFT table shows port 1 iqnname used for both initiator port iface records. 4. iBFT table CHAP shows CHAP information getting publish for each initiator port in iface records. Our observation is that first initiator port was able to login to the target because in iBFT table iface records initiator name and CHAP information matches with port 1 configuration. And it fails to login to the second port because in iBFT table iface records initiator name is publish for port 1 and CHAP information is published for port2 configuration.
Note: For iscsi_ibft, publish only one iqnname for two initiator ports is known. ================================================================================ We test same setup without CHAP: setup multipath boot from SAN without CHAP: ---------------------------------------- 1. Test is for sles15sp6 boot from SAN configration over bnx2i. 2. Each initiator port has its own iqnname, ipaddress. Result: ------- 1. Both the initiator port able to login to the target. 2. We were able to install sles15sp6 OS successfully, and able to boot into it. Note: For first boot, iscsiuio service was not enabled by default, So below setup were used. a. Add single option in kernel commandline. b. issue below command #systemctl enable iscsiuio.service Observation: Boot from SAN works without any issue.
We have also tested redhat on the same setup with CHAP multipath boot from SAN, and we did not see any issue with installation and first boot. Our observation with redhat is that first initiator port was able to login to the target because in iBFT table iface records initiator name, ipaadress and CHAP information matches with port 1 configuration. And for second port login also succeed because in iBFT table iface records initiator name is publish for port 1, Ip address from port2 and CHAP information is also published for port1 configuration. As per our understanding we think ibft CHAP information needs to be corrected, but not sure whether it is in kernel module iscsi_ibft.ko or open-iscsi tools.
Small correction attached images for pre-boot configuration and iscsiadm ibft output are from different setup but same issue.
I'll take this bug. Please don't set priority or bug "state" (i.e. "New", "In Progress", etc) yourself. David, do you set the IQNs of your two NICs, for multipathing, differently or the same?
(In reply to Lee Duncan from comment #6) > I'll take this bug. > > Please don't set priority or bug "state" (i.e. "New", "In Progress", etc) > yourself. > > David, do you set the IQNs of your two NICs, for multipathing, differently > or the same? The same, as to my understanding open-iscsi will only use 1 initiator name. If the two iBFTs have different iqns, how does open-iscsi choose? Does it just take whatever happens to be first in the iBFT table? I'll give this a try in a bit,, but it seems unlikely to be successful unless the target is set to allow both possibly iqns access to the each of the targets for this initiator. Though perhaps something is happening at the offload layer that would allow this to work.?.?
(In reply to Manish Rangankar from comment #2) > setup with CHAP multipath boot from SAN: > ---------------------------------------- ... > 3. Each initiator port has its own iqnname, ipaddress and chap credentials. This is not what the attachment on the description shows[1]. It shows the two ports having the same initiator name (as would be necessitated by the iBFT structure). I could see setting up two port groups on the target as a possibly way to get this to work[2], outside of that I'm confused as to how this worked with RHEL[3]. [1] I'm assuming from the userids that you intended the second port to be iqn.2023-09.com:3par-mahone04 but it's mahone03 as well [2] then you could have different chap credentials defined for the same initiator name. [3] unless it's using an information source other than the iBFT???
(In reply to David Bond from comment #8) > (In reply to Manish Rangankar from comment #2) > > setup with CHAP multipath boot from SAN: > > ---------------------------------------- > ... > > 3. Each initiator port has its own iqnname, ipaddress and chap credentials. > > This is not what the attachment on the description shows[1]. > Here I'm referring to the pre-boot configuration (Refer img preboot-config).
(In reply to David Bond from comment #8) > (In reply to Manish Rangankar from comment #2) > > setup with CHAP multipath boot from SAN: > > ---------------------------------------- > ... > > 3. Each initiator port has its own iqnname, ipaddress and chap credentials. > > This is not what the attachment on the description shows[1]. > > It shows the two ports having the same initiator name (as would be > necessitated by the iBFT structure). I could see setting up two port groups > on the target as a possibly way to get this to work[2], outside of that I'm > confused as to how this worked with RHEL[3]. > > > [1] I'm assuming from the userids that you intended the second port to be > iqn.2023-09.com:3par-mahone04 but it's mahone03 as well For bnx2i adaptors, it's always been the case where only one iqnname get publish for both ports. So, I don't expect it to be "iqn.2023-09.com:3par-mahone04", but yes for second port userid I was expecting mahone03, which corresponds to "iqn.2023-09.com:3par-mahone03". On RH we see this. > > [2] then you could have different chap credentials defined for the same > initiator name. > At preboot level configuration we have different initiator name, chap configured for each port. > [3] unless it's using an information source other than the iBFT??? Yes, information "#iscsiadm -m fw" is consumed from iBFT table only. Not sure if iscsiadm is doing from magic internally.
(In reply to Manish Rangankar from comment #10) > (In reply to David Bond from comment #8) > > (In reply to Manish Rangankar from comment #2) > > > setup with CHAP multipath boot from SAN: > > > ---------------------------------------- > > ... > > > 3. Each initiator port has its own iqnname, ipaddress and chap credentials. > > > > This is not what the attachment on the description shows[1]. > > > > It shows the two ports having the same initiator name (as would be > > necessitated by the iBFT structure). I could see setting up two port groups > > on the target as a possibly way to get this to work[2], outside of that I'm > > confused as to how this worked with RHEL[3]. > > > > > > [1] I'm assuming from the userids that you intended the second port to be > > iqn.2023-09.com:3par-mahone04 but it's mahone03 as well > > For bnx2i adaptors, it's always been the case where only one iqnname get > publish for both ports. So, I don't expect it to be > "iqn.2023-09.com:3par-mahone04", but yes for second port userid I was > expecting mahone03, which corresponds to "iqn.2023-09.com:3par-mahone03". On > RH we see this. As far as I know, if both adapters have the same IQN then both adapters need the same username/password for CHAP. Otherwise, how will the target know which one is which? CHAP isn't like logging into Linux, where you can have lots of different accounts, and each one has it's own password. It's initiator-based, so one initiator has one username/password. > > > > > [2] then you could have different chap credentials defined for the same > > initiator name. > > > > At preboot level configuration we have different initiator name, chap > configured for each port. So you change the initiator name between pre-boot and the running system. That's not right. > > > [3] unless it's using an information source other than the iBFT??? > > Yes, information "#iscsiadm -m fw" is consumed from iBFT table only. Not > sure if iscsiadm is doing from magic internally.
@Marvell, Please let us know what version of RedHat this was seen to work with. I'll pull the images and do an install to see if I can verify the difference in behavior. But to be honest, I don't see how the target is going to permit the second port to login with a mismatched initiator name (from the first port), and set of chap credentials (from the second port configuration).
@David version of RedHat this was seen to work with RHEL 8.10.
(In reply to Rajeev Singh from comment #13) > @David > version of RedHat this was seen to work with RHEL 8.10. I can't find media (yet) for 8.1. I tried 8.0 and ran into the same situation. To double check what was happening I setup a network trace, and checked the initial stages of the iscsi login. I see the both interfaces using the initiator information from the primary port. Would it be possible to collect a network trace (of both interfaces) of your working install from power on until the o/s finishes boot?
(In reply to David Bond from comment #14) > (In reply to Rajeev Singh from comment #13) > > @David > > version of RedHat this was seen to work with RHEL 8.10. > > I can't find media (yet) for 8.1. I tried 8.0 and ran into the same > situation. To double check what was happening I setup a network trace, and > checked the initial stages of the iscsi login. I see the both interfaces > using the initiator information from the primary port. > > Would it be possible to collect a network trace (of both interfaces) of your > working install from power on until the o/s finishes boot? @David please try with RHEL8.10 GA build and not 8.1
David, Lee, Any further updates?
(In reply to Manish Rangankar from comment #3) > Note: For iscsi_ibft, publish only one iqnname for two initiator ports is > known. > ============================================================================= > === > We test same setup without CHAP: > > setup multipath boot from SAN without CHAP: > ---------------------------------------- > 1. Test is for sles15sp6 boot from SAN configration over bnx2i. > 2. Each initiator port has its own iqnname, ipaddress. > > Result: > ------- > 1. Both the initiator port able to login to the target. > 2. We were able to install sles15sp6 OS successfully, and able to boot into > it. > Note: For first boot, iscsiuio service was not enabled by default, So below > setup were used. > a. Add single option in kernel commandline. > b. issue below command > #systemctl enable iscsiuio.service > > Observation: Boot from SAN works without any issue. Can you please validate that the correct IQN got used for both connections? As both IQNs are valid for logging into the target, you clearly would always get a valid connection, even when the wrong IQN is used for the connection.
Nilsesh: Did you guys try Hannes' suggestion? Any updates?
Lee, Hannes, Actual config is as below, P1 -> IQN1 -> CHAP1 -> T1 P2 -> IQN2 -> CHAP2 -> T2 The IQNs are setup correctly. Our observation is that first initiator port was able to login to the target because in iBFT table iface records initiator name and CHAP information matches with port 1 configuration. And it fails to login to the second port because in iBFT table iface records initiator name is publish for port 1 and CHAP information is published for port2 configuration. Config as seen in iBFT, (causing login failure for P2), P1 -> IQN1 -> CHAP1 -> T1 P2 -> IQN1 -> CHAP2 -> T2 (cause of login failure) (pls refer ibft_config information attachment) The iBFT should show P2 -> IQN2 -> CHAP2 -> T2 for login to succeed.