Bug 617912

Summary: loading WLAN module ssb crashes the system (b43 related)
Product: [openSUSE] openSUSE 11.3 Reporter: Oliver Neukum <oneukum>
Component: KernelAssignee: Greg Kroah-Hartman <gregkh>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: clarkt, coolo, forgotten_CHC7OcAR44, jeffm, Larry.Finger, tiwai
Version: RC 1   
Target Milestone: ---   
Hardware: i686   
OS: openSUSE 11.3   
Whiteboard: maint:released:11.3:35403
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Deadline: 2010-09-07   

Description Oliver Neukum 2010-06-28 13:32:53 UTC
The module ssb, which is autoloaded, if the system has a broadcom card, crashes the system. It is a hard lock. The problem is 100% reproducible.
Comment 1 Jeff Mahoney 2010-06-28 18:10:28 UTC
Can you blacklist it, drop to console, and load it by hand?

I have hardware with a b43 chip and it's booting fine with the 2.6.34-8-desktop and 2.6.34-9-desktop kernels on i586.
Comment 2 Oliver Neukum 2010-06-28 20:00:24 UTC
I can. Hard Lock with nothing in the logs.
Comment 3 Oliver Neukum 2010-06-28 20:06:29 UTC
I booted into /bin/bash. Hard lock, nothing in the logs, when I do that.
Comment 4 Stephan Kulow 2010-06-28 21:21:16 UTC
did it work with previous kernel versions?
Comment 5 Oliver Neukum 2010-06-29 06:21:26 UTC
It worked with SLED11 and SLED11SP1.
Comment 6 Oliver Neukum 2010-06-29 06:24:58 UTC
Vlado, does
01:00.0 Network controller: Broadcom Corporation BCM4312 802.11b/g LP-PHY (rev 01)
generally work the standard kernel?
Comment 7 Oliver Neukum 2010-06-29 08:26:15 UTC
I installed the SP1 kernel. The system still crashes. I suspect the earlier systems I tested with had a binary driver for the card. I will also try the vanilla 2.6.32 kernel.
Comment 8 Stefan Dirsch 2010-07-07 12:09:59 UTC
Same issue here. Even adding "BrokenModules=b43,ssb" to kernel boot command line doesn't help. Still ssb is being loaded. I need to boot via init=/bin/bash and
remove b43/ssb kernel modules manually to workaround that issue.

Hardware: Broadcom BCM4312 802.11b/g (0x14e4:0x4315) (NUE 816 Villemont SI SKU4)
Comment 9 Vladimir Botka 2010-07-14 16:08:43 UTC
Similar problem was solved in [1]. We created "ohci-hcd module without ssb support" and modified initrd accordingly.

[1] https://bugzilla.novell.com/show_bug.cgi?id=579095
Comment 10 Vladimir Botka 2010-07-15 12:36:57 UTC
Here I quoted the info from [1]:

"The fix is known and has been submitted to the 2.6.34 stable archive, and it
will be present in future updates of the 11.3 kernel; however, the distribution
media have the problem. Add Live CD, DVD and NET install CDs for i586/i686 are
affected. The media for x86_64 are OK.

This bug does not log any errors - the system just freezes. The workaround is
to add "brokenmodules=ssb" to the boot options line.

Once you have installed the system on the computer, a patched copy of ssb.ko
for the 2.6.34-12-desktop kernel is available at html://lwfinger.net/ssb.ko."

[1] https://bugzilla.novell.com/show_bug.cgi?id=621481#c0
Comment 11 Vladimir Botka 2010-07-15 12:39:05 UTC
*** Bug 621481 has been marked as a duplicate of this bug. ***
Comment 12 Greg Kroah-Hartman 2010-07-16 10:05:13 UTC
Fix now checked into the tree, should show up in next release.
Comment 13 Forgotten User CHC7OcAR44 2010-08-06 14:38:50 UTC
*** Bug 592989 has been marked as a duplicate of this bug. ***
Comment 14 Forgotten User CHC7OcAR44 2010-08-06 15:15:47 UTC
*** Bug 607505 has been marked as a duplicate of this bug. ***
Comment 15 Forgotten User CHC7OcAR44 2010-08-07 09:53:38 UTC
(In reply to comment #12)
> Fix now checked into the tree, should show up in next release.

How can I verify on my laptop before the next release ?

I currently have 11.3 rc2 installed.  When I replace there ssb.ko by html://lwfinger.net/ssb.ko, the kernel boots fine, but I cannot anymore load b43.ko;  I get :

[43079.180537] b43: disagrees about version of symbol ssb_device_is_enabled
[43079.180548] b43: Unknown symbol ssb_device_is_enabled
[43079.180809] b43: disagrees about version of symbol ssb_pmu_set_ldo_paref
[43079.180812] b43: Unknown symbol ssb_pmu_set_ldo_paref
[43079.181030] b43: disagrees about version of symbol ssb_pcicore_dev_irqvecs_enable
[43079.181034] b43: Unknown symbol ssb_pcicore_dev_irqvecs_enable
[43079.181799] b43: disagrees about version of symbol ssb_bus_may_powerdown
[43079.181805] b43: Unknown symbol ssb_bus_may_powerdown
[43079.181986] b43: disagrees about version of symbol ssb_dma_free_consistent
[43079.181989] b43: Unknown symbol ssb_dma_free_consistent
[43079.182439] b43: disagrees about version of symbol ssb_bus_suspend
[43079.182445] b43: Unknown symbol ssb_bus_suspend
[43079.182655] b43: disagrees about version of symbol ssb_bus_unregister
[43079.182658] b43: Unknown symbol ssb_bus_unregister
[43079.183055] b43: disagrees about version of symbol ssb_bus_resume
[43079.183058] b43: Unknown symbol ssb_bus_resume
[43079.183194] b43: disagrees about version of symbol ssb_set_devtypedata
[43079.183196] b43: Unknown symbol ssb_set_devtypedata
[43079.183621] b43: disagrees about version of symbol ssb_device_disable
[43079.183625] b43: Unknown symbol ssb_device_disable
[43079.183699] b43: disagrees about version of symbol ssb_pmu_set_ldo_voltage
[43079.183702] b43: Unknown symbol ssb_pmu_set_ldo_voltage
[43079.183887] b43: disagrees about version of symbol ssb_dma_alloc_consistent
[43079.183890] b43: Unknown symbol ssb_dma_alloc_consistent
[43079.184010] b43: disagrees about version of symbol ssb_dma_set_mask
[43079.184014] b43: Unknown symbol ssb_dma_set_mask
[43079.184704] b43: disagrees about version of symbol ssb_device_enable
[43079.184708] b43: Unknown symbol ssb_device_enable
[43079.184900] b43: disagrees about version of symbol ssb_driver_unregister
[43079.184903] b43: Unknown symbol ssb_driver_unregister
[43079.185159] b43: disagrees about version of symbol __ssb_driver_register
[43079.185164] b43: Unknown symbol __ssb_driver_register
[43079.185488] b43: disagrees about version of symbol ssb_bus_sdiobus_register
[43079.185492] b43: Unknown symbol ssb_bus_sdiobus_register
[43079.185571] b43: disagrees about version of symbol ssb_bus_pcmciabus_register
[43079.185574] b43: Unknown symbol ssb_bus_pcmciabus_register
[43079.185647] b43: disagrees about version of symbol ssb_bus_powerup
[43079.185650] b43: Unknown symbol ssb_bus_powerup
[43079.186720] b43: disagrees about version of symbol ssb_dma_translation
[43079.186724] b43: Unknown symbol ssb_dma_translation

looking further, ssb.ko from lwfinger's site is very bigger than 11.3 rc2's one

-rw-r--r-- 1 root root 1228279 Aug  6 23:12 ssb.ko-dubious
-rw-r--r-- 1 root root   78636 Jun 29 19:56 ssb.ko-orig

and here is my kernel :
kernel-desktop-2.6.34-12.1.i586
Comment 16 Clark Tompsett 2010-08-07 14:41:39 UTC
If your referring to the patches from the 2.5.35 kernel, they don't work on my HP 311 with the 4312 card.
Comment 17 Vladimir Botka 2010-08-07 15:27:46 UTC
(In reply to comment #15)
> How can I verify on my laptop before the next release ?
> I currently have 11.3 rc2 installed.  When I replace there ssb.ko by
> html://lwfinger.net/ssb.ko, the kernel boots fine, but I cannot anymore load
> b43.ko;

43xx wlan adapters from Broadcom like [1] need the wl driver. This driver is not included in the kernel but you can install the broadcom-wl and broadcom-wl-kmp-* packages from the "packman" community repository. You can also compile the wl driver from the source downloaded from [2] . The broadcom-wl package includes configuration that resolves the dependencies of b43 and bss.

You can check with "lspci" what PCI IDs wlan adapter you have and take a look in the README.txt on [2] whether your adapter is supported by the wl driver. 

[1] Broadcom BCM4312 802.11b/g (0x14e4:0x4315)
[2] http://www.broadcom.com/support/802.11/linux_sta.php
Comment 18 Larry Finger 2010-08-07 16:12:10 UTC
The above statement about the BCM4312 is NOT true. That adapter has been supported by b43 since 2.6.32. The only exception are some netbook system that can fail with 2 possible symptoms: (1) they get DMA failures, or (2) the system freezes on load of ssb. The second problem is the one seen by the OP. It has been fixed in mainline for 2.6.36, and the necessary patches have been backported to kernel 2.6.34.3. As a result, they will soon be in the openSUSE kernel.

The usage of out of kernel drivers should not be recommended unless you understand exactly why they are needed!
Comment 19 Forgotten User CHC7OcAR44 2010-08-07 16:30:02 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > How can I verify on my laptop before the next release ?
> > I currently have 11.3 rc2 installed.  When I replace there ssb.ko by
> > html://lwfinger.net/ssb.ko, the kernel boots fine, but I cannot anymore load
> > b43.ko;
> 
> 43xx wlan adapters from Broadcom like [1] need the wl driver. This driver is
> not included in the kernel but you can install the broadcom-wl and
> broadcom-wl-kmp-* packages from the "packman" community repository. You can
> also compile the wl driver from the source downloaded from [2] . The
> broadcom-wl package includes configuration that resolves the dependencies of
> b43 and bss.
> 
> You can check with "lspci" what PCI IDs wlan adapter you have and take a look
> in the README.txt on [2] whether your adapter is supported by the wl driver. 

02:00.0 Network controller: Broadcom Corporation BCM4312 802.11b/g LP-PHY (rev 01)
02:00.0 0280: 14e4:4315 (rev 01)

Well, when I use a 2.6.35-rc3 vanilla kernel, my kernel boots half the time;
the other time it crashes when booting, after some ssb messages.  The last message is not always the same, but in the last messages I see 4 ssb messages.

When my kernel boots, my b43 interface is usable enough to get wlan connectivity.  Injection and monitor mode do not work well, but that's another story.

I have backported the 'fixed' ssb from 2.6.34.2 into 2.6.34-12.3 (include/linux/ssb & drivers/ssb only), and now modprobe ssb does not anymore
halt silently my kernel, but still crashes with messages :

[ 3819.580158] b43-pci-bridge 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 3819.580199] b43-pci-bridge 0000:02:00.0: setting latency timer to 64
[ 3819.588357] ssb: Core 0 found: ChipCommon (cc 0x800, rev 0x16, vendor 0x4243)
[ 3819.588382] ssb: Core 1 found: IEEE 802.11 (cc 0x812, rev 0x0F, vendor 0x4243)
[ 3819.588408] ssb: Core 2 found: PCMCIA (cc 0x80D, rev 0x0A, vendor 0x4243)
[ 3819.588439] ssb: Core 3 found: PCI-E (cc 0x820, rev 0x09, vendor 0x4243)
[ 3819.596305] BUG: unable to handle kernel paging request at 18600010
[ 3819.596318] IP: [<f7d6f4aa>] ssb_is_sprom_available+0x1a/0x30 [ssb]
[ 3819.596341] *pdpt = 000000002fa76001 *pde = 0000000000000000 
[ 3819.596352] Oops: 0000 [#1] PREEMPT SMP 
[ 3819.596363] last sysfs file: /sys/bus/ssb/uevent
[ 3819.596369] Modules linked in: ssb(+) ip6t_LOG xt_tcpudp xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device coretemp edd nfs lockd fscache nfs_acl auth_rpcgss sunrpc af_packet cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq speedstep_lib mperf ip6t_REJECT nf_conntrack_ipv6 ip6table_raw xt_NOTRACK ipt_REJECT xt_state iptable_raw iptable_filter ip6table_mangle nf_conntrack_netbios_ns nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ip_tables ip6table_filter ip6_tables x_tables nls_iso8859_1 nls_cp437 vfat fat fuse loop dm_mod snd_hda_codec_atihdmi snd_hda_codec_idt firewire_ohci firewire_core crc_itu_t snd_hda_intel snd_hda_codec snd_hwdep hp_wmi snd_pcm uvcvideo pcmcia iTCO_wdt ohci1394 snd_timer rfkill sr_mod cdrom sg videodev v4l1_compat jmb38x_ms sdhci_pci sdhci hp_accel snd joydev lirc_ene0100 lirc_dev soundcore ieee1394 memstick mmc_core pcspkr pcmcia_core iTCO_vendor_support i2c_i801 wmi ac video snd_page_alloc battery lis3lv02d 
input_polldev button ext4 jbd2 crc16 radeon ttm drm_kms_helper drm i2c_algo_bit intel_agp netconsole configfs r8169 fan processor thermal thermal_sys [last unloaded: preloadtrace]
[ 3819.597039] 
[ 3819.597039] Pid: 8448, comm: insmod Tainted: G        W  2.6.34-12-desktop #1 3628/HP Pavilion dv6 Notebook PC
[ 3819.597039] EIP: 0060:[<f7d6f4aa>] EFLAGS: 00210206 CPU: 0
[ 3819.597039] EIP is at ssb_is_sprom_available+0x1a/0x30 [ssb]
[ 3819.597039] EAX: f2907c00 EBX: f2907c00 ECX: 18600000 EDX: 00000001
[ 3819.597039] ESI: f7d702c0 EDI: f1a59df0 EBP: f2907c00 ESP: f1a59d44
[ 3819.597039]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[ 3819.597039] Process insmod (pid: 8448, ti=f1a58000 task=f2c55300 task.ti=f1a58000)
[ 3819.597039] Stack:
[ 3819.597039]  f7d702d2 c056a056 00000080 f1a59d70 f1a59d94 00000000 f2907c00 f7d702c0
[ 3819.597039] <0> f1a59df0 00000000 f7d6dba0 00000000 00000000 00000000 00000000 00000000
[ 3819.597039] <0> 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3819.597039] Call Trace:
[ 3819.597039]  [<f7d702d2>] ssb_pci_get_invariants+0x12/0x580 [ssb]
[ 3819.597039]  [<f7d6dba0>] ssb_fetch_invariants+0x30/0x90 [ssb]
[ 3819.597039]  [<f7d6dcce>] ssb_bus_register+0xce/0x190 [ssb]
[ 3819.597039]  [<f7d6de7a>] ssb_bus_pcibus_register+0x2a/0x60 [ssb]
[ 3819.597039]  [<f7d70a80>] ssb_pcihost_probe+0xb0/0x140 [ssb]
[ 3819.597039]  [<c040dead>] __pci_device_probe+0xdd/0xf0
[ 3819.597039]  [<c040ec13>] pci_device_probe+0x23/0x40
[ 3819.597039]  [<c04a1e5d>] really_probe+0x5d/0x200
[ 3819.597039]  [<c04a21ce>] driver_probe_device+0x3e/0xa0
[ 3819.597039]  [<c04a22a9>] __driver_attach+0x79/0x80
[ 3819.597039]  [<c04a0fcb>] bus_for_each_dev+0x4b/0x70
[ 3819.597039]  [<c04a1cd6>] driver_attach+0x16/0x20
[ 3819.597039]  [<c04a18f7>] bus_add_driver+0x1c7/0x330
[ 3819.597039]  [<c04a248f>] driver_register+0x5f/0x100
[ 3819.597039]  [<c040ee2d>] __pci_register_driver+0x3d/0xb0
[ 3819.597039]  [<f7d7d041>] ssb_modinit+0x41/0x58 [ssb]
[ 3819.597039]  [<c020121f>] do_one_initcall+0x2f/0x190
[ 3819.597039]  [<c027a771>] sys_init_module+0xb1/0x220
[ 3819.597039]  [<c020324c>] sysenter_do_call+0x12/0x22
[ 3819.597039]  [<ffffe424>] 0xffffe424
[ 3819.597039] Code: 90 a1 ec 7b d7 f7 c3 8d 76 00 8d bc 27 00 00 00 00 83 78 14 01 ba 01 00 00 00 74 05 89 d0 c3 66 90 8b 88 48 02 00 00 85 c9 74 f1 <80> 79 10 1e 76 eb f6 80 4f 02 00 00 40 0f 95 c2 89 d0 c3 90 90 
[ 3819.597039] EIP: [<f7d6f4aa>] ssb_is_sprom_available+0x1a/0x30 [ssb] SS:ESP 0068:f1a59d44
[ 3819.597039] CR2: 0000000018600010
[ 3819.612153] ---[ end trace 4eaa2a86a8e2da24 ]---

> 
> [1] Broadcom BCM4312 802.11b/g (0x14e4:0x4315)
> [2] http://www.broadcom.com/support/802.11/linux_sta.php
Comment 20 Forgotten User CHC7OcAR44 2010-08-07 16:37:08 UTC
(In reply to comment #18)
> The above statement about the BCM4312 is NOT true. That adapter has been
> supported by b43 since 2.6.32. The only exception are some netbook system that
> can fail with 2 possible symptoms: (1) they get DMA failures, or (2) the system
> freezes on load of ssb. The second problem is the one seen by the OP. It has
> been fixed in mainline for 2.6.36, and the necessary patches have been
> backported to kernel 2.6.34.3. As a result, they will soon be in the openSUSE
> kernel.

That's exactly what I have : sometimes boot works, often boot freezes;
when boot works, b43 is usable.  Sometimes it falls in DMA error (not directly,
only after some work time).

Glad to hear that the fixes have been backported to 2.6.34.3. I'd be glad to test them here.  can you point me to them ?

> 
> The usage of out of kernel drivers should not be recommended unless you
> understand exactly why they are needed!
Comment 21 Larry Finger 2010-08-07 17:09:04 UTC
That kernel source is at kernel.org. A kernel with those patches has not yet been generated for openSUSE. You would need to build it from source.
Comment 22 Forgotten User CHC7OcAR44 2010-08-07 17:22:41 UTC
(In reply to comment #21)
> That kernel source is at kernel.org. A kernel with those patches has not yet
> been generated for openSUSE. You would need to build it from source.

I have no problem building from sources, but I pulled from 2.6.34.y and the last tag is 2.6.34.2.

Once the patches are there, is it enough to take drivers/ssb, include/linux/ssb and drivers/net/wireless/b43 ?
Comment 23 Forgotten User CHC7OcAR44 2010-08-07 17:30:50 UTC
I wrote :

> [ 3819.596305] BUG: unable to handle kernel paging request at 18600010
> [ 3819.596318] IP: [<f7d6f4aa>] ssb_is_sprom_available+0x1a/0x30 [ssb]
> [ 3819.596341] *pdpt = 000000002fa76001 *pde = 0000000000000000 
> [ 3819.596352] Oops: 0000 [#1] PREEMPT SMP 
> [ 3819.596363] last sysfs file: /sys/bus/ssb/uevent

This did not happen again.  I now installed ssb.ko and b43.ko from a wholly recompiled opensuse 2.6.34-12 with ssb from stable 2.6.34.2, and the modules are loaded perfectly.  I cannot test if it works, though, and b43 hwaddr is different from the one I get with 2.6.35-rc3.

I'll retry copying also b43 sources from stable 2.6.34.2.
Comment 24 Larry Finger 2010-08-07 17:44:29 UTC
Be careful. There is a different bug in 2.6.34.2 in the ssb sources. You should use 2.6.34.3.

I have no idea why 2.6.34.y.git has not been updated. From GregKH's emails, I thought it should also be at 2.6.34.3
Comment 25 Forgotten User CHC7OcAR44 2010-08-07 18:18:12 UTC
(In reply to comment #24)
> Be careful. There is a different bug in 2.6.34.2 in the ssb sources. You should
> use 2.6.34.3.
> 
> I have no idea why 2.6.34.y.git has not been updated. From GregKH's emails, I
> thought it should also be at 2.6.34.3

OK, I'll wait till 2.6.34.3 appears
Comment 26 Larry Finger 2010-08-07 18:36:51 UTC
I had misunderstood the emails I got. The patches are in 2.6.34.3-rc1. The patch set from 2.6.34.2 are at http://www.kernel.org/pub/linux/kernel/v2.6/stable-review/patch-2.6.34.3-rc1.bz2
Comment 27 Clark Tompsett 2010-08-09 19:33:29 UTC
I have tried the 2.6.35 kernel (default and desktop) from HEAD.  The desktop kernel will not load b43 at all.  The default kernel will load, sometimes, with options for b43 and brokenmodule=ssb used at boot time.  Even with the broken module entry, ssb is loaded.
Comment 28 Forgotten User CHC7OcAR44 2010-08-10 08:39:12 UTC
(In reply to comment #27)
> I have tried the 2.6.35 kernel (default and desktop) from HEAD.  The desktop
> kernel will not load b43 at all.  The default kernel will load, sometimes, with
> options for b43 and brokenmodule=ssb used at boot time.  Even with the broken
> module entry, ssb is loaded.

I have recompiled b43.ko and ssb.ko after having copied the include/linux/ssb and drivers/ssb directories from stable 2.6.34.3 (actually 2.6.34.2 + http://www.kernel.org/pub/linux/kernel/v2.6/stable-review/patch-2.6.34.3-rc1.bz2) into opensuse's 2.6.34-12.3 sources, and I do not have ssb boot problems anymore, and b43 works.
Comment 29 Swamp Workflow Management 2010-08-24 15:23:11 UTC
The SWAMPID for this issue is 35398.
This issue was rated as moderate.
Please submit fixed packages until 2010-09-07.
When done, please reassign the bug to security-team@suse.de.
Patchinfo will be handled by security team.
Comment 30 Swamp Workflow Management 2010-09-08 13:08:38 UTC
Update released for: kernel-debug, kernel-debug-base, kernel-debug-base-debuginfo, kernel-debug-debuginfo, kernel-debug-debugsource, kernel-debug-devel, kernel-debug-devel-debuginfo, kernel-default, kernel-default-base, kernel-default-base-debuginfo, kernel-default-debuginfo, kernel-default-debugsource, kernel-default-devel, kernel-default-devel-debuginfo, kernel-desktop, kernel-desktop-base, kernel-desktop-base-debuginfo, kernel-desktop-debuginfo, kernel-desktop-debugsource, kernel-desktop-devel, kernel-desktop-devel-debuginfo, kernel-devel, kernel-ec2-devel, kernel-pae, kernel-pae-base, kernel-pae-base-debuginfo, kernel-pae-debuginfo, kernel-pae-debugsource, kernel-pae-devel, kernel-pae-devel-debuginfo, kernel-source, kernel-source-vanilla, kernel-syms, kernel-trace, kernel-trace-base, kernel-trace-base-debuginfo, kernel-trace-debuginfo, kernel-trace-debugsource, kernel-trace-devel, kernel-trace-devel-debuginfo, kernel-vanilla, kernel-vanilla-base, kernel-vanilla-base-debuginfo, kernel-vanilla-debuginfo, kernel-vanilla-debugsource, kernel-vanilla-devel, kernel-vanilla-devel-debuginfo, kernel-vmi-devel, kernel-xen, kernel-xen-base, kernel-xen-base-debuginfo, kernel-xen-debuginfo, kernel-xen-debugsource, kernel-xen-devel, kernel-xen-devel-debuginfo, preload-kmp-default, preload-kmp-desktop
Products:
openSUSE 11.3 (debug, i586, x86_64)