Bug 155356

This looks like a strange PCI problem to me. The card doesn't even
show up in lspci output on 10.1, while it does show up on older kernels.

Greg, can you have a look please?

Israel, could you please run the hwinfo command as root and attach
the output here?

Concerning your attempt to compile Broadcom's driver on 10.1 - sorry, we're
not even trying to support that. Please ask Broadcom, it's their driver and
they should be able to help you. But most likely it's just missing an
#include <linux/in.h>

Created attachment 71314 [details]
output of hwinfo on suse 10.1 beta6 on the discussed machin.

the relevant section from hwinfo on suse 10.0 - same machin:
42: PCI a09.0: 0200 Ethernet controller
  [Created at pci.277]
  UDI: /org/freedesktop/Hal/devices/pci_14e4_1648_0
  Unique ID: mY_N.naPqmzBs9CA
  Parent ID: 6PqB.qlvrYsPk2G3
  SysFS ID: /devices/pci0000:08/0000:08:0b.0/0000:0a:09.0
  SysFS BusID: 0000:0a:09.0
  Hardware Class: network
  Model: "Broadcom NetXtreme BCM5704 Gigabit Ethernet"
  Vendor: pci 0x14e4 "Broadcom"
  Device: pci 0x1648 "NetXtreme BCM5704 Gigabit Ethernet"
  SubVendor: pci 0x14e4 "Broadcom"
  SubDevice: pci 0x1644
  Revision: 0x03
  Driver: "tg3"
  Device File: eth2
  Memory Range: 0xdc410000-0xdc41ffff (rw,non-prefetchable)
  Memory Range: 0xdc400000-0xdc40ffff (rw,non-prefetchable)
  Memory Range: 0x80100000-0x8010ffff (ro,prefetchable,disabled)
  IRQ: 217 (3282334 events)
  HW Address: 00:e0:81:34:41:b2
  Link detected: yes
  Module Alias: "pci:v000014E4d00001648sv000014E4sd00001644bc02sc00i00"
  Driver Info #0:
    Driver Status: tg3 is active
    Driver Activation Cmd: "modprobe tg3"
  Driver Info #1:
    Driver Status: bcm5700 is not active
    Driver Activation Cmd: "modprobe bcm5700"
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #39 (PCI bridge)

43: PCI a09.1: 0200 Ethernet controller
  [Created at pci.277]
  UDI: /org/freedesktop/Hal/devices/pci_14e4_1648
  Unique ID: DkES.naPqmzBs9CA
  Parent ID: 6PqB.qlvrYsPk2G3
  SysFS ID: /devices/pci0000:08/0000:08:0b.0/0000:0a:09.1
  SysFS BusID: 0000:0a:09.1
  Hardware Class: network
  Model: "Broadcom NetXtreme BCM5704 Gigabit Ethernet"
  Vendor: pci 0x14e4 "Broadcom"
  Device: pci 0x1648 "NetXtreme BCM5704 Gigabit Ethernet"
  SubVendor: pci 0x14e4 "Broadcom"
  SubDevice: pci 0x1644
  Revision: 0x03
  Driver: "tg3"
  Device File: eth3
  Memory Range: 0xdc430000-0xdc43ffff (rw,non-prefetchable)
  Memory Range: 0xdc420000-0xdc42ffff (rw,non-prefetchable)
  Memory Range: 0x80110000-0x8011ffff (ro,prefetchable,disabled)
  IRQ: 225 (no events)
  HW Address: 00:e0:81:34:41:b3
  Link detected: no
  Module Alias: "pci:v000014E4d00001648sv000014E4sd00001644bc02sc00i00"
  Driver Info #0:
    Driver Status: tg3 is active
    Driver Activation Cmd: "modprobe tg3"
  Driver Info #1:
    Driver Status: bcm5700 is not active
    Driver Activation Cmd: "modprobe bcm5700"
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #39 (PCI bridge)

Hm, that attachment seems to not include everything.

Care to try it again?

Also, the output of 'lspci -v' from this kernel, and the older 2.6.13 kernel
would be very helpful to have.

I will do that shortly. However, the output of hwinfo is very long, and what is left in my konsule is without the early lines. Any idea how to "freez" them?
Israel

Write it into a file and compress it:
hwinfo > myhwinfo
gzip myhwinfo
and upload myhwinfo.gz as attachment here.

No, don't compress it :(
just attach the uncompressed file if you can, makes it much easier...

Created attachment 71470 [details]
lspci -v while booted with kernel 2.6.16

Created attachment 71471 [details]
lspci -v when booted with kernel 2.6.13-gentoo-r2

Created attachment 71477 [details]
myhwinfo kernel 2.6.13

Created attachment 71478 [details]
myhwinfo kernel 2.6.16

This may be another issue with MMCONFIG. Could you please try
the latest kernel of the day and see if that works any
better? We disabled mmconfig on these. Thanks!

I have installed the new kernel - no change inresults. here is output of some diagnostics:
Linux gibooi 2.6.16-4-smp #1 SMP Thu Mar 23 00:28:25 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux
luli_b@gibooi:~> su -
Password:
gibooi:~ # ifconfig -a
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:66 errors:0 dropped:0 overruns:0 frame:0
          TX packets:66 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4510 (4.4 Kb)  TX bytes:4510 (4.4 Kb)

sit0      Link encap:IPv6-in-IPv4
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

gibooi:~ # /etc/init.d/network restart
Shutting down network interfaces:
Shutting down service network  .  .  .  .  .  .  .  .  .  .  .  .  .  done
Hint: you may set mandatory devices in /etc/sysconfig/network/config
Setting up network interfaces:
    lo
    lo        IP address: 127.0.0.1/8                                 done
Setting up service network  .  .  .  .  .  .  .  .  .  .  .  .  .  .  done
gibooi:~ # lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:07.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
gibooi:~ #         
gibooi:~ # lsmod | grep tg3
tg3                   115204  0
gibooi:~ #          

I have also tried ubuntu dapper with kernel 2.6.15 and gentoo with kernel 2.6.15 on the same machine. Tg3 failed on both (but worked all right with kernels lower than 2.6.14. So it seems to be somthing on the kernel itself and particulare to suse's version of it.
Israel

I have made extensive use of the tg3.c driver (v3.49) in all of the SLES10 beta
from 1 to 8 and have not had any problems with 5704 devices.  We have some
issues with other Broadcom devices but 5704 (and also 5701) have been working
just fine for us here at SGI on our IA64 platforms.  I invest many hours
of testing and qualification of this driver so you will hear from us if
there are any issues as well.  As of now, I have no issues to report.

We will soon potentially request that the SLES10 tg3.c driver be advanced
from v3.49 to a newer one.  However we do not yet know what version that
should be.  There are some features in later drivers that would be useful
to us (such as ability to query the firmware version running on the 57xx
devices) but we need additional testing and exposure before we can commit
to a newer version.

You should not attempt to use the Broadcom developed driver any longer.  Broadcom has abandoned support for their own driver and now put all of their efforts behind tg3.c.  tg3.c is an extremely high quality driver and we are getting a lot of help and support in the community from Michael Chan at Broadcom to keep the driver at a very high quality and performance level.

Chris Elmquist
SGI Network Engineering

SGI, you are going to ask for what?  Please, use the proper channels
to ask for such an upgrade, and please note that it is _very_ late to
be asking for something like that.

Also, I don't see where you are seeing that anyone was using the broadcom
driver in this bug report, are you sure you entered this into the proper
one?

Regarding the Broadcom driver, in the description it stated:

>finally, I have tried to compile  Broadcome's module on suse 10.1 beta6 and
>this is what i got:
>
>luli_b@gibooi:~> su -
>Password:
>gibooi:~ # cd /home/luli_b/tg3-343f/tg3-3.43f
>gibooi:/home/luli_b/tg3-343f/tg3-3.43f # make
>make -C /lib/modules/2.6.16-rc5-git2-2-smp/build
>SUBDIRS=/home/luli_b/tg3-343f/tg3-3.43f modules
[etc]

Comment #1 also said,

>Concerning your attempt to compile Broadcom's driver on 10.1 - sorry, we're
>not even trying to support that. Please ask Broadcom, it's their driver and
>they should be able to help you. But most likely it's just missing an
>#include <linux/in.h>

and my point was that asking Broadcom for help compiling their old driver
on these new kernels will not be a productive exercise.

Regarding SGI requesting an update to the tg3.c driver, that will absolutely
go through official channels and I probably should not have mentioned it here.
What I wanted to convey was that there are some updates to the tg3.c driver
that offer some features that v3.49 does not have.

We have not formally requested this advancement of the driver yet because
there are issues with other Broadcom devices (such as 5714) which are
not yet resolved in these later drivers.  So, it would not be good to request
these updates yet.

We understand that the window is closing and if need be, we will have to
accomodate it in some form of update later.

Created attachment 78096 [details]
Diff between 2.6.13 and 2.6.16 lspci output

Here's a diff between your lspci outputs showing that
the kernel is just not even detecting a whole bunch
of PCI devices due to some issue.

Including it here to make it easier for others to see it.

Ick, it looks like this might be a mmcfg issue.  Are you sure the
BIOS on this board is the latest version?  If not, please upgrade it, as there
might be a bug there.

The big change between 2.6.13 and newer kernels is that we do the PCI probing by
asking the hardware for what it has, before we trusted the BIOS, but that didn't
always work.  But for some systems, they did not report their devices properly,
so things like not seeing whole PCI busses happen (like this machine.)

Usually a BIOS upgrade fixes this.

If not, then can you try the latest Kernel of the day from our ftp site?  It
has some mmcfg fixes that should handle this issue.

Thanks for your patience and testing.

Solved!!!
after upgrading to 10.1RC1 all hardware is detected and functions right.
Here is a summery:
 last paragraphs of dmesg:

NET: Registered protocol family 17
tg3: eth2: Link is up at 100 Mbps, full duplex.
tg3: eth2: Flow control is on for TX and on for RX.
ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
eth2: no IPv6 routers present
BIOS EDD facility v0.16 2004-Jun-25, 4 devices found
ADDRCONF(NETDEV_UP): eth2: link is not ready
tg3: eth2: Link is up at 100 Mbps, full duplex.
tg3: eth2: Flow control is on for TX and on for RX.
ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
powernow-k8: Found 4 AMD Athlon 64 / Opteron processors (version 1.60.2)
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
eth2: no IPv6 routers present
end_request: I/O error, dev fd0, sector 0
end_request: I/O error, dev fd0, sector 0
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sdc1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
gibooi:~ # 
gibooi:~ # lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
01:07.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
08:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
08:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
08:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
08:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X IOAPIC (rev 01)
09:08.0 RAID bus controller: 3ware Inc 9550SX SATA-RAID
0a:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03)
0a:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03)
gibooi:~ #  
gibooi:~ # uname -a
Linux gibooi 2.6.16-20-smp #1 SMP Mon Apr 10 04:51:13 UTC 2006 x86_64 x86_64 x86_64 GNU/Linux
gibooi:~ #          
Since both the ethernet controlers and the 3ware card are detected now it is plausible that the mmcfg fixes did the trick (I am quiet realactant to touch the Bios...)
Thank you all very much!
Israel

Great, thanks for letting us know and for testing.