David Xu wrote:
> Do you have patch for i386 branch ? I want to test.
> On my Pentium-D machine:
>
> $ sysctl kern.sched.topology_spec
> kern.sched.topology_spec:
>
> 0, 1
>
>
>
> 0, 1
>
>
>
>
>
>
> it seems the kernel thinks that the Pentuim-D is sharing L2 cache,
> but in fact, the design of the Pentium Ds was simply two P4 cores
> sitting side by side. They do not share anything and they basically work
> independently.

Same thing on my intel Atom 330 board at home:



0, 1, 2, 3



0, 1, 2, 3



0, 1
HTT group



2, 3
HTT group








The intel Atom 330 consists of two "Diamondville" dies on
one package, each with its own 512 KB of L2 cache. There
is no L3 cache, or any other shared cache.

(Or maybe I misinterpret the XML output; I think that the
"level" and "cache-level" numbers are confusing.)

BTW, I noticed that the indentation problem (newline before
"") is already fixed in -current, 5 weeks ago.
Any plan to MFC this?

Best regards
Oliver

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Gesch�ftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M�n-
chen, HRB 125758, Gesch�ftsf�hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd

"Python tricks" is a tough one, cuz the language is so clean. E.g.,
C makes an art of confusing pointers with arrays and strings, which
leads to lotsa neat pointer tricks; APL mistakes everything for an
array, leading to neat one-liners; and Perl confuses everything
period, making each line a joyous adventure .
-- Tim Peters
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On Thu, Jul 15, 2010 at 10:01:48PM +0200, Oliver Fromme wrote:
>
> Jung-uk Kim wrote:
> > On Thursday 15 July 2010 01:56 pm, Andriy Gapon wrote:
> > > on 15/07/2010 19:57 Oliver Fromme said the following:
> > > > I patched topo_probe() so it calls topo_probe_0x4() after
> > > > topo_probe_0xb() if cpu_cores is still 0. I think this
> > > > is a better fallback procedure. With this patch, cpu_cores
> > > > gets the value 4 which is the correct one, finally:
> > > [...]
> > > I think that your addition achieves this effect, perhaps just not
> > > as explicitly as I would preferred.
> > >
> > > Jung-uk, what do you think?
> >
> > Yes, you're right. Please try new patch:
> >
> > http://people.freebsd.org/~jkim/mp_machdep2.diff
>
> Thank you!
>
> I will have access to that particular machine on Monday again,
> so testing the new patch will have to wait until Monday.
> But from looking at your patch it should have the same result
> as my simpler patch, so it should work fine.

I have a general question for everyone involved in this thread (which is
highly educational/interesting -- thank you for all the info!):

Does the problem reported affect actual performance/behaviour of FreeBSD
kernel-wise at all, or is it just a cosmetical issue with regards to
showing how many cores/threads there are?

--
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |

_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

> On Thursday 15 July 2010 03:07 pm, Jung-uk Kim wrote:
>> On Thursday 15 July 2010 01:56 pm, Andriy Gapon wrote:
>> > on 15/07/2010 19:57 Oliver Fromme said the following:
>> > > In topo_probe(), cpu_high is 0xd, so topo_probe_0xb() is
>> > > called. �But the cpuid 0xb instruction doesn't seem to
>> > > return useful data: �All values are zero already in the
>> > > first level, so cpu_cores remains 0.
>> > >
>> > > Back in topo_probe(), there is a fallback if cpu_cores is
>> > > stil 0: �It assigns mp_ncpu to cpu_cores, so it gets 8
>> > > which is wrong.
>> > >
>> > > I patched topo_probe() so it calls topo_probe_0x4() after
>> > > topo_probe_0xb() if cpu_cores is still 0. �I think this
>> > > is a better fallback procedure. �With this patch, cpu_cores
>> > > gets the value 4 which is the correct one, finally:
>> > >
>> > > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
>> > > FreeBSD/SMP: 2 package(s) x 4 core(s)
>> >
>> > Thank you for debugging this issue!
>> > Not sure if this is the best patch that there can be, but its
>> > direction is definitely correct.
>> > As the Intel document says (translated to our x86 mp_machdep.c
>> > terms): if cpu_high >= 0xb then we should execute
>> > cpuid_count(0xb, 0, p) and examine EBX value (p[1]), only if it's
>> > non-zero should we proceed with topo_probe_0xb(), otherwise we
>> > should fall back to topo_probe_0x4, etc.
>> >
>> > I think that your addition achieves this effect, perhaps just not
>> > as explicitly as I would preferred.
>> >
>> > Jung-uk, what do you think?
>>
>> Yes, you're right. �Please try new patch:
>>
>> http://people.freebsd.org/~jkim/mp_machdep2.diff
>
> I uploaded the patch again, it's compile-tested this now.
>

Just tried with the patch against 8.1-rc2.

2x E5520 - OK, no changes
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads

2x E5440 - now OK
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)

1x 5050 - OK, no changes
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads

--
wbr,
pluknet
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On Thursday 15 July 2010 09:34 pm, David Xu wrote:
> Jung-uk Kim wrote:
> > On Wednesday 14 July 2010 05:40 pm, Jung-uk Kim wrote:
> >> On Wednesday 14 July 2010 01:31 pm, Andriy Gapon wrote:
> >>> on 14/07/2010 17:14 Oliver Fromme said the following:
> >>>> In a machine installed yesterday, 8.1-PRERELEASE doesn't
> >>>> seem to detect the number of CPU packages vs. cores per
> >>>>
> >>>> package correctly:
> >>>> | FreeBSD 8.1-PRERELEASE-20100713 #0: Tue Jul 13 19:51:18 UTC
> >>>> | 2010 [...]
> >>>> | (2133.42-MHz K8-class CPU) Origin = "GenuineIntel" Id =
> >>>> | 0x1067a Family = 6 Model = 17 Stepping = 10
> >>>> | Features=0xbfebfbff > >>>> |, SE
> >>>> | P,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE
> >>>> |, SSE 2,SS,HTT,TM,PBE>
> >>>> | Features2=0x40ce3bd > >>>> |3 ,C X16,xTPR,PDCM,DCA,SSE4.1,XSAVE> AMD
> >>>> | Features=0x20000800
> >>>> | AMD Features2=0x1
> >>>> | TSC: P-state invariant
> >>>> | real memory = 34359738368 (32768 MB)
> >>>> | avail memory = 33151377408 (31615 MB)
> >>>> | ACPI APIC Table:
> >>>> | FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
> >>>> | FreeBSD/SMP: 1 package(s) x 8 core(s)
> >>>> | cpu0 (BSP): APIC ID: 0
> >>>> | cpu1 (AP): APIC ID: 1
> >>>> | cpu2 (AP): APIC ID: 2
> >>>> | cpu3 (AP): APIC ID: 3
> >>>> | cpu4 (AP): APIC ID: 4
> >>>> | cpu5 (AP): APIC ID: 5
> >>>> | cpu6 (AP): APIC ID: 6
> >>>> | cpu7 (AP): APIC ID: 7
> >>>> | ioapic1 irqs 24-47 on motherboard
> >>>> | ioapic0 irqs 0-23 on motherboard
> >>>>
> >>>> I'm pretty sure that this is a 2 x 4 machine (2 CPU packages
> >>>> with 4 cores per package), not 1 x 8. That's what the BIOS
> >>>> displays during POST.
> >>>>
> >>>> I'm not sure if this is just a "cosmetic" issue, or if this
> >>>> is a critical thing ... I could imagine that performance
> >>>> might be sub-optimal if the CPU topology isn't detected
> >>>> correctly, but I'm not sure if FreeBSD can take advantage
> >>>> of the topology.
> >>>
> >>> Could you please try to do the following?
> >>> 1. Fetch topo-12212009.tar from the top of this page:
> >>> http://software.intel.com/en-us/articles/intel-64-architecture-
> >>>pr oc essor-topology-enumeration/ 2. Untar it and apply this
> >>> patch to the code:
> >>> http://people.freebsd.org/~avg/cpu-topology.diff
> >>> 3. Compile it by running sh mk_64.sh (supposing you have amd64
> >>> system installed) 4. Run cpu_topology64.out and report back its
> >>> output.
> >>
> >> It's funny that I actually wrote a convenience script (and
> >> cleaned up today):
> >>
> >> http://people.freebsd.org/~jkim/cpu_topology-12212009.sh
> >>
> >> BTW, current topology detection code is not optimal for some
> >> Intel processors if my memory serves.
> >
> > Surprisingly, I found a patch I wrote last year from /tmp of my
> > desktop and it is still applied cleanly. 8-)
> >
> > Just in case, it is available from here:
> >
> > http://people.freebsd.org/~jkim/mp_machdep.diff
> >
> > It is applicable to both sys/amd64/amd64/mp_machdep.c and
> > sys/i386/i386/mp_machdep.c. I have to warn you, though, it
> > hasn't been used for more than a year, so it may not work at all.
> > ;-)
> >
> > Jung-uk Kim
>
> Do you have patch for i386 branch ? I want to test.

The patch should apply fine on both sys/amd64/amd64/mp_machdep.c and
sys/i386/i386/mp_machdep.c.

http://people.freebsd.org/~jkim/mp_machdep2.diff

> On my Pentium-D machine:
>
> $ sysctl kern.sched.topology_spec
> kern.sched.topology_spec:
>
> 0, 1
>
>
>
> 0, 1
>
>
>
>
>
>
> it seems the kernel thinks that the Pentuim-D is sharing L2 cache,
> but in fact, the design of the Pentium Ds was simply two P4 cores
> sitting side by side. They do not share anything and they basically
> work independently.

cpu_topo() in mp_machdep.c is not that smart to distinguish whether
two cores in one package is sharing L2 cache or not:

/*
* Only HTT no multi-core.
*/
if (cpu_logical > 1 && cpu_cores == 1)
return (smp_topo_1level(CG_SHARE_L1, cpu_logical, cg_flags));
/*
* Only multi-core no HTT.
*/
if (cpu_cores > 1 && cpu_logical == 1)
return (smp_topo_1level(CG_SHARE_L2, cpu_cores, cg_flags));
/*
* Both HTT and multi-core.
*/
return (smp_topo_2level(CG_SHARE_L2, cpu_cores,
CG_SHARE_L1, cpu_logical, cg_flags));

In other words, if one package contains multiple cores, it is
*assumed* sharing L2 cache. The same is true for HTT/SMT case (i.e.,
assumed sharing L1 cache). It does not care about L3 cache at all.

Yes, we need some improvement in this area.

Jung-uk Kim
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On Friday 16 July 2010 03:55 am, Jeremy Chadwick wrote:
> On Thu, Jul 15, 2010 at 10:01:48PM +0200, Oliver Fromme wrote:
> > Jung-uk Kim wrote:
> > > On Thursday 15 July 2010 01:56 pm, Andriy Gapon wrote:
> > > > on 15/07/2010 19:57 Oliver Fromme said the following:
> > > > > I patched topo_probe() so it calls topo_probe_0x4() after
> > > > > topo_probe_0xb() if cpu_cores is still 0. I think this
> > > > > is a better fallback procedure. With this patch,
> > > > > cpu_cores gets the value 4 which is the correct one,
> > > > > finally:
> > > >
> > > > [...]
> > > > I think that your addition achieves this effect, perhaps
> > > > just not as explicitly as I would preferred.
> > > >
> > > > Jung-uk, what do you think?
> > >
> > > Yes, you're right. Please try new patch:
> > >
> > > http://people.freebsd.org/~jkim/mp_machdep2.diff
> >
> > Thank you!
> >
> > I will have access to that particular machine on Monday again,
> > so testing the new patch will have to wait until Monday.
> > But from looking at your patch it should have the same result
> > as my simpler patch, so it should work fine.
>
> I have a general question for everyone involved in this thread
> (which is highly educational/interesting -- thank you for all the
> info!):
>
> Does the problem reported affect actual performance/behaviour of
> FreeBSD kernel-wise at all, or is it just a cosmetical issue with
> regards to showing how many cores/threads there are?

Theoretically there is behavioral changes from scheduler. jeff@
should be able to tell you more about this.

Jung-uk Kim
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

Oliver Fromme wrote:
> Jung-uk Kim wrote:
> > On Thursday 15 July 2010 01:56 pm, Andriy Gapon wrote:
> > > on 15/07/2010 19:57 Oliver Fromme said the following:
> > > > I patched topo_probe() so it calls topo_probe_0x4() after
> > > > topo_probe_0xb() if cpu_cores is still 0. I think this
> > > > is a better fallback procedure. With this patch, cpu_cores
> > > > gets the value 4 which is the correct one, finally:
> > > [...]
> > > I think that your addition achieves this effect, perhaps just not
> > > as explicitly as I would preferred.
> > >
> > > Jung-uk, what do you think?
> >
> > Yes, you're right. Please try new patch:
> >
> > http://people.freebsd.org/~jkim/mp_machdep2.diff
>
> Thank you!
>
> I will have access to that particular machine on Monday again,
> so testing the new patch will have to wait until Monday.
> But from looking at your patch it should have the same result
> as my simpler patch, so it should work fine.

As expected, the patch works perfectly fine.
The cores are detected correctly:

FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)

kern.sched.topology_spec:

0, 1, 2, 3, 4, 5, 6, 7



0, 1, 2, 3



4, 5, 6, 7






Thanks for fixing this!

Best regards
Oliver

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Gesch�ftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht M�n-
chen, HRB 125758, Gesch�ftsf�hrer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd

Perl is worse than Python because people wanted it worse.
-- Larry Wall
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On 16 July 2010 19:47, Jung-uk Kim <> wrote:
> The patch should apply fine on both sys/amd64/amd64/mp_machdep.c and
> sys/i386/i386/mp_machdep.c.
>
> http://people.freebsd.org/~jkim/mp_machdep2.diff
>


Hi.

Just checked on Xen HVM with 3 cores.
1) 8.1 unmodified:
FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
FreeBSD/SMP: 1 package(s) x 3 core(s)

2) 8.1 + patch
FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
WARNING: Non-uniform processors.
WARNING: Using suboptimal topology.


--
wbr,
pluknet
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

on 10/08/2010 19:55 pluknet said the following:
> On 16 July 2010 19:47, Jung-uk Kim <> wrote:
>> The patch should apply fine on both sys/amd64/amd64/mp_machdep.c and
>> sys/i386/i386/mp_machdep.c.
>>
>> http://people.freebsd.org/~jkim/mp_machdep2.diff
>>
>
>
> Hi.
>
> Just checked on Xen HVM with 3 cores.
> 1) 8.1 unmodified:
> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> FreeBSD/SMP: 1 package(s) x 3 core(s)
>
> 2) 8.1 + patch
> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
> WARNING: Non-uniform processors.
> WARNING: Using suboptimal topology.

Can you debug, e.g. with printfs, what exactly goes wrong?
I wonder if in this case code follows some unusual/unexpected path.
BTW, could you please also provide CPU name/model/features as detected by the kernel?

Thanks!

--
Andriy Gapon
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On 19 August 2010 20:39, Andriy Gapon <> wrote:
> on 10/08/2010 19:55 pluknet said the following:
>> On 16 July 2010 19:47, Jung-uk Kim <> wrote:
>>> The patch should apply fine on both sys/amd64/amd64/mp_machdep.c and
>>> sys/i386/i386/mp_machdep.c.
>>>
>>> http://people.freebsd.org/~jkim/mp_machdep2.diff
>>>
>>
>>
>> Hi.
>>
>> Just checked on Xen HVM with 3 cores.
>> 1) 8.1 unmodified:
>> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
>> FreeBSD/SMP: 1 package(s) x 3 core(s)
>>
>> 2) 8.1 + patch
>> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
>> FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
>> WARNING: Non-uniform processors.
>> WARNING: Using suboptimal topology.
>
> Can you debug, e.g. with printfs, what exactly goes wrong?
> I wonder if in this case code follows some unusual/unexpected path.

Sorry, I'm a bit busy right now.
I hope to debug this somewhere in the next week.

> BTW, could you please also provide CPU name/model/features as detected by the kernel?

Sure.
CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2763.12-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping = 5
Features=0x1781fbbf
Features2=0x80982201>
TSC: P-state invariant
real memory = 4194304000 (4000 MB)
avail memory = 3932786688 (3750 MB)
ACPI APIC Table:
FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 2
cpu2 (AP/HT): APIC ID: 4

Just a thought.
# HTT might somehow correlate with current maxcpus limit (32).

>
> Thanks!


--
wbr,
pluknet
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.

On Thursday 19 August 2010 12:56 pm, pluknet wrote:
> On 19 August 2010 20:39, Andriy Gapon <> wrote:
> > on 10/08/2010 19:55 pluknet said the following:
> >> On 16 July 2010 19:47, Jung-uk Kim <> wrote:
> >>> The patch should apply fine on both
> >>> sys/amd64/amd64/mp_machdep.c and sys/i386/i386/mp_machdep.c.
> >>>
> >>> http://people.freebsd.org/~jkim/mp_machdep2.diff
> >>
> >> Hi.
> >>
> >> Just checked on Xen HVM with 3 cores.
> >> 1) 8.1 unmodified:
> >> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> >> FreeBSD/SMP: 1 package(s) x 3 core(s)
> >>
> >> 2) 8.1 + patch
> >> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> >> FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
> >> WARNING: Non-uniform processors.
> >> WARNING: Using suboptimal topology.
> >
> > Can you debug, e.g. with printfs, what exactly goes wrong?
> > I wonder if in this case code follows some unusual/unexpected
> > path.
>
> Sorry, I'm a bit busy right now.
> I hope to debug this somewhere in the next week.
>
> > BTW, could you please also provide CPU name/model/features as
> > detected by the kernel?
>
> Sure.
> CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2763.12-MHz
> 686-class CPU) Origin = "GenuineIntel" Id = 0x106a5 Family = 6
> Model = 1a Stepping = 5
> Features=0x1781fbbf >E,MCA,CMOV,PAT,MMX,FXSR,SSE,SSE2,HTT>
> Features2=0x80982201>
> TSC: P-state invariant
> real memory = 4194304000 (4000 MB)
> avail memory = 3932786688 (3750 MB)
> ACPI APIC Table:
> FreeBSD/SMP: Multiprocessor System Detected: 3 CPUs
> FreeBSD/SMP: 0 package(s) x 1 core(s) x 32 HTT threads
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP/HT): APIC ID: 2
> cpu2 (AP/HT): APIC ID: 4
>
> Just a thought.
> # HTT might somehow correlate with current maxcpus limit (32).

One thing I am not sure is whether those CPUID instructions are
executed on *real* CPUs or translated in HVM. On top of that, I am
not even sure they will be executed on *correct* cores. I bet they
won't. If that's the case, we should add exception for virtualized
environment as we did for default HZ.

Jung-uk Kim
_______________________________________________
___________________________________________________

Posted on the Freebsd-stable mailing list. Go to http://lists.freebsd.org/mailman/listinfo/freebsd-stable to subscribe.