Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)
Rusty,

In short, I'm starting to agree with others who suggest that this just
feels *way* too late. Perhaps if we had been using tdb2 format in
Samba4 for a while, it wouldn't be an issue, but there hasn't been major
use of the db layer (only the API) anywhere, and we are getting way to
close to the release.

On Thu, 2012-06-14 at 11:50 +0930, Rusty Russell wrote:

> I just did more stresstesting to prove ntdb was "safe", and found
> another bug. Gah! So I'm going to eat humble pie and suggest we
> proceed cautiously:
>
> (1) Finish the dbwrap conversions (still using dbwrap_tdb as the
> backend) for beta2. That's what I'm debugging now, though as you
> point out, s3/messaging and several s4 databases still need
> conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).
>
> (2) Make dbwrap select ntdb or tdb depending on the filename handed to
> it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
> name if necessary (good practice anyway, since that name is good for
> log messages).
>
> (3) Have the ntdb version automatically upgrade tdb versions:
> if (!file_exists(ntdbname) && file_exists(tdbname)) {
> ... traverse and convert ...
> ... rename tdbname to tdbname.bak ...
> fail to open ntdb if something goes wrong here.
> }
>
> (4) We can switch the callers based on an 'old tdb format' lpcfg. This
> can be done at any stage, even an individual db basis if we want.
>
> (5) ldb files will be handled as planned (that code is already written):
> new ldb files will be created as ntdb unless 'old tdb format' is
> set.
>
> I'm hoping to commit the dbwrap conversions today or tomorrow, depending
> when debugging finishes.
>
> > I might also help out in converting things to dbwrap if that
> > would be useful. What do others think? For the s4-part, I would
> > also like to hear the opinion of (among others) Andrew and Metze.
>
> That would be nice, thanks! Want to do s3/messaging and s4?

Would we get this in faster if we just dropped the s4 and ldb parts, at
least initially?

When I first argued against just dropping this whole thing, it was
Easter, and exceptional circumstances meant that it couldn't all be
decided then.

If we could live with TDB as is (I may have misunderstood the
conclusion of that discussion), and the 'walk over the whole tree' work
is the dbwrap conversion (rather than the tdb_compat API), would we do
better to punt the actual use of ntdb until after 4.0?

We do still have other major changes to land - the SMB3 stuff is still
pending, but we don't have long, and it's mostly moving around upper
layers (the work at the guts of it included gensec back in
Jan/Feb/March).

I guess I'm just feeling that we are doing in June what (particularly)
for our now-agreed timeline we really should also have been doing in
Jan/Feb/March.

I'm sorry to lower the boom over this major project of yours, but I hope
you understand.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)
Rusty,

In short, I'm starting to agree with others who suggest that this just
feels *way* too late. Perhaps if we had been using tdb2 format in
Samba4 for a while, it wouldn't be an issue, but there hasn't been major
use of the db layer (only the API) anywhere, and we are getting way to
close to the release.

On Thu, 2012-06-14 at 11:50 +0930, Rusty Russell wrote:

> I just did more stresstesting to prove ntdb was "safe", and found
> another bug. Gah! So I'm going to eat humble pie and suggest we
> proceed cautiously:
>
> (1) Finish the dbwrap conversions (still using dbwrap_tdb as the
> backend) for beta2. That's what I'm debugging now, though as you
> point out, s3/messaging and several s4 databases still need
> conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).
>
> (2) Make dbwrap select ntdb or tdb depending on the filename handed to
> it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
> name if necessary (good practice anyway, since that name is good for
> log messages).
>
> (3) Have the ntdb version automatically upgrade tdb versions:
> if (!file_exists(ntdbname) && file_exists(tdbname)) {
> ... traverse and convert ...
> ... rename tdbname to tdbname.bak ...
> fail to open ntdb if something goes wrong here.
> }
>
> (4) We can switch the callers based on an 'old tdb format' lpcfg. This
> can be done at any stage, even an individual db basis if we want.
>
> (5) ldb files will be handled as planned (that code is already written):
> new ldb files will be created as ntdb unless 'old tdb format' is
> set.
>
> I'm hoping to commit the dbwrap conversions today or tomorrow, depending
> when debugging finishes.
>
> > I might also help out in converting things to dbwrap if that
> > would be useful. What do others think? For the s4-part, I would
> > also like to hear the opinion of (among others) Andrew and Metze.
>
> That would be nice, thanks! Want to do s3/messaging and s4?

Would we get this in faster if we just dropped the s4 and ldb parts, at
least initially?

When I first argued against just dropping this whole thing, it was
Easter, and exceptional circumstances meant that it couldn't all be
decided then.

If we could live with TDB as is (I may have misunderstood the
conclusion of that discussion), and the 'walk over the whole tree' work
is the dbwrap conversion (rather than the tdb_compat API), would we do
better to punt the actual use of ntdb until after 4.0?

We do still have other major changes to land - the SMB3 stuff is still
pending, but we don't have long, and it's mostly moving around upper
layers (the work at the guts of it included gensec back in
Jan/Feb/March).

I guess I'm just feeling that we are doing in June what (particularly)
for our now-agreed timeline we really should also have been doing in
Jan/Feb/March.

I'm sorry to lower the boom over this major project of yours, but I hope
you understand.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


)
On Fri, 2012-06-15 at 10:33 +0930, Rusty Russell wrote:
> On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> > One comment on the code: I think we don't need the allocator
> > attribute, given that we have ntdb_parse_record.
>
> I think it's still nice to have, because the allocator attribute also
> allows the struct ntdb_context to be a talloc object. Samba doesn't
> care very much since we wrap everything, but it's neat for other projects.
>
> ntdb_parse_record() is more efficient than tdb_parse_record(), but it
> can still create a copy. This means you're doing a double-copy to get a
> talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
> simple code.

Agreed, I would love to have it for the few places where we use libtdb
directly in sssd, which is heavily based on talloc()

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <>
Principal Software Engineer at Red Hat, Inc. <>

)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)
Rusty,

In short, I'm starting to agree with others who suggest that this just
feels *way* too late. Perhaps if we had been using tdb2 format in
Samba4 for a while, it wouldn't be an issue, but there hasn't been major
use of the db layer (only the API) anywhere, and we are getting way to
close to the release.

On Thu, 2012-06-14 at 11:50 +0930, Rusty Russell wrote:

> I just did more stresstesting to prove ntdb was "safe", and found
> another bug. Gah! So I'm going to eat humble pie and suggest we
> proceed cautiously:
>
> (1) Finish the dbwrap conversions (still using dbwrap_tdb as the
> backend) for beta2. That's what I'm debugging now, though as you
> point out, s3/messaging and several s4 databases still need
> conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).
>
> (2) Make dbwrap select ntdb or tdb depending on the filename handed to
> it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
> name if necessary (good practice anyway, since that name is good for
> log messages).
>
> (3) Have the ntdb version automatically upgrade tdb versions:
> if (!file_exists(ntdbname) && file_exists(tdbname)) {
> ... traverse and convert ...
> ... rename tdbname to tdbname.bak ...
> fail to open ntdb if something goes wrong here.
> }
>
> (4) We can switch the callers based on an 'old tdb format' lpcfg. This
> can be done at any stage, even an individual db basis if we want.
>
> (5) ldb files will be handled as planned (that code is already written):
> new ldb files will be created as ntdb unless 'old tdb format' is
> set.
>
> I'm hoping to commit the dbwrap conversions today or tomorrow, depending
> when debugging finishes.
>
> > I might also help out in converting things to dbwrap if that
> > would be useful. What do others think? For the s4-part, I would
> > also like to hear the opinion of (among others) Andrew and Metze.
>
> That would be nice, thanks! Want to do s3/messaging and s4?

Would we get this in faster if we just dropped the s4 and ldb parts, at
least initially?

When I first argued against just dropping this whole thing, it was
Easter, and exceptional circumstances meant that it couldn't all be
decided then.

If we could live with TDB as is (I may have misunderstood the
conclusion of that discussion), and the 'walk over the whole tree' work
is the dbwrap conversion (rather than the tdb_compat API), would we do
better to punt the actual use of ntdb until after 4.0?

We do still have other major changes to land - the SMB3 stuff is still
pending, but we don't have long, and it's mostly moving around upper
layers (the work at the guts of it included gensec back in
Jan/Feb/March).

I guess I'm just feeling that we are doing in June what (particularly)
for our now-agreed timeline we really should also have been doing in
Jan/Feb/March.

I'm sorry to lower the boom over this major project of yours, but I hope
you understand.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


)
On Fri, 2012-06-15 at 10:33 +0930, Rusty Russell wrote:
> On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> > One comment on the code: I think we don't need the allocator
> > attribute, given that we have ntdb_parse_record.
>
> I think it's still nice to have, because the allocator attribute also
> allows the struct ntdb_context to be a talloc object. Samba doesn't
> care very much since we wrap everything, but it's neat for other projects.
>
> ntdb_parse_record() is more efficient than tdb_parse_record(), but it
> can still create a copy. This means you're doing a double-copy to get a
> talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
> simple code.

Agreed, I would love to have it for the few places where we use libtdb
directly in sssd, which is heavily based on talloc()

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <>
Principal Software Engineer at Red Hat, Inc. <>

)
On Fri, 15 Jun 2012 11:05:42 +1000, Andrew Bartlett <> wrote:
> Rusty,
>
> In short, I'm starting to agree with others who suggest that this just
> feels *way* too late. Perhaps if we had been using tdb2 format in
> Samba4 for a while, it wouldn't be an issue, but there hasn't been major
> use of the db layer (only the API) anywhere, and we are getting way to
> close to the release.

Agreed.

I'll commit the current ntdb work, but it's not used by anything. At
least it'll be built, and tested with 'make test', and be there for
inspection. Library version is 0.9, giving us some wiggle room if we
need to break something.

I'll also push the dbwrap enhancements, too (dbwrap_local_open and the
API extensions which allow wider use).

I'll continue to work on the dbwrap conversion with anyone who's
interested. All welcome to join the fun!

Here's what I've got (I'm currently debugging 'source3/printing: convert
to dbwrap.' which breaks on a full 'make test' for some weird reason):

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/dbwrap

> I'm sorry to lower the boom over this major project of yours, but I hope
> you understand.

Thanks for the thought, but don't apologize! It's the right decision,
for the right reasons. If I wish the universe were different, that's my
problem :)

Cheers,
Rusty.
)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)
Rusty,

In short, I'm starting to agree with others who suggest that this just
feels *way* too late. Perhaps if we had been using tdb2 format in
Samba4 for a while, it wouldn't be an issue, but there hasn't been major
use of the db layer (only the API) anywhere, and we are getting way to
close to the release.

On Thu, 2012-06-14 at 11:50 +0930, Rusty Russell wrote:

> I just did more stresstesting to prove ntdb was "safe", and found
> another bug. Gah! So I'm going to eat humble pie and suggest we
> proceed cautiously:
>
> (1) Finish the dbwrap conversions (still using dbwrap_tdb as the
> backend) for beta2. That's what I'm debugging now, though as you
> point out, s3/messaging and several s4 databases still need
> conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).
>
> (2) Make dbwrap select ntdb or tdb depending on the filename handed to
> it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
> name if necessary (good practice anyway, since that name is good for
> log messages).
>
> (3) Have the ntdb version automatically upgrade tdb versions:
> if (!file_exists(ntdbname) && file_exists(tdbname)) {
> ... traverse and convert ...
> ... rename tdbname to tdbname.bak ...
> fail to open ntdb if something goes wrong here.
> }
>
> (4) We can switch the callers based on an 'old tdb format' lpcfg. This
> can be done at any stage, even an individual db basis if we want.
>
> (5) ldb files will be handled as planned (that code is already written):
> new ldb files will be created as ntdb unless 'old tdb format' is
> set.
>
> I'm hoping to commit the dbwrap conversions today or tomorrow, depending
> when debugging finishes.
>
> > I might also help out in converting things to dbwrap if that
> > would be useful. What do others think? For the s4-part, I would
> > also like to hear the opinion of (among others) Andrew and Metze.
>
> That would be nice, thanks! Want to do s3/messaging and s4?

Would we get this in faster if we just dropped the s4 and ldb parts, at
least initially?

When I first argued against just dropping this whole thing, it was
Easter, and exceptional circumstances meant that it couldn't all be
decided then.

If we could live with TDB as is (I may have misunderstood the
conclusion of that discussion), and the 'walk over the whole tree' work
is the dbwrap conversion (rather than the tdb_compat API), would we do
better to punt the actual use of ntdb until after 4.0?

We do still have other major changes to land - the SMB3 stuff is still
pending, but we don't have long, and it's mostly moving around upper
layers (the work at the guts of it included gensec back in
Jan/Feb/March).

I guess I'm just feeling that we are doing in June what (particularly)
for our now-agreed timeline we really should also have been doing in
Jan/Feb/March.

I'm sorry to lower the boom over this major project of yours, but I hope
you understand.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


)
On Fri, 2012-06-15 at 10:33 +0930, Rusty Russell wrote:
> On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> > One comment on the code: I think we don't need the allocator
> > attribute, given that we have ntdb_parse_record.
>
> I think it's still nice to have, because the allocator attribute also
> allows the struct ntdb_context to be a talloc object. Samba doesn't
> care very much since we wrap everything, but it's neat for other projects.
>
> ntdb_parse_record() is more efficient than tdb_parse_record(), but it
> can still create a copy. This means you're doing a double-copy to get a
> talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
> simple code.

Agreed, I would love to have it for the few places where we use libtdb
directly in sssd, which is heavily based on talloc()

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <>
Principal Software Engineer at Red Hat, Inc. <>

)
On Fri, 15 Jun 2012 11:05:42 +1000, Andrew Bartlett <> wrote:
> Rusty,
>
> In short, I'm starting to agree with others who suggest that this just
> feels *way* too late. Perhaps if we had been using tdb2 format in
> Samba4 for a while, it wouldn't be an issue, but there hasn't been major
> use of the db layer (only the API) anywhere, and we are getting way to
> close to the release.

Agreed.

I'll commit the current ntdb work, but it's not used by anything. At
least it'll be built, and tested with 'make test', and be there for
inspection. Library version is 0.9, giving us some wiggle room if we
need to break something.

I'll also push the dbwrap enhancements, too (dbwrap_local_open and the
API extensions which allow wider use).

I'll continue to work on the dbwrap conversion with anyone who's
interested. All welcome to join the fun!

Here's what I've got (I'm currently debugging 'source3/printing: convert
to dbwrap.' which breaks on a full 'make test' for some weird reason):

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/dbwrap

> I'm sorry to lower the boom over this major project of yours, but I hope
> you understand.

Thanks for the thought, but don't apologize! It's the right decision,
for the right reasons. If I wish the universe were different, that's my
problem :)

Cheers,
Rusty.
)
On Thu, Jun 14, 2012 at 11:50:38AM +0930, Rusty Russell wrote:
> On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> > Rusty, I looked at the code to some extent, and I also have a couple of
> > questions and comments:
> >
> > * To re-raise Volker's concern:
> >
> > You have been working on TDB2 for a long time. At least
> > a year or so. Now you seem to have overhauled the database
> > format and the code to some (if I am not mistaken) non-trivial
> > extent in a very short timeframe. That makes me worry too, that
> > this might not be very stable yet. And are the benchmarks now more
> > meaningful than the last ones.
> >
> > We are changing a core infrastructure component here.
> > The change will affect virturall all parts of samba.
>
> But yes, I've been working on TDB2 on and off for over a year. Which is
> really crazy, since it's about four months of actual work. The rest has
> been timesliced with other things, and the work to integrate it into
> Samba itself.

One comment on the code: I think we don't need the allocator
attribute, given that we have ntdb_parse_record.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:
)

Hi all,

Sorry this has been delayed: two things happened. Firstly,
other duties involved me going to Hong Kong for a week. Secondly,
porting revealed an unacceptable slowdown for smaller databases going
from tdb to tdb2, so after much benchmarking, the format was simplfied
to be closer to the original tdb. See benchmarks below taken from that
commit message; we still pay a slight penalty for 64 bit.

See my ntdb-wip head:

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

So far:

All of source4/ is converted to ntdb, as is ldb (it handles the
switch internally). I've written a dbwrap_open_local() which switches
between the ntdb and tdb backends based on 'use old tdb = yes'
configution option for dbwrap users. If this isn't set, I plan to use
the tdb backend if a tdb file is there, otherwise use ntdb, but I
haven't implemented that.

The general rule of conversions has to be to rename databases to
".ntdb", so it's absolutely clear. The dbwrap_open_ntdb() will change
.tdb names to .ntdb names for the moment, though dbwrap_open_tdb() will
do the reverse mapping, so you can use either method (not yet
implemented).

Everything not using dbwrap is being converted; CLEAR_IF_FIRST
or INTERNAL databases are fairly non-controversial. If something else
should not be converted, feel free to change it to use dbwrap.

Note that NTDB_DATA/struct ntdb_data is a synonym for TDB_DATA/struct
TDB_DATA if tdb.h is included before ntdb.h: without this, compatibility
becomes a nightmare, as these are used all over Samba.

To come:

There's a bit more source3 to convert, then lots of testing and
making sure the s3->s4 upgrade scripts work well. I'll be working on
this all this week.

BTW, here are the benchmarks which made me rework the NTDB hash code:

Insert Re-ins Fetch Size dbspeed
(nsec) (nsec) (nsec) (Kb) (ops/sec)
TDB (10000 hashsize):
100 records: 3882 3320 1609 53 203204
1000 records: 3651 3281 1571 115 218021
10000 records: 3404 3326 1595 880 202874
100000 records: 4317 3825 2097 8262 126811
1000000 records: 11568 11578 9320 77005 25046

TDB2 (1024 hashsize, expandable):
100 records: 3867 3329 1699 17 187100
1000 records: 4040 3249 1639 154 186255
10000 records: 4143 3300 1695 1226 185110
100000 records: 4481 3425 1800 17848 163483
1000000 records: 4055 3534 1878 106386 160774

NTDB (8192 hashsize)
100 records: 4259 3376 1692 82 190852
1000 records: 3640 3275 1566 130 195106
10000 records: 4337 3438 1614 773 188362
100000 records: 4750 5165 1746 9001 169197
1000000 records: 4897 5180 2341 83838 121901

Analysis:
1) TDB wins on first insert on small databases, beating TDB2 by
~15%, NTDB by ~10% on dbspeed.
2) TDB starts to lose when hash chains get 10 long (fetch 10% slower
than TDB2/NTDB).
3) TDB does horribly when hash chains get 100 long (fetch 4x slower
than NTDB, 5x slower than TDB2, insert about 2-3x slower).
4) TDB2 databases are 40% larger than TDB1. NTDB is about 15% larger
than TDB1.

Cheers,
Rusty.
)
On Mon, 2012-06-04 at 21:11 +0930, Rusty Russell wrote:
> Hi all,
>
> Sorry this has been delayed: two things happened. Firstly,
> other duties involved me going to Hong Kong for a week. Secondly,
> porting revealed an unacceptable slowdown for smaller databases going
> from tdb to tdb2, so after much benchmarking, the format was simplfied
> to be closer to the original tdb. See benchmarks below taken from that
> commit message; we still pay a slight penalty for 64 bit.
>
> See my ntdb-wip head:
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/ntdb-wip

Rusty,

With beta2 scheduled for Tuesday, I'm wondering where we are at with
landing this code?

The only comment I've seen on the code itself is Volker's comment
concerned about the change you mention here. It is a reasonable
concern, but I'm not sure what we can do about it, given we already
decided that the options for not using TDB2 also were not acceptable.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org

)
On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> Rusty, I looked at the code to some extent, and I also have a couple of
> questions and comments:
>
> * To re-raise Volker's concern:
>
> You have been working on TDB2 for a long time. At least
> a year or so. Now you seem to have overhauled the database
> format and the code to some (if I am not mistaken) non-trivial
> extent in a very short timeframe. That makes me worry too, that
> this might not be very stable yet. And are the benchmarks now more
> meaningful than the last ones.
>
> We are changing a core infrastructure component here.
> The change will affect virturall all parts of samba.

But yes, I've been working on TDB2 on and off for over a year. Which is
really crazy, since it's about four months of actual work. The rest has
been timesliced with other things, and the work to integrate it into
Samba itself.

I just did more stresstesting to prove ntdb was "safe", and found
another bug. Gah! So I'm going to eat humble pie and suggest we
proceed cautiously:

(1) Finish the dbwrap conversions (still using dbwrap_tdb as the
backend) for beta2. That's what I'm debugging now, though as you
point out, s3/messaging and several s4 databases still need
conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).

(2) Make dbwrap select ntdb or tdb depending on the filename handed to
it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
name if necessary (good practice anyway, since that name is good for
log messages).

(3) Have the ntdb version automatically upgrade tdb versions:
if (!file_exists(ntdbname) && file_exists(tdbname)) {
... traverse and convert ...
... rename tdbname to tdbname.bak ...
fail to open ntdb if something goes wrong here.
}

(4) We can switch the callers based on an 'old tdb format' lpcfg. This
can be done at any stage, even an individual db basis if we want.

(5) ldb files will be handled as planned (that code is already written):
new ldb files will be created as ntdb unless 'old tdb format' is
set.

I'm hoping to commit the dbwrap conversions today or tomorrow, depending
when debugging finishes.

> I might also help out in converting things to dbwrap if that
> would be useful. What do others think? For the s4-part, I would
> also like to hear the opinion of (among others) Andrew and Metze.

That would be nice, thanks! Want to do s3/messaging and s4?

Thoughts?
Rusty.
)
On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> One comment on the code: I think we don't need the allocator
> attribute, given that we have ntdb_parse_record.

I think it's still nice to have, because the allocator attribute also
allows the struct ntdb_context to be a talloc object. Samba doesn't
care very much since we wrap everything, but it's neat for other projects.

ntdb_parse_record() is more efficient than tdb_parse_record(), but it
can still create a copy. This means you're doing a double-copy to get a
talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
simple code.

Cheers,
Rusty.
)
Rusty,

In short, I'm starting to agree with others who suggest that this just
feels *way* too late. Perhaps if we had been using tdb2 format in
Samba4 for a while, it wouldn't be an issue, but there hasn't been major
use of the db layer (only the API) anywhere, and we are getting way to
close to the release.

On Thu, 2012-06-14 at 11:50 +0930, Rusty Russell wrote:

> I just did more stresstesting to prove ntdb was "safe", and found
> another bug. Gah! So I'm going to eat humble pie and suggest we
> proceed cautiously:
>
> (1) Finish the dbwrap conversions (still using dbwrap_tdb as the
> backend) for beta2. That's what I'm debugging now, though as you
> point out, s3/messaging and several s4 databases still need
> conversion (I generally skipped CLEAR_IF_FIRST or INTERNAL dbs).
>
> (2) Make dbwrap select ntdb or tdb depending on the filename handed to
> it (ie. foo.ntdb or foo.tdb). Make any INTERNAL dbs use a dummy
> name if necessary (good practice anyway, since that name is good for
> log messages).
>
> (3) Have the ntdb version automatically upgrade tdb versions:
> if (!file_exists(ntdbname) && file_exists(tdbname)) {
> ... traverse and convert ...
> ... rename tdbname to tdbname.bak ...
> fail to open ntdb if something goes wrong here.
> }
>
> (4) We can switch the callers based on an 'old tdb format' lpcfg. This
> can be done at any stage, even an individual db basis if we want.
>
> (5) ldb files will be handled as planned (that code is already written):
> new ldb files will be created as ntdb unless 'old tdb format' is
> set.
>
> I'm hoping to commit the dbwrap conversions today or tomorrow, depending
> when debugging finishes.
>
> > I might also help out in converting things to dbwrap if that
> > would be useful. What do others think? For the s4-part, I would
> > also like to hear the opinion of (among others) Andrew and Metze.
>
> That would be nice, thanks! Want to do s3/messaging and s4?

Would we get this in faster if we just dropped the s4 and ldb parts, at
least initially?

When I first argued against just dropping this whole thing, it was
Easter, and exceptional circumstances meant that it couldn't all be
decided then.

If we could live with TDB as is (I may have misunderstood the
conclusion of that discussion), and the 'walk over the whole tree' work
is the dbwrap conversion (rather than the tdb_compat API), would we do
better to punt the actual use of ntdb until after 4.0?

We do still have other major changes to land - the SMB3 stuff is still
pending, but we don't have long, and it's mostly moving around upper
layers (the work at the guts of it included gensec back in
Jan/Feb/March).

I guess I'm just feeling that we are doing in June what (particularly)
for our now-agreed timeline we really should also have been doing in
Jan/Feb/March.

I'm sorry to lower the boom over this major project of yours, but I hope
you understand.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org


)
On Fri, 2012-06-15 at 10:33 +0930, Rusty Russell wrote:
> On Thu, 14 Jun 2012 09:34:17 +0200, Volker Lendecke <> wrote:
> > One comment on the code: I think we don't need the allocator
> > attribute, given that we have ntdb_parse_record.
>
> I think it's still nice to have, because the allocator attribute also
> allows the struct ntdb_context to be a talloc object. Samba doesn't
> care very much since we wrap everything, but it's neat for other projects.
>
> ntdb_parse_record() is more efficient than tdb_parse_record(), but it
> can still create a copy. This means you're doing a double-copy to get a
> talloc object. Again, Samba doesn't care, but ntdb_fetch() is good for
> simple code.

Agreed, I would love to have it for the few places where we use libtdb
directly in sssd, which is heavily based on talloc()

Simo.

--
Simo Sorce
Samba Team GPL Compliance Officer <>
Principal Software Engineer at Red Hat, Inc. <>

)
On Fri, 15 Jun 2012 11:05:42 +1000, Andrew Bartlett <> wrote:
> Rusty,
>
> In short, I'm starting to agree with others who suggest that this just
> feels *way* too late. Perhaps if we had been using tdb2 format in
> Samba4 for a while, it wouldn't be an issue, but there hasn't been major
> use of the db layer (only the API) anywhere, and we are getting way to
> close to the release.

Agreed.

I'll commit the current ntdb work, but it's not used by anything. At
least it'll be built, and tested with 'make test', and be there for
inspection. Library version is 0.9, giving us some wiggle room if we
need to break something.

I'll also push the dbwrap enhancements, too (dbwrap_local_open and the
API extensions which allow wider use).

I'll continue to work on the dbwrap conversion with anyone who's
interested. All welcome to join the fun!

Here's what I've got (I'm currently debugging 'source3/printing: convert
to dbwrap.' which breaks on a full 'make test' for some weird reason):

https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/dbwrap

> I'm sorry to lower the boom over this major project of yours, but I hope
> you understand.

Thanks for the thought, but don't apologize! It's the right decision,
for the right reasons. If I wish the universe were different, that's my
problem :)

Cheers,
Rusty.
)
On Thu, Jun 14, 2012 at 11:50:38AM +0930, Rusty Russell wrote:
> On Wed, 13 Jun 2012 12:58:50 +0200, Michael Adam <> wrote:
> > Rusty, I looked at the code to some extent, and I also have a couple of
> > questions and comments:
> >
> > * To re-raise Volker's concern:
> >
> > You have been working on TDB2 for a long time. At least
> > a year or so. Now you seem to have overhauled the database
> > format and the code to some (if I am not mistaken) non-trivial
> > extent in a very short timeframe. That makes me worry too, that
> > this might not be very stable yet. And are the benchmarks now more
> > meaningful than the last ones.
> >
> > We are changing a core infrastructure component here.
> > The change will affect virturall all parts of samba.
>
> But yes, I've been working on TDB2 on and off for over a year. Which is
> really crazy, since it's about four months of actual work. The rest has
> been timesliced with other things, and the work to integrate it into
> Samba itself.

One comment on the code: I think we don't need the allocator
attribute, given that we have ntdb_parse_record.

Volker

--
SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:
)
On Fri 15.Jun.12 13:53, Rusty Russell wrote:
>Here's what I've got (I'm currently debugging 'source3/printing: convert
>to dbwrap.' which breaks on a full 'make test' for some weird reason):
>
> https://git.samba.org/rusty/samba.git/?p=rusty/samba.git;a=shortlog;h=refs/heads/dbwrap
>

Hitting printing bugs could mean you hit a real bug. The code is still
not in a very good state, especially job spooling :)


-- andreas

)