Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.

t.

On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:

> Hang in there. Don't lose track of the goal. We all have worked hard
> on various parts of the software. If we didn't care we wouldn't be so
> passionate about our work, and reviewing that work. Working remotely
> and virtually on a project is tough. So it's important to step back
> from the tone of the conversation and work on the issues. There are
> some very practical steps that can be taken to work this out.
>
> Having the current String library unreliable and breakable is not a good thing.
>
> A few more examples, and a bit discussion could lead to an upstream
> patch request could be made for changes that need to be made. In the
> meantime this could replace the current string library or the current
> string library could be pulled out of core until something acceptable
> is found.
>
> --Rick
>
> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>> Alan, I'm frustrated that you keep suggesting I've made mistakes. You've
>> raised numerous concerns about my code. I'm not necessarily saying your
>> concerns without merit, but I just don't see how they're real issues, and
>> you refuse to provide any actual examples!
>>
>> Why is posting a specific example so hard?
>>
>> I provided over 120 examples in specific tests, based on Rick and Mark's
>> test suite and also tests based on adaptations of Tom's code, plus several
>> more I wrote.
>>
>> Regarding that constructor, the = operator calls copy(), which uses
>> strlen_P() and strcpy_P().
>>
>> Regarding the tone of these messages, it seems to me you're repeatedly
>> claiming I've done things incorrectly. You've called my approach
>> "ineffective". You repeatedly suggest problems can occur, but never give an
>> example. But yes, I could respond more softly. I'll try to do so in the
>> future.
>>
>>
>> -Paul
>>
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.

t.

On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:

> Hang in there. Don't lose track of the goal. We all have worked hard
> on various parts of the software. If we didn't care we wouldn't be so
> passionate about our work, and reviewing that work. Working remotely
> and virtually on a project is tough. So it's important to step back
> from the tone of the conversation and work on the issues. There are
> some very practical steps that can be taken to work this out.
>
> Having the current String library unreliable and breakable is not a good thing.
>
> A few more examples, and a bit discussion could lead to an upstream
> patch request could be made for changes that need to be made. In the
> meantime this could replace the current string library or the current
> string library could be pulled out of core until something acceptable
> is found.
>
> --Rick
>
> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>> Alan, I'm frustrated that you keep suggesting I've made mistakes. You've
>> raised numerous concerns about my code. I'm not necessarily saying your
>> concerns without merit, but I just don't see how they're real issues, and
>> you refuse to provide any actual examples!
>>
>> Why is posting a specific example so hard?
>>
>> I provided over 120 examples in specific tests, based on Rick and Mark's
>> test suite and also tests based on adaptations of Tom's code, plus several
>> more I wrote.
>>
>> Regarding that constructor, the = operator calls copy(), which uses
>> strlen_P() and strcpy_P().
>>
>> Regarding the tone of these messages, it seems to me you're repeatedly
>> claiming I've done things incorrectly. You've called my approach
>> "ineffective". You repeatedly suggest problems can occur, but never give an
>> example. But yes, I could respond more softly. I'll try to do so in the
>> future.
>>
>>
>> -Paul
>>
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. How about we create a branch just for this change, and then allow
people to test it out more widely. We could let it cook for a bit, and
then determine to merge in.

--Rick

On Thu, Jan 20, 2011 at 3:25 PM, Tom Igoe <> wrote:
> I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.
>
> t.
>
> On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:
>
>> Hang in there. Don't lose track of the goal. We all have worked hard
>> on various parts of the software.  If we didn't care we wouldn't be so
>> passionate about our work, and reviewing that work.  Working remotely
>> and virtually on a project is tough. So it's important to step back
>> from the tone of the conversation and work on the issues. There are
>> some very practical steps that can be taken to work this out.
>>
>> Having the current String library unreliable and breakable is not a good thing.
>>
>> A few more examples, and a bit discussion could lead to an upstream
>> patch request could be made for changes that need to be made. In the
>> meantime this could replace the current string library or the current
>> string library could be pulled out of core until something acceptable
>> is found.
>>
>> --Rick
>>
>> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>>> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
>>> raised numerous concerns about my code.  I'm not necessarily saying your
>>> concerns without merit, but I just don't see how they're real issues, and
>>> you refuse to provide any actual examples!
>>>
>>> Why is posting a specific example so hard?
>>>
>>> I provided over 120 examples in specific tests, based on Rick and Mark's
>>> test suite and also tests based on adaptations of Tom's code, plus several
>>> more I wrote.
>>>
>>> Regarding that constructor, the = operator calls copy(), which uses
>>> strlen_P() and strcpy_P().
>>>
>>> Regarding the tone of these messages, it seems to me you're repeatedly
>>> claiming I've done things incorrectly.  You've called my approach
>>> "ineffective".  You repeatedly suggest problems can occur, but never give an
>>> example.  But yes, I could respond more softly.  I'll try to do so in the
>>> future.
>>>
>>>
>>> -Paul
>>>
>>>
>>>
>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.

t.

On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:

> Hang in there. Don't lose track of the goal. We all have worked hard
> on various parts of the software. If we didn't care we wouldn't be so
> passionate about our work, and reviewing that work. Working remotely
> and virtually on a project is tough. So it's important to step back
> from the tone of the conversation and work on the issues. There are
> some very practical steps that can be taken to work this out.
>
> Having the current String library unreliable and breakable is not a good thing.
>
> A few more examples, and a bit discussion could lead to an upstream
> patch request could be made for changes that need to be made. In the
> meantime this could replace the current string library or the current
> string library could be pulled out of core until something acceptable
> is found.
>
> --Rick
>
> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>> Alan, I'm frustrated that you keep suggesting I've made mistakes. You've
>> raised numerous concerns about my code. I'm not necessarily saying your
>> concerns without merit, but I just don't see how they're real issues, and
>> you refuse to provide any actual examples!
>>
>> Why is posting a specific example so hard?
>>
>> I provided over 120 examples in specific tests, based on Rick and Mark's
>> test suite and also tests based on adaptations of Tom's code, plus several
>> more I wrote.
>>
>> Regarding that constructor, the = operator calls copy(), which uses
>> strlen_P() and strcpy_P().
>>
>> Regarding the tone of these messages, it seems to me you're repeatedly
>> claiming I've done things incorrectly. You've called my approach
>> "ineffective". You repeatedly suggest problems can occur, but never give an
>> example. But yes, I could respond more softly. I'll try to do so in the
>> future.
>>
>>
>> -Paul
>>
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. How about we create a branch just for this change, and then allow
people to test it out more widely. We could let it cook for a bit, and
then determine to merge in.

--Rick

On Thu, Jan 20, 2011 at 3:25 PM, Tom Igoe <> wrote:
> I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.
>
> t.
>
> On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:
>
>> Hang in there. Don't lose track of the goal. We all have worked hard
>> on various parts of the software.  If we didn't care we wouldn't be so
>> passionate about our work, and reviewing that work.  Working remotely
>> and virtually on a project is tough. So it's important to step back
>> from the tone of the conversation and work on the issues. There are
>> some very practical steps that can be taken to work this out.
>>
>> Having the current String library unreliable and breakable is not a good thing.
>>
>> A few more examples, and a bit discussion could lead to an upstream
>> patch request could be made for changes that need to be made. In the
>> meantime this could replace the current string library or the current
>> string library could be pulled out of core until something acceptable
>> is found.
>>
>> --Rick
>>
>> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>>> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
>>> raised numerous concerns about my code.  I'm not necessarily saying your
>>> concerns without merit, but I just don't see how they're real issues, and
>>> you refuse to provide any actual examples!
>>>
>>> Why is posting a specific example so hard?
>>>
>>> I provided over 120 examples in specific tests, based on Rick and Mark's
>>> test suite and also tests based on adaptations of Tom's code, plus several
>>> more I wrote.
>>>
>>> Regarding that constructor, the = operator calls copy(), which uses
>>> strlen_P() and strcpy_P().
>>>
>>> Regarding the tone of these messages, it seems to me you're repeatedly
>>> claiming I've done things incorrectly.  You've called my approach
>>> "ineffective".  You repeatedly suggest problems can occur, but never give an
>>> example.  But yes, I could respond more softly.  I'll try to do so in the
>>> future.
>>>
>>>
>>> -Paul
>>>
>>>
>>>
>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Rick Anderson wrote:
> How about we create a branch just for this change, and then allow
> people to test it out more widely. We could let it cook for a bit, and
> then determine to merge in.
>

Anything that gets more people to actually give the code and try and
evaluate how well it does (or doesn't) work seems like a good idea.
Would a branch achieve that?

I'm not very experienced with git, so I'd prefer someone else do it.
Please feel free to grab the files and put them into a branch however
you feel is best. They're still here:

http://www.pjrc.com/teensy/string_class_experimental.html

I'd really like to start working on making String useful, which mostly
involves integration with the stream class (issue #454). But it seems
premature to pour lots of effort into integration while the status of
String itself is so uncertain.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.

t.

On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:

> Hang in there. Don't lose track of the goal. We all have worked hard
> on various parts of the software. If we didn't care we wouldn't be so
> passionate about our work, and reviewing that work. Working remotely
> and virtually on a project is tough. So it's important to step back
> from the tone of the conversation and work on the issues. There are
> some very practical steps that can be taken to work this out.
>
> Having the current String library unreliable and breakable is not a good thing.
>
> A few more examples, and a bit discussion could lead to an upstream
> patch request could be made for changes that need to be made. In the
> meantime this could replace the current string library or the current
> string library could be pulled out of core until something acceptable
> is found.
>
> --Rick
>
> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>> Alan, I'm frustrated that you keep suggesting I've made mistakes. You've
>> raised numerous concerns about my code. I'm not necessarily saying your
>> concerns without merit, but I just don't see how they're real issues, and
>> you refuse to provide any actual examples!
>>
>> Why is posting a specific example so hard?
>>
>> I provided over 120 examples in specific tests, based on Rick and Mark's
>> test suite and also tests based on adaptations of Tom's code, plus several
>> more I wrote.
>>
>> Regarding that constructor, the = operator calls copy(), which uses
>> strlen_P() and strcpy_P().
>>
>> Regarding the tone of these messages, it seems to me you're repeatedly
>> claiming I've done things incorrectly. You've called my approach
>> "ineffective". You repeatedly suggest problems can occur, but never give an
>> example. But yes, I could respond more softly. I'll try to do so in the
>> future.
>>
>>
>> -Paul
>>
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. How about we create a branch just for this change, and then allow
people to test it out more widely. We could let it cook for a bit, and
then determine to merge in.

--Rick

On Thu, Jan 20, 2011 at 3:25 PM, Tom Igoe <> wrote:
> I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.
>
> t.
>
> On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:
>
>> Hang in there. Don't lose track of the goal. We all have worked hard
>> on various parts of the software.  If we didn't care we wouldn't be so
>> passionate about our work, and reviewing that work.  Working remotely
>> and virtually on a project is tough. So it's important to step back
>> from the tone of the conversation and work on the issues. There are
>> some very practical steps that can be taken to work this out.
>>
>> Having the current String library unreliable and breakable is not a good thing.
>>
>> A few more examples, and a bit discussion could lead to an upstream
>> patch request could be made for changes that need to be made. In the
>> meantime this could replace the current string library or the current
>> string library could be pulled out of core until something acceptable
>> is found.
>>
>> --Rick
>>
>> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>>> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
>>> raised numerous concerns about my code.  I'm not necessarily saying your
>>> concerns without merit, but I just don't see how they're real issues, and
>>> you refuse to provide any actual examples!
>>>
>>> Why is posting a specific example so hard?
>>>
>>> I provided over 120 examples in specific tests, based on Rick and Mark's
>>> test suite and also tests based on adaptations of Tom's code, plus several
>>> more I wrote.
>>>
>>> Regarding that constructor, the = operator calls copy(), which uses
>>> strlen_P() and strcpy_P().
>>>
>>> Regarding the tone of these messages, it seems to me you're repeatedly
>>> claiming I've done things incorrectly.  You've called my approach
>>> "ineffective".  You repeatedly suggest problems can occur, but never give an
>>> example.  But yes, I could respond more softly.  I'll try to do so in the
>>> future.
>>>
>>>
>>> -Paul
>>>
>>>
>>>
>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Rick Anderson wrote:
> How about we create a branch just for this change, and then allow
> people to test it out more widely. We could let it cook for a bit, and
> then determine to merge in.
>

Anything that gets more people to actually give the code and try and
evaluate how well it does (or doesn't) work seems like a good idea.
Would a branch achieve that?

I'm not very experienced with git, so I'd prefer someone else do it.
Please feel free to grab the files and put them into a branch however
you feel is best. They're still here:

http://www.pjrc.com/teensy/string_class_experimental.html

I'd really like to start working on making String useful, which mostly
involves integration with the stream class (issue #454). But it seems
premature to pour lots of effort into integration while the status of
String itself is so uncertain.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Best thing to do is to follow Rick's guide for git development, clone it yourself, then submit a pull request. That way people can get at it and test it. When Dave's back from his conference, if it's all good, he'll merge it.

Rick's guide:
http://code.google.com/p/arduino/wiki/GitandGithubGuideforArduinoDevelopment


On Jan 21, 2011, at 2:09 PM, Paul Stoffregen wrote:

> Rick Anderson wrote:
>> How about we create a branch just for this change, and then allow
>> people to test it out more widely. We could let it cook for a bit, and
>> then determine to merge in.
>>
>
> Anything that gets more people to actually give the code and try and evaluate how well it does (or doesn't) work seems like a good idea. Would a branch achieve that?
>
> I'm not very experienced with git, so I'd prefer someone else do it. Please feel free to grab the files and put them into a branch however you feel is best. They're still here:
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> I'd really like to start working on making String useful, which mostly involves integration with the stream class (issue #454). But it seems premature to pour lots of effort into integration while the status of String itself is so uncertain.
>
>
> -Paul
>
>
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

Hi Everyone,

Here is my effort to rewrite the String class. These files are meant to
replace the existing String implementation with only minimal changes to
the API.

http://www.pjrc.com/teensy/string_class_experimental.html

If you give this a try (and I certainly hope you will before
commenting), I believe you'll find it works much better than the
existing code. Of course, if you find any bugs, I'll try to get them
fixed ASAP.

I realize there is controversy regarding the string class. Please
understand the scope of this effort was to make the existing String
class (Arduino 0019 to 0022) work as well as it can. Certainly a
completely different API is a longer term possibility, assuming anyone
makes a good implementation (which isn't necessarily easy even if the
API lacks all operators). But in the short term, this code is intended
to dramatically improve what is already in place, with only minimal
user-visible change.... other than changing String to work properly!


-Paul



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Can you summarize the approach / changes / improvements?

On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
> Hi Everyone,
>
> Here is my effort to rewrite the String class.  These files are meant to
> replace the existing String implementation with only minimal changes to the
> API.
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> If you give this a try (and I certainly hope you will before commenting), I
> believe you'll find it works much better than the existing code.  Of course,
> if you find any bugs, I'll try to get them fixed ASAP.
>
> I realize there is controversy regarding the string class.  Please
> understand the scope of this effort was to make the existing String class
> (Arduino 0019 to 0022) work as well as it can.  Certainly a completely
> different API is a longer term possibility, assuming anyone makes a good
> implementation (which isn't necessarily easy even if the API lacks all
> operators).  But in the short term, this code is intended to dramatically
> improve what is already in place, with only minimal user-visible change....
> other than changing String to work properly!
>
>
> -Paul
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David A. Mellis wrote:
> Can you summarize the approach / changes / improvements?
>

If you compare the new and old code side-by-side, you'll see it's a lot
of incremental improvements. While almost everything has been
rewritten, there shouldn't be any really shocking changes.

Here's a list of what I recall...

Several functions were added to accept primitive types (int, long, char,
etc) as right hand side parameters. That solves a lot of inefficient
allocation patterns.

The concat operator "+" uses references to pass its left hand side input
to its output, of course modifying it along the way. Keeping a "same
object" path allows a chain of + operators to work nicely. Together
with left side converted on the stack, realloc's ability to grow the
memory in place is leveraged. An inherited class is used, where the
inheritance rules prevent the compiler from modifying the left-most
input, but allows the final result to be used seamlessly.

Compiler optimization and rvalue refs improve efficiency for
assignments. Please understand these optimizations apply to the results
of functions and expressions, not the operators. These optimizations
are enabled by adding 2 lines in Compiler.java.

An updated malloc() is included. The malloc() shipped with Arduino has
bugs.

All string size changes use realloc. Never is a buffer copied if
realloc can change the space allocation.

Compare and other functions that had read-base buffer overflows are fixed.

Replace and other functions were rewritten to be more efficient.

Search functions that previously created many temporary objects were
rewritten to work only within the original string.

Write-based functions modify the original string. That's far more
efficient in the common case where only the result is wanted. This is
the only API change that should be really visible.

A few new functions were added.

Brian Cook's "no overhead" flash string, as modified by Mikal Hart for
the FlashString library, was included as a constructor and in a limited
number of important left side cases.

Many tests were added to the Arduino Test Suite. Improvements in how
the test suite measures memory usage and CPU time were made.

But most importantly, MANY hours of testing were performed, not just for
correct behavior, but also for reasonably efficient memory allocation
patterns. While I did make a few tiny tweaks to the API, this is
designed to be a drop-in replacement. It's a huge improvement in
quality over what's currently in 0022.


-Paul





> On Wed, Jan 19, 2011 at 6:43 PM, Paul Stoffregen <> wrote:
>
>> Hi Everyone,
>>
>> Here is my effort to rewrite the String class. These files are meant to
>> replace the existing String implementation with only minimal changes to the
>> API.
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> If you give this a try (and I certainly hope you will before commenting), I
>> believe you'll find it works much better than the existing code. Of course,
>> if you find any bugs, I'll try to get them fixed ASAP.
>>
>> I realize there is controversy regarding the string class. Please
>> understand the scope of this effort was to make the existing String class
>> (Arduino 0019 to 0022) work as well as it can. Certainly a completely
>> different API is a longer term possibility, assuming anyone makes a good
>> implementation (which isn't necessarily easy even if the API lacks all
>> operators). But in the short term, this code is intended to dramatically
>> improve what is already in place, with only minimal user-visible change....
>> other than changing String to work properly!
>>
>>
>> -Paul
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 19 January 2011 19:35, Paul Stoffregen <> wrote:

> An updated malloc() is included.  The malloc() shipped with Arduino has
> bugs.

If you are patching malloc you need to do so by submitting the patch
to the avr-libc maintainers, not by including a forked file as part of
the arduino build. Distros may already include avr-gcc & and
avr-libc and use them in preference, and in such cases your changes
would be ineffective.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Were there plans to move to a more recent avr-libc?

(I ask because I'm maintaining some code that would like to use features
that have previously been unavailable due to the older avr-libc...)

= Mike

On Jan 19, 2011, at 1:29 PM, Alan Burlison wrote:

> On 19 January 2011 19:35, Paul Stoffregen <> wrote:
>
>> An updated malloc() is included. The malloc() shipped with Arduino has
>> bugs.
>
> If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers, not by including a forked file as part of
> the arduino build. Distros may already include avr-gcc & and
> avr-libc and use them in preference, and in such cases your changes
> would be ineffective.
>
> --
> Alan Burlison
> --
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > If you are patching malloc you need to do so by submitting the patch
> to the avr-libc maintainers,

It'd be pointless to submit malloc to the avr-libc maintainers, since
it's pretty much only a copy taken from their latest published version
(you'd know that if you'd looked at the code). If/when Arduino updates
its included toolchain, this won't be needed.

However, until Arduino's tools update, simply dropping a new malloc.c
into the hardware/arduino/cores/arduino fixes the immediate problems.

I limited myself to merely fixing the problems with String. When and
how to update the software Arduino uses is a far larger issue than my
efforts to merely make String work.


-Paul


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. >> If you are patching malloc you need to do so by submitting the patch
>> to the avr-libc maintainers,
>
> It'd be pointless to submit malloc to the avr-libc maintainers, since it's
> pretty much only a copy taken from their latest published version (you'd
> know that if you'd looked at the code).  If/when Arduino updates its
> included toolchain, this won't be needed.

I have looked at the code, I compared the malloc.c in your tarball to
that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
from the avr-libc site and there are significant differences, which is
why I assumed you'd made changes that should really be submitted
upstream, hence my comment.

And it might be helpful if you were slightly less aggressive and
didn't jump to (incorrect) conclusions quite so quickly.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I have looked at the code, I compared the malloc.c in your tarball to
> that in the avr-libc 1.7.0 (dated 16-Jun-2010 20:29) which I pulled
> from the avr-libc site and there are significant differences,

All I did was lift the code right out of 1.7.0, delete a bunch of unused
stuff and rearrange and substitute things from headers so it'd compile
as a stand-alone file. Actually, I did add a variable called
__brkval_maximum, and 2 lines of code that update it. I'm sure someone
could redo it much better/cleaner than I did....

But what does all this have to do with making String work nicely for
Arduino users? I'm guessing the point was made earlier, where you wrote
"Distros may already include avr-gcc & and avr-libc and use them in
preference, and in such cases your changes would be ineffective."

That's is absolutely not an issue! The Arduino build process compiles
all the "core" files into a static library. The linker, as used by
Arduino, will always resolve malloc/realloc/free from core.a before the
avr-libc library. If I am mistaken, please show me the specific case?


Anyway, I've published all the code, and unless the current String API
is completely abandoned, I believe this code can substantially benefit
Arduino users. It simply works so much better than what has shipped in
Arduino 0019, 0020, 0021 and 0022. I am contributing this to the
community, in the collaborative spirit of open source.

Over the last few weeks I've poured many, many hours into throughly
testing not only the functionality, but allocation patterns during
actual use. If actual bugs or even usage cases that cause inefficient
memory utilization are found, I'm certainly willing to invest more time
to fix them. I'd really appreciate if such issues were reported in the
form of a reproducible problem.

Other than actual runtime issues, I'm afraid I'm going to have to pull
back from discussions on matters of style.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 01:03, Paul Stoffregen wrote:

> All I did was lift the code right out of 1.7.0, delete a bunch of unused
> stuff and rearrange and substitute things from headers so it'd compile
> as a stand-alone file. Actually, I did add a variable called
> __brkval_maximum, and 2 lines of code that update it. I'm sure someone
> could redo it much better/cleaner than I did....

Right - that makes sense - I couldn't figure out exactly what was
happening with malloc from your initial post. If anyone uses the
variable, it will cause breakage when your version of malloc.c is
removed. If it's generally useful, it might be better submitting it
upstream.

> That's is absolutely not an issue! The Arduino build process compiles
> all the "core" files into a static library. The linker, as used by
> Arduino, will always resolve malloc/realloc/free from core.a before the
> avr-libc library. If I am mistaken, please show me the specific case?

It depends exactly on how the components are linked together, and not
everyone uses the Arduino IDE with the Arduino environment. Ideally the
avr-libc malloc would be declared to be a weak symbol to allow
predicable replacement, but it isn't.

> Over the last few weeks I've poured many, many hours into throughly
> testing not only the functionality, but allocation patterns during
> actual use. If actual bugs or even usage cases that cause inefficient
> memory utilization are found, I'm certainly willing to invest more time
> to fix them. I'd really appreciate if such issues were reported in the
> form of a reproducible problem.

There were a couple of thing in the library I didn't understand and
thought might be potential problems, but I'm rather discouraged from
pursuing them, based on your previous reaction to comments.

> Other than actual runtime issues, I'm afraid I'm going to have to pull
> back from discussions on matters of style.

You can't arbitrarily decide which discussions are off-limits for
something like String.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > It depends exactly on how the components are linked together, and not
> everyone uses the Arduino IDE with the Arduino environment.

Can you provide an example?

> There were a couple of thing in the library I didn't understand and
> thought might be potential problems,

If there are actual problems, please explain how to reproduce them.






_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 13:32, Paul Stoffregen wrote:

>> It depends exactly on how the components are linked together, and not
>> everyone uses the Arduino IDE with the Arduino environment.
>
> Can you provide an example?

The version of malloc used will depend on the ordering of the
libraries/object files on the link line.

>> There were a couple of thing in the library I didn't understand and
>> thought might be potential problems,
>
> If there are actual problems, please explain how to reproduce them.

The constructor that takes a PROGMEM string seems to store it as-is
without reading it from PROGMEM and into SRAM.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > The version of malloc used will depend on the ordering of the
> libraries/object files on the link line.

Can you show me any Makefile or other build system that is genuinely
useful for projects using the Arduino core, where this is actually an issue?


> The constructor that takes a PROGMEM string seems to store it as-is
> without reading it from PROGMEM and into SRAM.

That constructor is used many times during the test suite examples I
provided. Did you run them?

Again, I'm asking for an example. If my code were broken as you suggest
(it isn't), even a very trivial example would quickly reproduce the problem.



_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. On 20/01/2011 14:15, Paul Stoffregen wrote:

> Again, I'm asking for an example. If my code were broken as you suggest
> (it isn't), even a very trivial example would quickly reproduce the
> problem.

I've tried to review your code and ask questions about things that
aren't immediately clear, this being one of them. However your
continued aggressive tone makes it quite impossible to do so. You can't
expect to get any useful feedback from anyone if your response to every
question is just a confrontational counter question - I merely wondered
how the PROGMEM/SRAM copy was managed in that case.

I really can't have a conversation with you if that's how you are going
to respond and I therefore won't be participating in this process any
longer.

That's a shame because after reading your code and having received
answers to a small number comments and queries like the one above, I was
going to suggest that your version was adopted.

--
Alan Burlison
--

_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Alan, I'm frustrated that you keep suggesting I've made mistakes.
You've raised numerous concerns about my code. I'm not necessarily
saying your concerns without merit, but I just don't see how they're
real issues, and you refuse to provide any actual examples!

Why is posting a specific example so hard?

I provided over 120 examples in specific tests, based on Rick and Mark's
test suite and also tests based on adaptations of Tom's code, plus
several more I wrote.

Regarding that constructor, the = operator calls copy(), which uses
strlen_P() and strcpy_P().

Regarding the tone of these messages, it seems to me you're repeatedly
claiming I've done things incorrectly. You've called my approach
"ineffective". You repeatedly suggest problems can occur, but never
give an example. But yes, I could respond more softly. I'll try to do
so in the future.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Hang in there. Don't lose track of the goal. We all have worked hard
on various parts of the software. If we didn't care we wouldn't be so
passionate about our work, and reviewing that work. Working remotely
and virtually on a project is tough. So it's important to step back
from the tone of the conversation and work on the issues. There are
some very practical steps that can be taken to work this out.

Having the current String library unreliable and breakable is not a good thing.

A few more examples, and a bit discussion could lead to an upstream
patch request could be made for changes that need to be made. In the
meantime this could replace the current string library or the current
string library could be pulled out of core until something acceptable
is found.

--Rick

On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
> raised numerous concerns about my code.  I'm not necessarily saying your
> concerns without merit, but I just don't see how they're real issues, and
> you refuse to provide any actual examples!
>
> Why is posting a specific example so hard?
>
> I provided over 120 examples in specific tests, based on Rick and Mark's
> test suite and also tests based on adaptations of Tom's code, plus several
> more I wrote.
>
> Regarding that constructor, the = operator calls copy(), which uses
> strlen_P() and strcpy_P().
>
> Regarding the tone of these messages, it seems to me you're repeatedly
> claiming I've done things incorrectly.  You've called my approach
> "ineffective".  You repeatedly suggest problems can occur, but never give an
> example.  But yes, I could respond more softly.  I'll try to do so in the
> future.
>
>
> -Paul
>
>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.

t.

On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:

> Hang in there. Don't lose track of the goal. We all have worked hard
> on various parts of the software. If we didn't care we wouldn't be so
> passionate about our work, and reviewing that work. Working remotely
> and virtually on a project is tough. So it's important to step back
> from the tone of the conversation and work on the issues. There are
> some very practical steps that can be taken to work this out.
>
> Having the current String library unreliable and breakable is not a good thing.
>
> A few more examples, and a bit discussion could lead to an upstream
> patch request could be made for changes that need to be made. In the
> meantime this could replace the current string library or the current
> string library could be pulled out of core until something acceptable
> is found.
>
> --Rick
>
> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>> Alan, I'm frustrated that you keep suggesting I've made mistakes. You've
>> raised numerous concerns about my code. I'm not necessarily saying your
>> concerns without merit, but I just don't see how they're real issues, and
>> you refuse to provide any actual examples!
>>
>> Why is posting a specific example so hard?
>>
>> I provided over 120 examples in specific tests, based on Rick and Mark's
>> test suite and also tests based on adaptations of Tom's code, plus several
>> more I wrote.
>>
>> Regarding that constructor, the = operator calls copy(), which uses
>> strlen_P() and strcpy_P().
>>
>> Regarding the tone of these messages, it seems to me you're repeatedly
>> claiming I've done things incorrectly. You've called my approach
>> "ineffective". You repeatedly suggest problems can occur, but never give an
>> example. But yes, I could respond more softly. I'll try to do so in the
>> future.
>>
>>
>> -Paul
>>
>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. How about we create a branch just for this change, and then allow
people to test it out more widely. We could let it cook for a bit, and
then determine to merge in.

--Rick

On Thu, Jan 20, 2011 at 3:25 PM, Tom Igoe <> wrote:
> I gave Paul's code some quick and dirty tests before he posted it here, and didn't have any luck breaking it. On the other hand, I did see fixes to a few things that were I had managed to break before. So from a neophyte user's point of view, I think it's a vast improvement on the library.
>
> t.
>
> On Jan 20, 2011, at 11:04 AM, Rick Anderson wrote:
>
>> Hang in there. Don't lose track of the goal. We all have worked hard
>> on various parts of the software.  If we didn't care we wouldn't be so
>> passionate about our work, and reviewing that work.  Working remotely
>> and virtually on a project is tough. So it's important to step back
>> from the tone of the conversation and work on the issues. There are
>> some very practical steps that can be taken to work this out.
>>
>> Having the current String library unreliable and breakable is not a good thing.
>>
>> A few more examples, and a bit discussion could lead to an upstream
>> patch request could be made for changes that need to be made. In the
>> meantime this could replace the current string library or the current
>> string library could be pulled out of core until something acceptable
>> is found.
>>
>> --Rick
>>
>> On Thu, Jan 20, 2011 at 10:31 AM, Paul Stoffregen <> wrote:
>>> Alan, I'm frustrated that you keep suggesting I've made mistakes.  You've
>>> raised numerous concerns about my code.  I'm not necessarily saying your
>>> concerns without merit, but I just don't see how they're real issues, and
>>> you refuse to provide any actual examples!
>>>
>>> Why is posting a specific example so hard?
>>>
>>> I provided over 120 examples in specific tests, based on Rick and Mark's
>>> test suite and also tests based on adaptations of Tom's code, plus several
>>> more I wrote.
>>>
>>> Regarding that constructor, the = operator calls copy(), which uses
>>> strlen_P() and strcpy_P().
>>>
>>> Regarding the tone of these messages, it seems to me you're repeatedly
>>> claiming I've done things incorrectly.  You've called my approach
>>> "ineffective".  You repeatedly suggest problems can occur, but never give an
>>> example.  But yes, I could respond more softly.  I'll try to do so in the
>>> future.
>>>
>>>
>>> -Paul
>>>
>>>
>>>
>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Rick Anderson wrote:
> How about we create a branch just for this change, and then allow
> people to test it out more widely. We could let it cook for a bit, and
> then determine to merge in.
>

Anything that gets more people to actually give the code and try and
evaluate how well it does (or doesn't) work seems like a good idea.
Would a branch achieve that?

I'm not very experienced with git, so I'd prefer someone else do it.
Please feel free to grab the files and put them into a branch however
you feel is best. They're still here:

http://www.pjrc.com/teensy/string_class_experimental.html

I'd really like to start working on making String useful, which mostly
involves integration with the stream class (issue #454). But it seems
premature to pour lots of effort into integration while the status of
String itself is so uncertain.


-Paul




_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Best thing to do is to follow Rick's guide for git development, clone it yourself, then submit a pull request. That way people can get at it and test it. When Dave's back from his conference, if it's all good, he'll merge it.

Rick's guide:
http://code.google.com/p/arduino/wiki/GitandGithubGuideforArduinoDevelopment


On Jan 21, 2011, at 2:09 PM, Paul Stoffregen wrote:

> Rick Anderson wrote:
>> How about we create a branch just for this change, and then allow
>> people to test it out more widely. We could let it cook for a bit, and
>> then determine to merge in.
>>
>
> Anything that gets more people to actually give the code and try and evaluate how well it does (or doesn't) work seems like a good idea. Would a branch achieve that?
>
> I'm not very experienced with git, so I'd prefer someone else do it. Please feel free to grab the files and put them into a branch however you feel is best. They're still here:
>
> http://www.pjrc.com/teensy/string_class_experimental.html
>
> I'd really like to start working on making String useful, which mostly involves integration with the stream class (issue #454). But it seems premature to pour lots of effort into integration while the status of String itself is so uncertain.
>
>
> -Paul
>
>
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Is anyone still looking at this String class code?

Alan, did you find any cases where the StringSumHelper class could go
wrong? Or issues with anonymous temporaries and other operator related
concerns you had?

Regarding a StringBuilder-like insert() function, I certainly could add
one if it's strongly desired. The basic approach already exists inside
the replace() function.

Regarding a pull request, should it include only the change to the core,
or also gcc options added to Compiler.java? I'm guessing the test suite
would be considered separately? Before going to all that git trouble,
it'd be good to hear from David, if this is even the direction Arduino
is going to keep going, or if the API is going to change to something
different?

I'm interested to work on integration with the stream class. I've
already invested many, many long hours fixing the String class, only to
hear about half way through that David may decide to abandon the API
completely! Sometimes it's amazing how challenging it can be to
contribute to an open source project. I really want to help you guys,
and all Arduino users. At the moment I'm doing nothing when I could be
doing much much more, pretty much due to lack of input and response.


-Paul


Tom Igoe wrote:
> Best thing to do is to follow Rick's guide for git development, clone it yourself, then submit a pull request. That way people can get at it and test it. When Dave's back from his conference, if it's all good, he'll merge it.
>
> Rick's guide:
> http://code.google.com/p/arduino/wiki/GitandGithubGuideforArduinoDevelopment
>
>
> On Jan 21, 2011, at 2:09 PM, Paul Stoffregen wrote:
>
>
>> Rick Anderson wrote:
>>
>>> How about we create a branch just for this change, and then allow
>>> people to test it out more widely. We could let it cook for a bit, and
>>> then determine to merge in.
>>>
>>>
>> Anything that gets more people to actually give the code and try and evaluate how well it does (or doesn't) work seems like a good idea. Would a branch achieve that?
>>
>> I'm not very experienced with git, so I'd prefer someone else do it. Please feel free to grab the files and put them into a branch however you feel is best. They're still here:
>>
>> http://www.pjrc.com/teensy/string_class_experimental.html
>>
>> I'd really like to start working on making String useful, which mostly involves integration with the stream class (issue #454). But it seems premature to pour lots of effort into integration while the status of String itself is so uncertain.
>>
>>
>> -Paul
>>
>>
>>
>>
>
>
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.