I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.

Please let me respond with a question. What API sacrifices would you be
willing to suffer in exchange for (the eventual possibility of) a very
easy to use and very reliable (if CPU inefficient) string library? Or
rather than sacrifice, how about merely delay release portions in 0019?

The main difficulty I see is the need for a continuous buffer. If the
API didn't expose the internal buffer, that would free us (bad pun) to
craft internal memory management in almost any way. I would imagine
using small, fixed size, movable blocks, but the point is *anybody*
could come up with a better way and the API wouldn't limit their freedom
to improve the library. Maybe one determined individual will toil on it
day and night and turn out something quickly (maybe even me), or maybe
it'll develop slowly as many people collaborate and gradually improve
the library. But if you publish an API now in 0019 that ties our hands,
you will forever constrain the possible improvements anyone could
contribute.

I know a C++ string library without C-style access to the buffer would
be absolutely crazy on a PC. But really, is direct access to the
internal buffer really needed in Arduino? If you're trying to make
things simpler for beginners, maybe omitting C-style access could be a
net benefit?

This isn't a hugely drastic idea. All I'm proposing is commenting out
parts of the code that expose the internal buffer (and maybe other
things if anything thinks will be a roadblock to improvement), at least
for 0019. If they're wanted later, it'll be easy to just put the code
right back in. I believe the only other required change would be in
Print.cpp, to avoid using toCharArray(). For example:

void Print::print(const String &s)
{
for (int n=0; ; n++) {
char c = s.charAt(n);
if (!c) return;
print(c);
}
}

I know this probably sounds like a crazy idea, but if you want really
substantial contributions, now is your last chance to limit the API.


-Paul






> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>
> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>
> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>
> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>
>
>
>
> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>
>
>> My 2 bits, keep String in the core. Some will use it, some won't.
>>
>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>
>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>
>> giuliano
>>
>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>
>>
>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>
>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>
>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>
>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>
>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>
>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>
>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>
>>>> Mikal
>>>>
>>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.

Please let me respond with a question. What API sacrifices would you be
willing to suffer in exchange for (the eventual possibility of) a very
easy to use and very reliable (if CPU inefficient) string library? Or
rather than sacrifice, how about merely delay release portions in 0019?

The main difficulty I see is the need for a continuous buffer. If the
API didn't expose the internal buffer, that would free us (bad pun) to
craft internal memory management in almost any way. I would imagine
using small, fixed size, movable blocks, but the point is *anybody*
could come up with a better way and the API wouldn't limit their freedom
to improve the library. Maybe one determined individual will toil on it
day and night and turn out something quickly (maybe even me), or maybe
it'll develop slowly as many people collaborate and gradually improve
the library. But if you publish an API now in 0019 that ties our hands,
you will forever constrain the possible improvements anyone could
contribute.

I know a C++ string library without C-style access to the buffer would
be absolutely crazy on a PC. But really, is direct access to the
internal buffer really needed in Arduino? If you're trying to make
things simpler for beginners, maybe omitting C-style access could be a
net benefit?

This isn't a hugely drastic idea. All I'm proposing is commenting out
parts of the code that expose the internal buffer (and maybe other
things if anything thinks will be a roadblock to improvement), at least
for 0019. If they're wanted later, it'll be easy to just put the code
right back in. I believe the only other required change would be in
Print.cpp, to avoid using toCharArray(). For example:

void Print::print(const String &s)
{
for (int n=0; ; n++) {
char c = s.charAt(n);
if (!c) return;
print(c);
}
}

I know this probably sounds like a crazy idea, but if you want really
substantial contributions, now is your last chance to limit the API.


-Paul






> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>
> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>
> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>
> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>
>
>
>
> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>
>
>> My 2 bits, keep String in the core. Some will use it, some won't.
>>
>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>
>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>
>> giuliano
>>
>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>
>>
>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>
>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>
>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>
>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>
>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>
>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>
>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>
>>>> Mikal
>>>>
>>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David,

Bravo for such a clear articulation of the guiding philosophy of the Arduino environment. I would really encourage you to craft a summary of this and put in prominently on the Arduino home page and incorporate it into the public description of Arduino — perhaps an Arduino credo of sorts.

It seems that there are often lots of discussions that pop up in the forums about adding this feature or changing that feature, and in the ensuing debates there is always a lot of back and forth about the philosophy. There will always be those (perhaps healthy) debates, but I think it would help the overall culture if people could always be pointed to a clear statement of the philosophy by which design decisions could be evaluated.

And, to my mind this is obviously independent of whatever decision is made about the string library.

.andy

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.

Please let me respond with a question. What API sacrifices would you be
willing to suffer in exchange for (the eventual possibility of) a very
easy to use and very reliable (if CPU inefficient) string library? Or
rather than sacrifice, how about merely delay release portions in 0019?

The main difficulty I see is the need for a continuous buffer. If the
API didn't expose the internal buffer, that would free us (bad pun) to
craft internal memory management in almost any way. I would imagine
using small, fixed size, movable blocks, but the point is *anybody*
could come up with a better way and the API wouldn't limit their freedom
to improve the library. Maybe one determined individual will toil on it
day and night and turn out something quickly (maybe even me), or maybe
it'll develop slowly as many people collaborate and gradually improve
the library. But if you publish an API now in 0019 that ties our hands,
you will forever constrain the possible improvements anyone could
contribute.

I know a C++ string library without C-style access to the buffer would
be absolutely crazy on a PC. But really, is direct access to the
internal buffer really needed in Arduino? If you're trying to make
things simpler for beginners, maybe omitting C-style access could be a
net benefit?

This isn't a hugely drastic idea. All I'm proposing is commenting out
parts of the code that expose the internal buffer (and maybe other
things if anything thinks will be a roadblock to improvement), at least
for 0019. If they're wanted later, it'll be easy to just put the code
right back in. I believe the only other required change would be in
Print.cpp, to avoid using toCharArray(). For example:

void Print::print(const String &s)
{
for (int n=0; ; n++) {
char c = s.charAt(n);
if (!c) return;
print(c);
}
}

I know this probably sounds like a crazy idea, but if you want really
substantial contributions, now is your last chance to limit the API.


-Paul






> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>
> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>
> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>
> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>
>
>
>
> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>
>
>> My 2 bits, keep String in the core. Some will use it, some won't.
>>
>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>
>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>
>> giuliano
>>
>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>
>>
>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>
>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>
>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>
>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>
>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>
>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>
>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>
>>>> Mikal
>>>>
>>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David,

Bravo for such a clear articulation of the guiding philosophy of the Arduino environment. I would really encourage you to craft a summary of this and put in prominently on the Arduino home page and incorporate it into the public description of Arduino — perhaps an Arduino credo of sorts.

It seems that there are often lots of discussions that pop up in the forums about adding this feature or changing that feature, and in the ensuing debates there is always a lot of back and forth about the philosophy. There will always be those (perhaps healthy) debates, but I think it would help the overall culture if people could always be pointed to a clear statement of the philosophy by which design decisions could be evaluated.

And, to my mind this is obviously independent of whatever decision is made about the string library.

.andy

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Paul, forgive me, I've got four other projects in my head on deadline before I leave tomorrow, so had to release the String details from my head. Which functions would we lose to do this?

t.

On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:

>
>> I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.
>
> Please let me respond with a question. What API sacrifices would you be willing to suffer in exchange for (the eventual possibility of) a very easy to use and very reliable (if CPU inefficient) string library? Or rather than sacrifice, how about merely delay release portions in 0019?
>
> The main difficulty I see is the need for a continuous buffer. If the API didn't expose the internal buffer, that would free us (bad pun) to craft internal memory management in almost any way. I would imagine using small, fixed size, movable blocks, but the point is *anybody* could come up with a better way and the API wouldn't limit their freedom to improve the library. Maybe one determined individual will toil on it day and night and turn out something quickly (maybe even me), or maybe it'll develop slowly as many people collaborate and gradually improve the library. But if you publish an API now in 0019 that ties our hands, you will forever constrain the possible improvements anyone could contribute.
>
> I know a C++ string library without C-style access to the buffer would be absolutely crazy on a PC. But really, is direct access to the internal buffer really needed in Arduino? If you're trying to make things simpler for beginners, maybe omitting C-style access could be a net benefit?
>
> This isn't a hugely drastic idea. All I'm proposing is commenting out parts of the code that expose the internal buffer (and maybe other things if anything thinks will be a roadblock to improvement), at least for 0019. If they're wanted later, it'll be easy to just put the code right back in. I believe the only other required change would be in Print.cpp, to avoid using toCharArray(). For example:
>
> void Print::print(const String &s)
> {
> for (int n=0; ; n++) {
> char c = s.charAt(n);
> if (!c) return;
> print(c);
> }
> }
>
> I know this probably sounds like a crazy idea, but if you want really substantial contributions, now is your last chance to limit the API.
>
>
> -Paul
>
>
>
>
>
>
>> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>>
>> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>>
>> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>>
>> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>>
>>
>>
>>
>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>
>>
>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>
>>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>>
>>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>>
>>> giuliano
>>>
>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>
>>>
>>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>>
>>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>
>>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>>
>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>>
>>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>>
>>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>>
>>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>>
>>>>> Mikal
>>>>>
>>>>>
>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.

Please let me respond with a question. What API sacrifices would you be
willing to suffer in exchange for (the eventual possibility of) a very
easy to use and very reliable (if CPU inefficient) string library? Or
rather than sacrifice, how about merely delay release portions in 0019?

The main difficulty I see is the need for a continuous buffer. If the
API didn't expose the internal buffer, that would free us (bad pun) to
craft internal memory management in almost any way. I would imagine
using small, fixed size, movable blocks, but the point is *anybody*
could come up with a better way and the API wouldn't limit their freedom
to improve the library. Maybe one determined individual will toil on it
day and night and turn out something quickly (maybe even me), or maybe
it'll develop slowly as many people collaborate and gradually improve
the library. But if you publish an API now in 0019 that ties our hands,
you will forever constrain the possible improvements anyone could
contribute.

I know a C++ string library without C-style access to the buffer would
be absolutely crazy on a PC. But really, is direct access to the
internal buffer really needed in Arduino? If you're trying to make
things simpler for beginners, maybe omitting C-style access could be a
net benefit?

This isn't a hugely drastic idea. All I'm proposing is commenting out
parts of the code that expose the internal buffer (and maybe other
things if anything thinks will be a roadblock to improvement), at least
for 0019. If they're wanted later, it'll be easy to just put the code
right back in. I believe the only other required change would be in
Print.cpp, to avoid using toCharArray(). For example:

void Print::print(const String &s)
{
for (int n=0; ; n++) {
char c = s.charAt(n);
if (!c) return;
print(c);
}
}

I know this probably sounds like a crazy idea, but if you want really
substantial contributions, now is your last chance to limit the API.


-Paul






> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>
> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>
> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>
> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>
>
>
>
> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>
>
>> My 2 bits, keep String in the core. Some will use it, some won't.
>>
>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>
>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>
>> giuliano
>>
>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>
>>
>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>
>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>
>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>
>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>
>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>
>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>
>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>
>>>> Mikal
>>>>
>>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David,

Bravo for such a clear articulation of the guiding philosophy of the Arduino environment. I would really encourage you to craft a summary of this and put in prominently on the Arduino home page and incorporate it into the public description of Arduino — perhaps an Arduino credo of sorts.

It seems that there are often lots of discussions that pop up in the forums about adding this feature or changing that feature, and in the ensuing debates there is always a lot of back and forth about the philosophy. There will always be those (perhaps healthy) debates, but I think it would help the overall culture if people could always be pointed to a clear statement of the philosophy by which design decisions could be evaluated.

And, to my mind this is obviously independent of whatever decision is made about the string library.

.andy

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Paul, forgive me, I've got four other projects in my head on deadline before I leave tomorrow, so had to release the String details from my head. Which functions would we lose to do this?

t.

On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:

>
>> I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.
>
> Please let me respond with a question. What API sacrifices would you be willing to suffer in exchange for (the eventual possibility of) a very easy to use and very reliable (if CPU inefficient) string library? Or rather than sacrifice, how about merely delay release portions in 0019?
>
> The main difficulty I see is the need for a continuous buffer. If the API didn't expose the internal buffer, that would free us (bad pun) to craft internal memory management in almost any way. I would imagine using small, fixed size, movable blocks, but the point is *anybody* could come up with a better way and the API wouldn't limit their freedom to improve the library. Maybe one determined individual will toil on it day and night and turn out something quickly (maybe even me), or maybe it'll develop slowly as many people collaborate and gradually improve the library. But if you publish an API now in 0019 that ties our hands, you will forever constrain the possible improvements anyone could contribute.
>
> I know a C++ string library without C-style access to the buffer would be absolutely crazy on a PC. But really, is direct access to the internal buffer really needed in Arduino? If you're trying to make things simpler for beginners, maybe omitting C-style access could be a net benefit?
>
> This isn't a hugely drastic idea. All I'm proposing is commenting out parts of the code that expose the internal buffer (and maybe other things if anything thinks will be a roadblock to improvement), at least for 0019. If they're wanted later, it'll be easy to just put the code right back in. I believe the only other required change would be in Print.cpp, to avoid using toCharArray(). For example:
>
> void Print::print(const String &s)
> {
> for (int n=0; ; n++) {
> char c = s.charAt(n);
> if (!c) return;
> print(c);
> }
> }
>
> I know this probably sounds like a crazy idea, but if you want really substantial contributions, now is your last chance to limit the API.
>
>
> -Paul
>
>
>
>
>
>
>> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>>
>> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>>
>> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>>
>> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>>
>>
>>
>>
>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>
>>
>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>
>>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>>
>>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>>
>>> giuliano
>>>
>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>
>>>
>>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>>
>>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>
>>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>>
>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>>
>>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>>
>>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>>
>>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>>
>>>>> Mikal
>>>>>
>>>>>
>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > Which functions would we lose to do this?
>

getBytes and toCharArray

Does anyone see any others?


-Paul




> t.
>
> On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:
>
>
>>> I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.
>>>
>> Please let me respond with a question. What API sacrifices would you be willing to suffer in exchange for (the eventual possibility of) a very easy to use and very reliable (if CPU inefficient) string library? Or rather than sacrifice, how about merely delay release portions in 0019?
>>
>> The main difficulty I see is the need for a continuous buffer. If the API didn't expose the internal buffer, that would free us (bad pun) to craft internal memory management in almost any way. I would imagine using small, fixed size, movable blocks, but the point is *anybody* could come up with a better way and the API wouldn't limit their freedom to improve the library. Maybe one determined individual will toil on it day and night and turn out something quickly (maybe even me), or maybe it'll develop slowly as many people collaborate and gradually improve the library. But if you publish an API now in 0019 that ties our hands, you will forever constrain the possible improvements anyone could contribute.
>>
>> I know a C++ string library without C-style access to the buffer would be absolutely crazy on a PC. But really, is direct access to the internal buffer really needed in Arduino? If you're trying to make things simpler for beginners, maybe omitting C-style access could be a net benefit?
>>
>> This isn't a hugely drastic idea. All I'm proposing is commenting out parts of the code that expose the internal buffer (and maybe other things if anything thinks will be a roadblock to improvement), at least for 0019. If they're wanted later, it'll be easy to just put the code right back in. I believe the only other required change would be in Print.cpp, to avoid using toCharArray(). For example:
>>
>> void Print::print(const String &s)
>> {
>> for (int n=0; ; n++) {
>> char c = s.charAt(n);
>> if (!c) return;
>> print(c);
>> }
>> }
>>
>> I know this probably sounds like a crazy idea, but if you want really substantial contributions, now is your last chance to limit the API.
>>
>>
>> -Paul
>>
>>
>>
>>
>>
>>
>>
>>> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>>>
>>> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>>>
>>> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>>>
>>> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>>>
>>>
>>>
>>>
>>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>>
>>>
>>>
>>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>>
>>>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>>>
>>>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>>>
>>>> giuliano
>>>>
>>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>>
>>>>
>>>>
>>>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>>>
>>>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>>
>>>>>
>>>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>>>
>>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>>>
>>>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>>>
>>>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>>>
>>>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>>>
>>>>>> Mikal
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.

I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.



Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.



If your sketch uses String, you will never have confidence in its reliability unless you've done a careful analysis of its memory usage at each operation. And you'll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like



str.replace("0123456789", "ABCDEFGHIJ");



calls malloc() at least 7 times.



I'm not comfortable with this, and I wouldn't let my students use it.



And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don't think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that's fine, but that's what libraries are for. I don't think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.



What do you think?



Mikal I agree with Mikal, String should be a library class.

Mark

On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment. Specifically the whole
> idea of Strings spooks me.
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations. This makes it easy for even innocent constructs to cause
> weird failures. Because RAM gets clobbered, these are often
> devilishly difficult to diagnose.
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you've done a careful analysis of its memory usage
> at each operation. And you'll need to repeat that analysis every time
> your program grows. This will require being intimate with
> implementation details. For example, you need to be aware that a call
> like
>
> str.replace("0123456789", "ABCDEFGHIJ");
>
> calls malloc() at least 7 times.
>
> I'm not comfortable with this, and I wouldn't let my students use it.
>
> And yet I realize that there is some modest utility in having a String
> class around. But does it belong in the core? I don't think so. The
> core should be reserved, as David once explained to me, for those very
> basic and essential (and reliable!) tools that define Arduino. If you
> want to build upon these fundamentals, that's fine, but that's what
> libraries are for. I don't think String qualifies to stand up there
> next to the venerable digitalWrite, delay, shiftOut, etc.
>
> What do you think?
>
> Mikal
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. I'm biased here, since I started the original TextStiring library a few years ago, and have been using it, Hernando's upgrades, and various other methods for String functionality for a long time. Having spent the last few weeks working with String, I'm convinced that it's finally at a point where its utility is far more than modest. It's invaluable for any network-based work, and allows so much more ease-of-use for things like HTTP, AT commands, and so forth. If you're an experienced programmer and thoroughly comfortable with memory management, perhaps it's a modest gain, but for the majority of my students, and many arduino users who are not strong C programmers, it's likely to be an essential part of their work. I see it as fundamental.

That said, I am just as happy with it being as core or library, but I would not want to do without it.

t.


On Aug 16, 2010, at 8:11 AM, Mark Sproul wrote:

> I agree with Mikal, String should be a library class.
>
> Mark
>
> On 8/16/10 2:01 AM, Hart, Mikal N wrote:
>> I've never been very comfortable with classes that depend on dynamic allocation in a microcontroller environment. Specifically the whole idea of Strings spooks me.
>>
>> Our new core String class uses unchecked allocation for all nontrivial operations. This makes it easy for even innocent constructs to cause weird failures. Because RAM gets clobbered, these are often devilishly difficult to diagnose.
>>
>> If your sketch uses String, you will never have confidence in its reliability unless you’ve done a careful analysis of its memory usage at each operation. And you’ll need to repeat that analysis every time your program grows. This will require being intimate with implementation details. For example, you need to be aware that a call like
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>> calls malloc() at least 7 times.
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>> And yet I realize that there is some modest utility in having a String class around. But does it belong in the core? I don’t think so. The core should be reserved, as David once explained to me, for those very basic and essential (and reliable!) tools that define Arduino. If you want to build upon these fundamentals, that’s fine, but that’s what libraries are for. I don’t think String qualifies to stand up there next to the venerable digitalWrite, delay, shiftOut, etc.
>>
>> What do you think?
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Mikal,

Thank you for raising this issue. This kind of discussion plays an
important role in clarifying the philosophy and approach of the
Arduino platform. I'll try to explain how I'm thinking about it, but
I'm curious to hear more about what you and others think.

On the one hand, I agree that dynamic allocation (and most uses of
classes or other C++ features) is uneasy on a microcontroller, where
you have limited RAM and few possibilities for debugging output. I
know the string class badly needs error checking. I'd also like to
keep its use optional for those who would rather be more explicit and
low-level with their handling of string data.

On the other hand, we're trying to make microcontrollers useful for
new groups of people, and a string class opens up big new domains of
functionality for those people. In the same way that digitalWrite(13,
HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
(1 << PB4), a statement like s = s + 'a' can be used by people and in
situations that could never handle: s = realloc(s, strlen(s) + 1); if
(s) strcat(s, 'a'). It's just a whole different level of abstraction
and understanding.

For something as fundamental and useful as strings, it seems important
to include the implementation in the distribution itself, in a way
that allows it to be used naturally with other functionality (e.g.
Serial.print()). This probably doesn't technically need to be in the
core, but that was the easiest initial implementation. Do you have
ideas about how it could be provided as a library but integrate
cleanly with the core?

In general, I think there's a place for a framework that abstracts the
messy details of AVR C (e.g. register manipulation) such that it's
comfortable to a C programmer who's used to things like realloc() and
strcat(). But with Arduino we're trying to reach people that will
probably never be comfortable with those constructs, but still need to
get complex things done. That means that we're probably stuck
somewhere in the messy realm between proper embedded development and
full-featured desktop programming, with attendant tradeoffs in
ease-of-use and efficiency / reliability.

Anyway, that's my perspective. Other thoughts welcome.

David

On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
> I've never been very comfortable with classes that depend on dynamic
> allocation in a microcontroller environment.  Specifically the whole idea of
> Strings spooks me.
>
>
>
> Our new core String class uses unchecked allocation for all nontrivial
> operations.  This makes it easy for even innocent constructs to cause weird
> failures.  Because RAM gets clobbered, these are often devilishly difficult
> to diagnose.
>
>
>
> If your sketch uses String, you will never have confidence in its
> reliability unless you’ve done a careful analysis of its memory usage at
> each operation.  And you’ll need to repeat that analysis every time your
> program grows.  This will require being intimate with implementation
> details.  For example, you need to be aware that a call like
>
>
>
>       str.replace("0123456789", "ABCDEFGHIJ");
>
>
>
> calls malloc() at least 7 times.
>
>
>
> I’m not comfortable with this, and I wouldn’t let my students use it.
>
>
>
> And yet I realize that there is some modest utility in having a String class
> around.  But does it belong in the core?  I don’t think so.  The core should
> be reserved, as David once explained to me, for those very basic and
> essential (and reliable!) tools that define Arduino.  If you want to build
> upon these fundamentals, that’s fine, but that’s what libraries are for.  I
> don’t think String qualifies to stand up there next to the venerable
> digitalWrite, delay, shiftOut, etc.
>
>
>
> What do you think?
>
>
>
> Mikal
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. great post Dave

exactly what I think about the issue.

m

On Mon, Aug 16, 2010 at 17:11, David A. Mellis <> wrote:

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <>
> wrote:
> > I've never been very comfortable with classes that depend on dynamic
> > allocation in a microcontroller environment. Specifically the whole idea
> of
> > Strings spooks me.
> >
> >
> >
> > Our new core String class uses unchecked allocation for all nontrivial
> > operations. This makes it easy for even innocent constructs to cause
> weird
> > failures. Because RAM gets clobbered, these are often devilishly
> difficult
> > to diagnose.
> >
> >
> >
> > If your sketch uses String, you will never have confidence in its
> > reliability unless you’ve done a careful analysis of its memory usage at
> > each operation. And you’ll need to repeat that analysis every time your
> > program grows. This will require being intimate with implementation
> > details. For example, you need to be aware that a call like
> >
> >
> >
> > str.replace("0123456789", "ABCDEFGHIJ");
> >
> >
> >
> > calls malloc() at least 7 times.
> >
> >
> >
> > I’m not comfortable with this, and I wouldn’t let my students use it.
> >
> >
> >
> > And yet I realize that there is some modest utility in having a String
> class
> > around. But does it belong in the core? I don’t think so. The core
> should
> > be reserved, as David once explained to me, for those very basic and
> > essential (and reliable!) tools that define Arduino. If you want to
> build
> > upon these fundamentals, that’s fine, but that’s what libraries are for.
> I
> > don’t think String qualifies to stand up there next to the venerable
> > digitalWrite, delay, shiftOut, etc.
> >
> >
> >
> > What do you think?
> >
> >
> >
> > Mikal
> >
> > _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.

I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.

But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.

The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.

Thanks for this thread. This is a topic that has interested me for many years.

Mikal




-----Original Message-----
Sent: Monday, August 16, 2010 10:58 AM
Cc: Hart, Mikal N; ; Arduino Developers
Subject: Re: [Developers] [Team] RFC: String in the core?


> That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>

+1 for ease-of-use vs efficiency
-1 for ease-of-use vs reliability

At the very least, if the String class is known to be unreliable, that
ugly truth should be clearly and prominently stated everywhere the
String class is mentioned.

The name "Arduino" has a very strong reputation, mostly very positive.
However, Arduino is also also widely seen as a hobbyist or "not for
serious use" platform. High quality code will gradually strengthen the
Arduino "brand" or reputation. A string library without even buffer
overflow checking would not be a positive step in that direction.



> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you've done a careful analysis of its memory usage at
>> each operation. And you'll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I'm not comfortable with this, and I wouldn't let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don't think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that's fine, but that's what libraries are for. I
>> don't think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.
I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.

I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.

Mark




On 8/16/10 1:31 PM, Hart, Mikal N wrote:
> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>
> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>
> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>
> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>
> Thanks for this thread. This is a topic that has interested me for many years.
>
> Mikal
>


_______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. My 2 bits, keep String in the core. Some will use it, some won't.

However, what is really needed is some means to detect RAM exhaustion.
malloc returning 0 is the normal way to do this. The String class
should do something with this. Print a diagnostic, set an error bit in
the string, call a registered function, just something so folks have a
chance of figuring out what the hell happened.

Frankly, I'd love there to be some way to detect RAM exhaustion when
the stack runs out of memory also.

giuliano

On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:

> I agree with Mikal completely. I never use the String library. Due
> to the memory limitations I never use malloc of any kind. To
> dangerous.
>
> I do agree with the other comments that it is very useful for things
> like dealing with HTML. If you are using HTML, then you are already
> using the ethernet libraries, why not make the String stuff a
> library just like Ethernet. If you aren't doing stuff using
> libraries such as ethenet, xbee, modem communications or other, you
> probably aren't using String either.
>
> Mark
>
>
>
>
> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>> Firstly, Tom, apologies for the choice of the word "modest". It
>> wasn't my goal to denigrate String, rather to point out that there
>> are some fundamental differences between it and what we currently
>> find in the core.
>>
>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>> abstraction. And I agree wholeheartedly that String is a similarly
>> useful and compelling improvement on realloc/strcat.
>>
>> But whereas reading and writing ports is a fundamental and
>> necessary operation that was in dire need of accessible
>> abstraction, I disagree about the dynamic allocation of buffers.
>> Far from arguing that students should be compelled to learn the ins
>> and outs of realloc, I personally believe that no one should use
>> dynamic allocation except perhaps in a few narrow, well-understood
>> niches. I never do in my libraries and sketches on the grounds
>> that if you can predict your memory requirements, you should
>> statically allocate your buffers. And if you can't, well, with
>> only a few hundred bytes shielding you from disaster, you probably
>> ought to rethink your design.
>>
>> The problem with String is that it is indeed a very easy,
>> seductive, and elegant abstraction. It's concerning that for the
>> first time in Arduino history we are introducing core code that can
>> rather easily crash your project unless you are lucky enough to
>> have a small memory footprint or can take the time to carefully
>> analyze the ramifications of str += Serial.read(). Unfortunately,
>> the very people who find the String abstraction most seductive are
>> those the least equipped to perform that analysis or diagnose the
>> weird crash that may follow. Paul's point about the possible
>> impact on the Arduino brand is something to think about.
>>
>> Thanks for this thread. This is a topic that has interested me for
>> many years.
>>
>> Mikal
>>
>
>
> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Regardless of whether it's core or a library, I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible. I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.

MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.

Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?

Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.




On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:

> My 2 bits, keep String in the core. Some will use it, some won't.
>
> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>
> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>
> giuliano
>
> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>
>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>
>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>
>> Mark
>>
>>
>>
>>
>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>
>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>
>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>
>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>
>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>
>>> Mikal
>>>
>>
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.

Please let me respond with a question. What API sacrifices would you be
willing to suffer in exchange for (the eventual possibility of) a very
easy to use and very reliable (if CPU inefficient) string library? Or
rather than sacrifice, how about merely delay release portions in 0019?

The main difficulty I see is the need for a continuous buffer. If the
API didn't expose the internal buffer, that would free us (bad pun) to
craft internal memory management in almost any way. I would imagine
using small, fixed size, movable blocks, but the point is *anybody*
could come up with a better way and the API wouldn't limit their freedom
to improve the library. Maybe one determined individual will toil on it
day and night and turn out something quickly (maybe even me), or maybe
it'll develop slowly as many people collaborate and gradually improve
the library. But if you publish an API now in 0019 that ties our hands,
you will forever constrain the possible improvements anyone could
contribute.

I know a C++ string library without C-style access to the buffer would
be absolutely crazy on a PC. But really, is direct access to the
internal buffer really needed in Arduino? If you're trying to make
things simpler for beginners, maybe omitting C-style access could be a
net benefit?

This isn't a hugely drastic idea. All I'm proposing is commenting out
parts of the code that expose the internal buffer (and maybe other
things if anything thinks will be a roadblock to improvement), at least
for 0019. If they're wanted later, it'll be easy to just put the code
right back in. I believe the only other required change would be in
Print.cpp, to avoid using toCharArray(). For example:

void Print::print(const String &s)
{
for (int n=0; ; n++) {
char c = s.charAt(n);
if (!c) return;
print(c);
}
}

I know this probably sounds like a crazy idea, but if you want really
substantial contributions, now is your last chance to limit the API.


-Paul






> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>
> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>
> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>
> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>
>
>
>
> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>
>
>> My 2 bits, keep String in the core. Some will use it, some won't.
>>
>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>
>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>
>> giuliano
>>
>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>
>>
>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>
>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>
>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>
>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>
>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>
>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>
>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>
>>>> Mikal
>>>>
>>>>
>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. David,

Bravo for such a clear articulation of the guiding philosophy of the Arduino environment. I would really encourage you to craft a summary of this and put in prominently on the Arduino home page and incorporate it into the public description of Arduino — perhaps an Arduino credo of sorts.

It seems that there are often lots of discussions that pop up in the forums about adding this feature or changing that feature, and in the ensuing debates there is always a lot of back and forth about the philosophy. There will always be those (perhaps healthy) debates, but I think it would help the overall culture if people could always be pointed to a clear statement of the philosophy by which design decisions could be evaluated.

And, to my mind this is obviously independent of whatever decision is made about the string library.

.andy

> Mikal,
>
> Thank you for raising this issue. This kind of discussion plays an
> important role in clarifying the philosophy and approach of the
> Arduino platform. I'll try to explain how I'm thinking about it, but
> I'm curious to hear more about what you and others think.
>
> On the one hand, I agree that dynamic allocation (and most uses of
> classes or other C++ features) is uneasy on a microcontroller, where
> you have limited RAM and few possibilities for debugging output. I
> know the string class badly needs error checking. I'd also like to
> keep its use optional for those who would rather be more explicit and
> low-level with their handling of string data.
>
> On the other hand, we're trying to make microcontrollers useful for
> new groups of people, and a string class opens up big new domains of
> functionality for those people. In the same way that digitalWrite(13,
> HIGH) saves hours of teaching and conceptual overhead versus PORTB |=
> (1 << PB4), a statement like s = s + 'a' can be used by people and in
> situations that could never handle: s = realloc(s, strlen(s) + 1); if
> (s) strcat(s, 'a'). It's just a whole different level of abstraction
> and understanding.
>
> For something as fundamental and useful as strings, it seems important
> to include the implementation in the distribution itself, in a way
> that allows it to be used naturally with other functionality (e.g.
> Serial.print()). This probably doesn't technically need to be in the
> core, but that was the easiest initial implementation. Do you have
> ideas about how it could be provided as a library but integrate
> cleanly with the core?
>
> In general, I think there's a place for a framework that abstracts the
> messy details of AVR C (e.g. register manipulation) such that it's
> comfortable to a C programmer who's used to things like realloc() and
> strcat(). But with Arduino we're trying to reach people that will
> probably never be comfortable with those constructs, but still need to
> get complex things done. That means that we're probably stuck
> somewhere in the messy realm between proper embedded development and
> full-featured desktop programming, with attendant tradeoffs in
> ease-of-use and efficiency / reliability.
>
> Anyway, that's my perspective. Other thoughts welcome.
>
> David
>
> On Mon, Aug 16, 2010 at 2:01 AM, Hart, Mikal N <> wrote:
>> I've never been very comfortable with classes that depend on dynamic
>> allocation in a microcontroller environment. Specifically the whole idea of
>> Strings spooks me.
>>
>>
>>
>> Our new core String class uses unchecked allocation for all nontrivial
>> operations. This makes it easy for even innocent constructs to cause weird
>> failures. Because RAM gets clobbered, these are often devilishly difficult
>> to diagnose.
>>
>>
>>
>> If your sketch uses String, you will never have confidence in its
>> reliability unless you’ve done a careful analysis of its memory usage at
>> each operation. And you’ll need to repeat that analysis every time your
>> program grows. This will require being intimate with implementation
>> details. For example, you need to be aware that a call like
>>
>>
>>
>> str.replace("0123456789", "ABCDEFGHIJ");
>>
>>
>>
>> calls malloc() at least 7 times.
>>
>>
>>
>> I’m not comfortable with this, and I wouldn’t let my students use it.
>>
>>
>>
>> And yet I realize that there is some modest utility in having a String class
>> around. But does it belong in the core? I don’t think so. The core should
>> be reserved, as David once explained to me, for those very basic and
>> essential (and reliable!) tools that define Arduino. If you want to build
>> upon these fundamentals, that’s fine, but that’s what libraries are for. I
>> don’t think String qualifies to stand up there next to the venerable
>> digitalWrite, delay, shiftOut, etc.
>>
>>
>>
>> What do you think?
>>
>>
>>
>> Mikal
>>
>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Paul, forgive me, I've got four other projects in my head on deadline before I leave tomorrow, so had to release the String details from my head. Which functions would we lose to do this?

t.

On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:

>
>> I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.
>
> Please let me respond with a question. What API sacrifices would you be willing to suffer in exchange for (the eventual possibility of) a very easy to use and very reliable (if CPU inefficient) string library? Or rather than sacrifice, how about merely delay release portions in 0019?
>
> The main difficulty I see is the need for a continuous buffer. If the API didn't expose the internal buffer, that would free us (bad pun) to craft internal memory management in almost any way. I would imagine using small, fixed size, movable blocks, but the point is *anybody* could come up with a better way and the API wouldn't limit their freedom to improve the library. Maybe one determined individual will toil on it day and night and turn out something quickly (maybe even me), or maybe it'll develop slowly as many people collaborate and gradually improve the library. But if you publish an API now in 0019 that ties our hands, you will forever constrain the possible improvements anyone could contribute.
>
> I know a C++ string library without C-style access to the buffer would be absolutely crazy on a PC. But really, is direct access to the internal buffer really needed in Arduino? If you're trying to make things simpler for beginners, maybe omitting C-style access could be a net benefit?
>
> This isn't a hugely drastic idea. All I'm proposing is commenting out parts of the code that expose the internal buffer (and maybe other things if anything thinks will be a roadblock to improvement), at least for 0019. If they're wanted later, it'll be easy to just put the code right back in. I believe the only other required change would be in Print.cpp, to avoid using toCharArray(). For example:
>
> void Print::print(const String &s)
> {
> for (int n=0; ; n++) {
> char c = s.charAt(n);
> if (!c) return;
> print(c);
> }
> }
>
> I know this probably sounds like a crazy idea, but if you want really substantial contributions, now is your last chance to limit the API.
>
>
> -Paul
>
>
>
>
>
>
>> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>>
>> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>>
>> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>>
>> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>>
>>
>>
>>
>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>
>>
>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>
>>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>>
>>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>>
>>> giuliano
>>>
>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>
>>>
>>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>>
>>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>
>>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>>
>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>>
>>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>>
>>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>>
>>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>>
>>>>> Mikal
>>>>>
>>>>>
>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. > Which functions would we lose to do this?
>

getBytes and toCharArray

Does anyone see any others?


-Paul




> t.
>
> On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:
>
>
>>> I would love to see some of you who are passionate about its memory issues take a crack at correcting them, without breaking the API if at all possible.
>>>
>> Please let me respond with a question. What API sacrifices would you be willing to suffer in exchange for (the eventual possibility of) a very easy to use and very reliable (if CPU inefficient) string library? Or rather than sacrifice, how about merely delay release portions in 0019?
>>
>> The main difficulty I see is the need for a continuous buffer. If the API didn't expose the internal buffer, that would free us (bad pun) to craft internal memory management in almost any way. I would imagine using small, fixed size, movable blocks, but the point is *anybody* could come up with a better way and the API wouldn't limit their freedom to improve the library. Maybe one determined individual will toil on it day and night and turn out something quickly (maybe even me), or maybe it'll develop slowly as many people collaborate and gradually improve the library. But if you publish an API now in 0019 that ties our hands, you will forever constrain the possible improvements anyone could contribute.
>>
>> I know a C++ string library without C-style access to the buffer would be absolutely crazy on a PC. But really, is direct access to the internal buffer really needed in Arduino? If you're trying to make things simpler for beginners, maybe omitting C-style access could be a net benefit?
>>
>> This isn't a hugely drastic idea. All I'm proposing is commenting out parts of the code that expose the internal buffer (and maybe other things if anything thinks will be a roadblock to improvement), at least for 0019. If they're wanted later, it'll be easy to just put the code right back in. I believe the only other required change would be in Print.cpp, to avoid using toCharArray(). For example:
>>
>> void Print::print(const String &s)
>> {
>> for (int n=0; ; n++) {
>> char c = s.charAt(n);
>> if (!c) return;
>> print(c);
>> }
>> }
>>
>> I know this probably sounds like a crazy idea, but if you want really substantial contributions, now is your last chance to limit the API.
>>
>>
>> -Paul
>>
>>
>>
>>
>>
>>
>>
>>> I had incorporated some of Mikal's changes in particular into a later version of TextString, and they helped. Not sure if those made the transition into Hernando's latest version because I was focusing on the API/docs end of things, though I am in favor of any reliability enhancements, as long as usability is maintained.
>>>
>>> MIkal, I see lots of places where you can't predict the length of what you're going to get, and have to deal with it. That's where the dynamic allocation need comes from, for me. But if you have alternative strategies that are as simple, I want to hear them. I'd like it if we found a solution that we both could live with, because I think you are as hardcore on one requirement as I am on the other.
>>>
>>> Mark, I agree that not everyone's dealing with string-based devices. However, modules that encapsulate a lot of functionality are showing up a lot more, and they are way useful, for the kinds of folks I work with. I hear not only from students, but from colleagues and readers all the time about how this or that serial device or AT-based device made it possible for them to make something they'd never dreamed of before. That's what I want to see more of. So to me, it means we need to provide reliable ways of encapsulating the functionality needed to deal with them. So if you have a means to detect the end of RAM, I'd love to see it. It could be useful in a lot of places. Got ideas?
>>>
>>> Xiaoyang looked at a few ways of handling some of the messy errors that could pop up when you overrun a string or a memory address, but we didn't come to a resolution. Dave wanted to keep compatibility with Hernando's String library, which is a good idea, but it'd be great if someone on memory wanted to hack away under the hood to improve stability.
>>>
>>>
>>>
>>>
>>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>>
>>>
>>>
>>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>>
>>>> However, what is really needed is some means to detect RAM exhaustion. malloc returning 0 is the normal way to do this. The String class should do something with this. Print a diagnostic, set an error bit in the string, call a registered function, just something so folks have a chance of figuring out what the hell happened.
>>>>
>>>> Frankly, I'd love there to be some way to detect RAM exhaustion when the stack runs out of memory also.
>>>>
>>>> giuliano
>>>>
>>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>>
>>>>
>>>>
>>>>> I agree with Mikal completely. I never use the String library. Due to the memory limitations I never use malloc of any kind. To dangerous.
>>>>>
>>>>> I do agree with the other comments that it is very useful for things like dealing with HTML. If you are using HTML, then you are already using the ethernet libraries, why not make the String stuff a library just like Ethernet. If you aren't doing stuff using libraries such as ethenet, xbee, modem communications or other, you probably aren't using String either.
>>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>>
>>>>>
>>>>>> Firstly, Tom, apologies for the choice of the word "modest". It wasn't my goal to denigrate String, rather to point out that there are some fundamental differences between it and what we currently find in the core.
>>>>>>
>>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx abstraction. And I agree wholeheartedly that String is a similarly useful and compelling improvement on realloc/strcat.
>>>>>>
>>>>>> But whereas reading and writing ports is a fundamental and necessary operation that was in dire need of accessible abstraction, I disagree about the dynamic allocation of buffers. Far from arguing that students should be compelled to learn the ins and outs of realloc, I personally believe that no one should use dynamic allocation except perhaps in a few narrow, well-understood niches. I never do in my libraries and sketches on the grounds that if you can predict your memory requirements, you should statically allocate your buffers. And if you can't, well, with only a few hundred bytes shielding you from disaster, you probably ought to rethink your design.
>>>>>>
>>>>>> The problem with String is that it is indeed a very easy, seductive, and elegant abstraction. It's concerning that for the first time in Arduino history we are introducing core code that can rather easily crash your project unless you are lucky enough to have a small memory footprint or can take the time to carefully analyze the ramifications of str += Serial.read(). Unfortunately, the very people who find the String abstraction most seductive are those the least equipped to perform that analysis or diagnose the weird crash that may follow. Paul's point about the possible impact on the Arduino brand is something to think about.
>>>>>>
>>>>>> Thanks for this thread. This is a topic that has interested me for many years.
>>>>>>
>>>>>> Mikal
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe. Ahh yes, those. Bummer. Until a few days ago, I would have said OK,
but I've run across a few really useful cases where getBytes() is handy.
But since they're both only capable of getting the array and not
changing it, could we perhaps solve this by some error checking before
returning the value, to make sure it doesn't overrun the memory?

Xiaoyang brought up something similar with substring() and we didn't
find a decent way to return the error, because it's not a compile
error. Suggestions?

8/16/2010 06:25 PM, Paul Stoffregen wrote:
>
>> Which functions would we lose to do this?
>
> getBytes and toCharArray
>
> Does anyone see any others?
>
>
> -Paul
>
>
>
>
>> t.
>>
>> On Aug 16, 2010, at 4:47 PM, Paul Stoffregen wrote:
>>
>>>> I would love to see some of you who are passionate about its memory
>>>> issues take a crack at correcting them, without breaking the API if
>>>> at all possible.
>>> Please let me respond with a question. What API sacrifices would
>>> you be willing to suffer in exchange for (the eventual possibility
>>> of) a very easy to use and very reliable (if CPU inefficient) string
>>> library? Or rather than sacrifice, how about merely delay release
>>> portions in 0019?
>>>
>>> The main difficulty I see is the need for a continuous buffer. If
>>> the API didn't expose the internal buffer, that would free us (bad
>>> pun) to craft internal memory management in almost any way. I would
>>> imagine using small, fixed size, movable blocks, but the point is
>>> *anybody* could come up with a better way and the API wouldn't limit
>>> their freedom to improve the library. Maybe one determined
>>> individual will toil on it day and night and turn out something
>>> quickly (maybe even me), or maybe it'll develop slowly as many
>>> people collaborate and gradually improve the library. But if you
>>> publish an API now in 0019 that ties our hands, you will forever
>>> constrain the possible improvements anyone could contribute.
>>>
>>> I know a C++ string library without C-style access to the buffer
>>> would be absolutely crazy on a PC. But really, is direct access to
>>> the internal buffer really needed in Arduino? If you're trying to
>>> make things simpler for beginners, maybe omitting C-style access
>>> could be a net benefit?
>>>
>>> This isn't a hugely drastic idea. All I'm proposing is commenting
>>> out parts of the code that expose the internal buffer (and maybe
>>> other things if anything thinks will be a roadblock to improvement),
>>> at least for 0019. If they're wanted later, it'll be easy to just
>>> put the code right back in. I believe the only other required
>>> change would be in Print.cpp, to avoid using toCharArray(). For
>>> example:
>>>
>>> void Print::print(const String &s)
>>> {
>>> for (int n=0; ; n++) {
>>> char c = s.charAt(n);
>>> if (!c) return;
>>> print(c);
>>> }
>>> }
>>>
>>> I know this probably sounds like a crazy idea, but if you want
>>> really substantial contributions, now is your last chance to limit
>>> the API.
>>>
>>>
>>> -Paul
>>>
>>>
>>>
>>>
>>>
>>>
>>>> I had incorporated some of Mikal's changes in particular into a
>>>> later version of TextString, and they helped. Not sure if those
>>>> made the transition into Hernando's latest version because I was
>>>> focusing on the API/docs end of things, though I am in favor of any
>>>> reliability enhancements, as long as usability is maintained.
>>>>
>>>> MIkal, I see lots of places where you can't predict the length of
>>>> what you're going to get, and have to deal with it. That's where
>>>> the dynamic allocation need comes from, for me. But if you have
>>>> alternative strategies that are as simple, I want to hear them.
>>>> I'd like it if we found a solution that we both could live with,
>>>> because I think you are as hardcore on one requirement as I am on
>>>> the other.
>>>>
>>>> Mark, I agree that not everyone's dealing with string-based
>>>> devices. However, modules that encapsulate a lot of functionality
>>>> are showing up a lot more, and they are way useful, for the kinds
>>>> of folks I work with. I hear not only from students, but from
>>>> colleagues and readers all the time about how this or that serial
>>>> device or AT-based device made it possible for them to make
>>>> something they'd never dreamed of before. That's what I want to
>>>> see more of. So to me, it means we need to provide reliable ways
>>>> of encapsulating the functionality needed to deal with them. So if
>>>> you have a means to detect the end of RAM, I'd love to see it. It
>>>> could be useful in a lot of places. Got ideas?
>>>>
>>>> Xiaoyang looked at a few ways of handling some of the messy errors
>>>> that could pop up when you overrun a string or a memory address,
>>>> but we didn't come to a resolution. Dave wanted to keep
>>>> compatibility with Hernando's String library, which is a good idea,
>>>> but it'd be great if someone on memory wanted to hack away under
>>>> the hood to improve stability.
>>>>
>>>>
>>>>
>>>>
>>>> On Aug 16, 2010, at 2:27 PM, giuliano carlini wrote:
>>>>
>>>>
>>>>> My 2 bits, keep String in the core. Some will use it, some won't.
>>>>>
>>>>> However, what is really needed is some means to detect RAM
>>>>> exhaustion. malloc returning 0 is the normal way to do this. The
>>>>> String class should do something with this. Print a diagnostic,
>>>>> set an error bit in the string, call a registered function, just
>>>>> something so folks have a chance of figuring out what the hell
>>>>> happened.
>>>>>
>>>>> Frankly, I'd love there to be some way to detect RAM exhaustion
>>>>> when the stack runs out of memory also.
>>>>>
>>>>> giuliano
>>>>>
>>>>> On Aug 16, 2010, at 11:10 AM, Mark Sproul wrote:
>>>>>
>>>>>> I agree with Mikal completely. I never use the String library.
>>>>>> Due to the memory limitations I never use malloc of any kind. To
>>>>>> dangerous.
>>>>>>
>>>>>> I do agree with the other comments that it is very useful for
>>>>>> things like dealing with HTML. If you are using HTML, then you
>>>>>> are already using the ethernet libraries, why not make the String
>>>>>> stuff a library just like Ethernet. If you aren't doing stuff
>>>>>> using libraries such as ethenet, xbee, modem communications or
>>>>>> other, you probably aren't using String either.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 8/16/10 1:31 PM, Hart, Mikal N wrote:
>>>>>>> Firstly, Tom, apologies for the choice of the word "modest". It
>>>>>>> wasn't my goal to denigrate String, rather to point out that
>>>>>>> there are some fundamental differences between it and what we
>>>>>>> currently find in the core.
>>>>>>>
>>>>>>> I enjoyed David's nice analogy with the DigitalWrite/PORTx
>>>>>>> abstraction. And I agree wholeheartedly that String is a
>>>>>>> similarly useful and compelling improvement on realloc/strcat.
>>>>>>>
>>>>>>> But whereas reading and writing ports is a fundamental and
>>>>>>> necessary operation that was in dire need of accessible
>>>>>>> abstraction, I disagree about the dynamic allocation of
>>>>>>> buffers. Far from arguing that students should be compelled to
>>>>>>> learn the ins and outs of realloc, I personally believe that no
>>>>>>> one should use dynamic allocation except perhaps in a few
>>>>>>> narrow, well-understood niches. I never do in my libraries and
>>>>>>> sketches on the grounds that if you can predict your memory
>>>>>>> requirements, you should statically allocate your buffers. And
>>>>>>> if you can't, well, with only a few hundred bytes shielding you
>>>>>>> from disaster, you probably ought to rethink your design.
>>>>>>>
>>>>>>> The problem with String is that it is indeed a very easy,
>>>>>>> seductive, and elegant abstraction. It's concerning that for
>>>>>>> the first time in Arduino history we are introducing core code
>>>>>>> that can rather easily crash your project unless you are lucky
>>>>>>> enough to have a small memory footprint or can take the time to
>>>>>>> carefully analyze the ramifications of str += Serial.read().
>>>>>>> Unfortunately, the very people who find the String abstraction
>>>>>>> most seductive are those the least equipped to perform that
>>>>>>> analysis or diagnose the weird crash that may follow. Paul's
>>>>>>> point about the possible impact on the Arduino brand is
>>>>>>> something to think about.
>>>>>>>
>>>>>>> Thanks for this thread. This is a topic that has interested me
>>>>>>> for many years.
>>>>>>>
>>>>>>> Mikal
>>>>>>>
>>>>>> _______________________________________________
___________________________________________________

Posted on the Developers mailing list. Go to http://arduino.cc/mailman/listinfo/developers_arduino.cc to subscribe.