Arduino Forum

General Category => General Discussion => Topic started by: YemSalat on Nov 08, 2015, 03:22 pm

Title: Designing a new programming language for Arduino
Post by: YemSalat on Nov 08, 2015, 03:22 pm
Latest upate: link to post (http://forum.arduino.cc/index.php?topic=358211.msg2528555#msg2528555)

Test the language: http://webcloudtools.com/sprk/lang-test/#hello-world (http://webcloudtools.com/sprk/lang-test/#hello-world)

Project on GitHub: https://github.com/YemSalat/spark-lang (https://github.com/YemSalat/spark-lang)

Hi everyone, I've been away from the forum for a while, but recently I have been lurking around, reading the forum posts, being amazed at how often some of the beginner questions come up. And I started thinking:

What would a language designed specifically for Arduino beginners look like?

I think many would agree that even though Arduino provides a lot of helpful materials (libraries, examples, etc.) - the language that it is based on, C++ - can still be a pretty big obstacle in getting beginner projects done.


Here is a short list of things that in my opinion could be improved if there was a language designed specifically for Arduino beginners:

Simplify syntax, remove the 'cruft'

C++ has a pretty complex syntax, which is full of semicolons, star symbols, parenthesis, etc.
By removing most of this stuff the language would look much more approachable (think Python/Ruby).
For example start by making semicolons optional and allowing for simpler code structures.
E.g. what a for loop variation could look like:

a = 25
repeat 10 times { a += 5 }



Inferred types instead of explicit declaration

Don't get me wrong - I am all for strictly typed languages, especially when there is almost no other way around it on embedded systems. But having to declare types, while very important in terms of programming techniques - has very little to do with what the beginner is trying to achieve, mainly finishing their project.


Better error messages

For example instead of saying something like ".. expected ')' before '{' token or .. at line at line .."

It could say - "You are missing a ')', make sure all your opening and closing parenthesis match on line .."
And overall all the errors could be more beginner friendly.


Tailored to the environment

How many times did you have to tell people that they should not use C++ Strings when programming for Arduino? Wouldn't it be nice if things like that were handled on the language level.
Apart from that, I think it would be very helpful to have some entities from the Arduino and micro controllers realm be present in the language. For example, it could have interrupts 'built-in' as a language structure, etc.

For example setting up an interrupt like this:
-------------------------

interrupt when pin 12 is RISING {
    // do stuff..
}

-------------------------



Convention over flexibility

A 'predefined path' can be very helpful to beginners. Languages that allow less flexibility and more conventional approaches to solving common problems tend to be easier to learn as they allow to focus on the primary goal without thinking too much about the implementation itself.


Taking care of performance issues and errors

Having a language that follows strict conventions - allows for better optimizing code and also preventing errors.
For example -  the compiler can check if a variable is ever used inside an interrupt - and make it volatile if the user forgot to do so.


Handy stuff

Some things in Arduino are harder to do then they should be. Things like concatenating strings, checking if one string is contained in another one, etc.
Wouldn't it be great if you could just write:

foo = "Hello"
foo + " World" // "Hello World"



Different approaches

Perhaps it would be good to try different approaches to programming. A semi-declarative way of programming could be easier for beginners to grasp.
Set up everything, then assign what happens when (e.g. when sensor reading is higher then X - turn on the pump)

Here is a quick 'sketch' of what it could look like.

A program where an LED light blinks when a button is pressed:


-------------------------

device LedLight {

    write:  pinLed // pin number

    action: blink {
        pinLed = HIGH
        wait 5s // does not prevent other code execution
        pinLed = LOW
    }
}


device Button ( LedLight led ) {

    read: pinButton
   
    // no need for multiple IF statements
    when pinButton is HIGH {
        led -> blink()
    }
}

led1 = create LedLight on pin 9
button1 = create Button on pin 1

-------------------------


What do you think? Do you reckon any of these ideas are worth considering?

What in your opinion should a beginner-oriented Arduino language look like? Is such a language even needed?


I am gonna try and come up with more uses cases where such approach can be useful, and who knows maybe one day I'll even make it happen :)


[ADDED]
Here is how I see it working:

device is the main entity in the language (think Class)

Every device follows this structure:

device DeviceName ( constructor parameters ) {

    write: pin1, pin2, pin3 // pins that the device can write to
    read: pin1, pin2, pin3 // pins that the device can read from

    // Here go predefined blocks that are built into the class
    init { ... } // basically constructor
    actions { ... } // all object 'methods'
    always { ... } // this code is executed all the time
    ....

    // then goes a list of all 'events' - code that executes when a condition is satisfied
    // they all start with keyword 'when'
    when pin1 is HIGH { ... }
    when pin2 is < 512 and pin3 > 128 { ... }
    when pin4 is RISING { ... }
    ...

}

So there isn't much code outside the devices. (There can be other entities though, like timers for example, with their own set of "actions" and "events")
PS I do appreciate any criticism though, as it is entirely possible that I am missing some huge flaws in this.
Title: Re: New programming language for beginners: what could it look like?
Post by: pYro_65 on Nov 08, 2015, 03:30 pm
If you can write the compiler, go for it. Once started, you'll see the actual effort required to do this task.
What are you going to write it in?... C++??
Title: Re: New programming language for beginners: what could it look like?
Post by: Robin2 on Nov 08, 2015, 03:33 pm
I think this is probably more appropriate for the Bar Sport section until you come up with a first working draft of your compiler.

(By the way, I agree with a lot of what you say.)

Making the compiler might not be monumentally difficult if you create a program to translate your code into C/C++ that could be processed by the Arduino IDE.

You may be interested in this Mirah (http://www.mirah.org/) project. Look for the Blogs by Charles Nutter who created it. He is very clever and claims that the underlying system can be used for other cross-converting projects.

...R
Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 03:41 pm
Syntax is the easy part.  Anyone can learn the syntax.  The hard part about programming is the logic and that's the same in any language. 
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 03:46 pm
Of course I am not planning to write my own compiler. Just write a grammar in some appropriate notation and use a compiler-compiler to generate the parser.

I wrote a few toy languages back at uni, with modern tools it is really not that hard (relative to writing your own compiler by hand)

The 'hardest' and most interesting part would be coming up with appropriate language structure. If we can come up with something useful - I will generate a compiler for it.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 03:54 pm
Making the compiler might not be monumentally difficult if you create a program to translate your code into C/C++ that could be processed by the Arduino IDE.
That is definitely one way of doing it, plus the Arduino IDE would also take care of all the other parts of the toolchain.
Since it accepts pure C/C++ - the compiler will just need to generate the code which fits with the setup / loop routines.
This way would also be easiest for people who don't already have the Arduino IDE installed

[EDIT] Sorry, I think there might be a confusion here, when I say 'compile' I actually mean transpile the language to C/C++ and then just use either Arduino IDE or avr-gcc directly to compile it to machine code. Writing my own compiler for avr would be crazy, I agree :)

Syntax is the easy part.  Anyone can learn the syntax.  The hard part about programming is the logic and that's the same in any language. 
I disagree, syntax has major effect on how easy it is to understand the code, plus syntax is not everything, the way language is structured is very important as well. Compare C to x86 Assembler for example.
Title: Re: New programming language for beginners: what could it look like?
Post by: UKHeliBob on Nov 08, 2015, 04:45 pm
We surely don't need yet another language.  Whilst it may at first appear that using C++ as the language for Arduino is a hurdle I see it as a positive advantage.  There is tons of help already available in this forum and elsewhere on line if people are willing to look for it or ask questions.

If a new language is to be used then there will initially be no experts or experience in its use and no reference material to go to.  When users want advanced features they will either need to switch to another language, probably C++, or the new language will need to have them built in, adding more complication again.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 04:54 pm
Just thinking of how some potential tasks can be solved with the proposed approach.
For example, rotating a stepper motor based on potentiometer reading (basic)

device Stepper {

    write: pinMotor // pin numb er

    // initialize
    init {
        degrees = 0
        self -> rotate( degrees )
        wait 500ms
    }

    // all object 'methods'
    actions {
        rotate ( int deg ) {
            pinMotor = deg
        }
    }

}

device Potentiometer ( Stepper st ) {

    read: pinReading

    // repeat actions within this block continiously
    always {
        st -> rotate( pinReading )
    }

}

stepper1 = create Stepper on pin 9
potentiometer1 = create Potentiometer on pin 1



[EDIT] @UKHeliBob, sorry, did not notice your reply at first.
We surely don't need yet another language.  Whilst it may at first appear that using C++ as the language for Arduino is a hurdle I see it as a positive advantage.  There is tons of help already available in this forum and elsewhere on line if people are willing to look for it or ask questions.
I do not disagree with you, Arduino does have a great (and huge!) community, yet I think it can be useful to think of other ways of solving project-related programming tasks

Quote
If a new language is to be used then there will initially be no experts or experience in its use and no reference material to go to.  When users want advanced features they will either need to switch to another language, probably C++, or the new language will need to have them built in, adding more complication again.
Well, there can be basic documentation and examples :) I think the question of "advanced language features" is not relevant for many Arduino users (which is fine) I think the community might be loosing a lot of people because the technology could be more beginner-friendly. Surely if people want more advanced features - they can use more appropriate tools for that.
Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 05:02 pm
Code: [Select]

device Stepper {

    write: pinMotor // pin numb er

    // initialize
    init {
        degrees = 0
        self -> rotate( degrees )
        wait 500ms
    }

    // all object 'methods'
    actions {
        rotate ( int deg ) {
            pinMotor = deg
        }
    }

}

device Potentiometer ( Stepper st ) {

    read: pinReading

    // repeat actions within this block continiously
    always {
        st -> rotate( pinReading )
    }

}

stepper1 = create Stepper on pin 9
potentiometer1 = create Potentiometer on pin 1


I don't see how this is any easier to understand or follow than the equivalent C++ code.  All you've really saved anyone from is a missing semicolon error, and I think that should be the least of the worries.  Most people can get the semicolon there.  That's not so hard.  With this code there is just as much structure to figure out, it's just different. 
Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 05:04 pm
Code: [Select]
potentiometer1 = create Potentiometer on pin 1

Take this line for instance.  How is this any easier than creating a potentiometer class in C++ and putting it in a library for the noobs so they can write.

Code: [Select]
Potentiometer myPot(1);  // creates a pot on pin 1

and later

Code: [Select]
int val = myPot.read();

You've just added 3 new keywords to learn and get right, create, on and pin.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 05:26 pm
potentiometer1 = create Potentiometer on pin 1

Take this line for instance.  How is this any easier than creating a potentiometer class in C++ and putting it in a library for the noobs so they can write.

Potentiometer myPot(1);  // creates a pot on pin 1

and later

int val = myPot.read();
The point of this is not just a different syntax, but different approach to the whole program structure.
Everything follows a strict convention and a lot of the stuff comes 'out of the box'
Timing is handled for you, so you don't need to worry about delay() halting your program flow.

You would loose some flexibility, but I think many things can be taken out if the language is aimed at beginners (pointer arithmetic for example)

Here is how I see it working:

device is the main entity in the language (think Class)

Every device follows this structure:

device DeviceName ( constructor parameters ) {

    write: pin1, pin2, pin3 // pins that the device can write to
    read: pin1, pin2, pin3 // pins that the device can read from

    // Here go predefined blocks that are built into the class
    init { ... } // basically constructor
    actions { ... } // all object 'methods'
    always { ... } // this code is executed all the time
    ....

    // then goes a list of all 'events' - code that executes when a condition is satisfied
    // they all start with keyword 'when'
    when pin1 is HIGH { ... }
    when pin2 is < 512 and pin3 > 128 { ... }
    when pin4 is RISING { ... }
    ...

}

So there isn't much code apart from within the devices. (There can be other entities though, like timers for example, with their own set of "actions" and "events")

There would be a small runtime that would take care of timing and executing events.
PS I do appreciate any criticism though, as it is entirely possible that I am missing some huge flaws in this.
Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 05:37 pm
PS I do appreciate any criticism though, as it is entirely possible that I am missing some huge flaws in this.

Absolutely.  It's nothing personal.  Every idea needs skeptics.  That's how we weed out the shite.  We don't always see the flaws in ideas we're too close to. 

I'm with you that it might be fun to create a new set of syntax and for some projects it might be useful.  I don't see how procedural code would work here once the plan got much bigger than read the pot, scale it, and write PWM to the led.  I'm afraid this would run out of usefulness once the code got any bit of complicated and then the noob would have to go back to square 1 learning C++. 

Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 05:57 pm
I don't see how procedural code would work here once the plan got much bigger than read the pot, scale it, and write PWM to the led.  I'm afraid this would run out of usefulness once the code got any bit of complicated and then the noob would have to go back to square 1 learning C++.
That's a fair point, that's why I am trying to come up with various use cases where it can be applied.
Overall I am planning to get away with 3 main entities for now: device, timer and state machine. I just haven't settled on all the concepts yet. If someone is planning on writing a web server or something like that - this would most likely not be the right tool for the job. I think it would be more well-suited for home automation projects and  things like that.

Also, its worth noting that a lot of those projects never get "much bigger than read the pot, scale it, and write PWM to the led"
Title: Re: New programming language for beginners: what could it look like?
Post by: Robin2 on Nov 08, 2015, 06:00 pm
@YemSalat, what you are proposing will not really benefit from discussion.

If you can start a Thread in the Other Software Development section with a working version of your "transpiler" so that people can try it out you may get converts.

I would like a Python-like syntax simply so that I don't get confused switching between Python and Arduino code.

I was almost tempted to say I would like a Ruby-like syntax, except that I abandoned Ruby in favour of Python simply because Python has a better infrastructure and is more widely used (at least in the Arduino context).

...R
Title: Re: New programming language for beginners: what could it look like?
Post by: MarkT on Nov 08, 2015, 06:07 pm
Have you looked at https://micropython.org/ (https://micropython.org/) ?  I keep meaning
to and see if its a good fit for this kind of use (presumably its meant to be!)
Title: Re: New programming language for beginners: what could it look like?
Post by: hbl2015 on Nov 08, 2015, 06:09 pm
I am a newbie to the Arduino IDE and am completely baffled by it's complexity. I think that C++ is a great language for experienced programmers, but lousy for beginners. Its syntax is cryptic, and mistakes are easily made, however, even with mistakes, the compiler still may be able to make sense out of it, resulting in incomprehensible results.
I think you should completely abandon C related languages as a choice, and make it more BASIC related. BASIC may make it slower, but can be improved by compiling it, rather than interpreted. It also has the advantage of being restricted in how many Statements can be used, not like C where there are unlimited functions available through its Libraries, but which are not that easily found by newbies.
BASIC has also the inherent advantage in that it is very user friendly and easily learned by non-programmers.
Title: Re: New programming language for beginners: what could it look like?
Post by: Vaclav on Nov 08, 2015, 06:11 pm
What do you think? Do you reckon any of these ideas are worth considering?

What in your opinion should a beginner-oriented Arduino language look like? Is such a language even needed?

OK, you asked...
Here are preamble statements :
Arduino is a brainchild of schoolteachers, albeit nobody will admit to that.
They put together nice piece of hardware, but stopped short to provide anything but very basic software.
( Most "libraries" are by folks outside Arduino )

That is on positive side of things.

On negative side - they bastardized perfectly workable language by hiding the foundation in the name of " not confusing the beginners". ( see the latest on 1.6.6. release)
The language foundation - C  - is nicely documented in a skinny book of around 200 plus pages.
It is all in there and anybody who WANTS to learn it can read it over the weekend.
The C++ is just an addition , but EVERYBODY needs the foundation first.

Any time you scan this forum you can see that some folks just do not want to learn the foundation.
They just want to run robots!

You will not move Arduino from kindergartner level to first grade by having another language.

And I am not trying to discourage you, just telling the way I feel about things.

You do whatever floats your boat, but without Arduino.cc first removing the "do not confuse beginners " attitude you will be on your own.

Personally , if you could became member of Arduino inner circle of developers  you could contribute more to everybody's benefit.

Cheers Vaclav


Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 06:13 pm
What I think might be more helpful would be a more noob friendly editor for the IDE.  Something that spits out error messages that sound more like advice.


Code: [Select]
error: void value not ignored as it aught to be

could either be, or have in addition

Code: [Select]
this function does not return a value, so you can't assign it to this variable


Something that spits out an extra level of warnings like:

Code: [Select]
WARNING:  non-volatile variable never changes in body of while loop.  Loop will be infinite.

or even the ability to let people step through code so they can see their stuff getting hung up in a while loop or why an if statement seems to be "skipped".  

I know in the java world there are code checkers that can even tell you for instance that a method is too long or does too many things and should be broken into individual functions.  Or that you've done something else stupid.  A code checker could be very useful for the noobs.
Title: Re: New programming language for beginners: what could it look like?
Post by: Delta_G on Nov 08, 2015, 06:13 pm
Basically, what I think would be useful would be to go through @PaulS's last 5000 posts and take the top 20 or so things he says and make compiler warnings out of them. 
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 08, 2015, 06:35 pm
@MarkT will have a look, cheers

@hbl2015, I am not sure if basic is a good base, but I agree that having to look for various libraries can be confusing and it can be nice to have some 'essentials' built into the language

@Vaclav, thanks for sharing your opinion on this, but to be honest I have a pretty low chance of getting into the Arduino hq (not that I would want to anyway)

@Delta_G, I think it may actually be harder to rewrite all C++' errors and warnings then to make a separate language oO

@Robin2, I will probably have a go at writing a very basic transpiler in the next few days, which would allow you to just copy the code to IDE. I won't implement the main framework yet cause I have not completely worked it through, but I will try to add a bit of syntactic sugar at first (implicit types, nicer for/if statements, etc.)

Title: Re: New programming language for beginners: what could it look like?
Post by: UKHeliBob on Nov 08, 2015, 07:28 pm
I think that however the Arduino is programmed there is a problem with the mixed background of its users.  I came to it with no experience of C let alone C++ but I had used other languages in the past, mostly various flavours of BASIC including VBA, Z80 assembler and monstrosities such as Mumps.  As a result I needed to learn the C/C++ syntax for constructs that I expected to be there but logic is logic whatever language is used.

It is difficult to be certain, but were I in the position of a complete novice then from what I have seen of the proposed new language I don't think that they will be any better off.  One problem will always be that, by definition, computer languages are created by experienced computer programmers.  What sounds simple and logical to them is gobbledegook to outsiders.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 09, 2015, 01:28 am
I think that however the Arduino is programmed there is a problem with the mixed background of its users.  I came to it with no experience of C let alone C++ but I had used other languages in the past, mostly various flavours of BASIC including VBA, Z80 assembler and monstrosities such as Mumps.  As a result I needed to learn the C/C++ syntax for constructs that I expected to be there but logic is logic whatever language is used.
I think your situation is slightly different because you already were familiar with programming logic and concepts before you started with C/C++. For many people here - C++ is the first language that they learn, and I think it is a bad choice for the first language (even a bit of prior experience with Basic would help a lot in my opinion)

Quote
It is difficult to be certain, but were I in the position of a complete novice then from what I have seen of the proposed new language I don't think that they will be any better off.  One problem will always be that, by definition, computer languages are created by experienced computer programmers.  What sounds simple and logical to them is gobbledegook to outsiders.
This is also a fair point, hopefully I can make it more useful and easy to understand, that is why I really appreciate any feedback.
Title: Re: New programming language for beginners: what could it look like?
Post by: Robin2 on Nov 09, 2015, 10:30 am
I will probably have a go at writing a very basic transpiler in the next few days,
I forgot to say that you also need to write the book that explains how to use your language.

Actually I think you should write the book first and then make a language that complies with the book. As you mentioned Ruby earlier you may be familiar with the "pickaxe" book - Programming Ruby by Dave Thomas. That is the sort of thing I have in mind - but maybe yours would not need 800 pages for Version 0.1

IMHO without proper documentation there is no point writing a language aimed at making things easier. And by proper documentation I mean explanations of why you should do things and what is the purpose of each thing, not just a few examples that each show a single "how" to use a feature with no attempt to show alternatives or exceptions.

...R
Title: Re: New programming language for beginners: what could it look like?
Post by: UKHeliBob on Nov 09, 2015, 11:23 am
Quote
I really appreciate any feedback.
Let me know when you have something to test and I will assume my alternative identity of "British Standard Idiot" and put it to the test.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 09, 2015, 03:18 pm
@Robin2 We'll see how it goes, I think version 0.1 will fit on a couple pages :)

@UKHeliBob hehe, that kind of attitude would certainly be helpful! I will post a separate topic in the "Projects" section once I have something to show.


A small update. So far I have settled on the tools, I will be using Jison to generate the parser - http://jison.org (http://jison.org)
Jison is a JavaScript port of Bison compiler-compiler, I have been working mostly with nodejs in the past couple years, so I feel most comfortable writing the parser in JS. It also completely solves the cross-platform compatibility issue, as it will work across all operating systems out of the box.

The parser will build a syntax tree, then I will convert it to C/C++ code which can be just copy-pasted into the Arduino IDE.

For the first "milestone" I am planning to accomplish the following:

- optional (inferred) types for variable declarations
- some default types - int, float, long (Strings might have to wait until the next one)
- if / else statements
- for loops
- functions
- some basic built-in functions for pin manipulation

So, this would be just a 'proof of concept' version, nothing too serious. I will upload it to github once I have something to upload :)

No timeframe for now, but hopefully I will present something by the end of the week.
Title: Re: New programming language for beginners: what could it look like?
Post by: Robin2 on Nov 09, 2015, 03:52 pm
@Robin2 We'll see how it goes, I think version 0.1 will fit on a couple pages :)
Within reason, the more pages in the manual the better. Most open source software lacks adequate documentation.

...R
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 12, 2015, 05:20 pm
@Robin2, if the project goes well - I will definitely make sure to write proper documentation and prepare some starter resources.


A quick update.

I switched from Jison to pegjs for the compiler (http://pegjs.org (http://pegjs.org)) as it supports PEG notation which is much simpler to write. It also has some really useful examples: https://github.com/PhilippeSigaud/Pegged/wiki/Grammar-Examples (https://github.com/PhilippeSigaud/Pegged/wiki/Grammar-Examples) Even a complete grammar for C!

So far I added the following features to the language parser:

- declaring variables (6 types so far: byte, float, int, long str, bool)
- 'optional' types
- optional semicolons
- assignment / comparison / math operations
- if and switch statements
- for loops
- literals for time (like: 1m 15s - converts to an integer containing the number of milliseconds)
- wait and every constructs for basic timing (like: every 5s { /* ... */ })

I am yet to build the compiler itself (I am also planning a separate evaluator), but the parser seems to work pretty well so far (couple small issues, but I know how to fix them).

Here is a simple example, this expression:
Code: [Select]
a = 5
b = (a + 2) * 3


Will generate the following syntax tree (in JSON format):
Code: [Select]

{
   "$$": "COMPILE",
   "body": [
      {
         "$$": "EXPRESSION",
         "expression": {
            "$$": "ASSIGNMENT",
            "operator": "=",
            "left": {
               "$$": "IDENTIFIER",
               "name": "a"
            },
            "right": {
               "$$": "LITERAL",
               "value": 5,
               "type": "int"
            }
         }
      },
      {
         "$$": "EXPRESSION",
         "expression": {
            "$$": "ASSIGNMENT",
            "operator": "=",
            "left": {
               "$$": "IDENTIFIER",
               "name": "b"
            },
            "right": {
               "$$": "MATH_EXPRESSION",
               "operator": "*",
               "left": {
                  "$$": "MATH_EXPRESSION",
                  "operator": "+",
                  "left": {
                     "$$": "IDENTIFIER",
                     "name": "a"
                  },
                  "right": {
                     "$$": "LITERAL",
                     "value": 2,
                     "type": "int"
                  }
               },
               "right": {
                  "$$": "LITERAL",
                  "value": 3,
                  "type": "int"
               }
            }
         }
      }
   ]
}



The next thing that I am quite excited about - is that I started creating and IDE for the language, and I just keep realizing how useful it can be for beginners.
I am using Ace (https://ace.c9.io/ (https://ace.c9.io/)) for the text editor and it already has a ton of useful features built in (like autocomplete!), but what is even more important - I can evaluate all the code in 'live' mode and detect errors right away, without having to 'compile'. Ace has some nice APIs for code highlighting, etc.

Granted that the parser is very simple so far, but I did a few tests and I am able to run it in a web worker (it allows running multi-process code in JavaScript as JS is single-threaded by nature) - and it parses 1000 lines of code instantly as I type (I will share a demo soon)

I am not going to spend too much time on the IDE right now, just gonna build some basic stuff around the text editor so its easy to test the language when the first version is out.

Again, no time frame so far, but aiming to release something by the end of the weekend or early next week.

I will also definitely need help with optimizing the resulting C/C++ code, hopefully there is enough people around here who can help with that :)
Title: Re: New programming language for beginners: what could it look like?
Post by: Robin2 on Nov 12, 2015, 10:15 pm
If those two lines require a tree like that the world's forests will soon be devastated.   :)

...R
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 13, 2015, 01:13 am
If those two lines require a tree like that the world's forests will soon be devastated.   :)

...R
Hehe :) thats just because the language is low level and not dynamic, so all the instructions are quite explicit and almost always can convert straight to C/C++ without much modification. Anything more 'sophisticated' would require a more complex runtime which we just can't afford on the Arduino.
(Also JSON format makes it look bigger then it is)


Just for the heck of it, I generated parse trees for the same expression in Python and C.

Original expression:
Code: [Select]
a = 5
b = (a + 2) * 3



Python AST (python has a built-in AST generator)
Code: [Select]
Module(
    None,
    Stmt(
      Assign(
        AssName('a', 'OP_ASSIGN'),
        Const(5)
      ),
      Assign(
        AssName('b', 'OP_ASSIGN'),
        Mul(
          Add(
            Name('a'),
            Const(2)
          ),
          Const(3)
        )
      )
    )
  )



C (generated with clang)
Code: [Select]
TranslationUnitDecl 0x10302d6c0 <<invalid sloc>> <invalid sloc>
|-TypedefDecl 0x10302dc00 <<invalid sloc>> <invalid sloc> implicit __int128_t '__int128'
|-TypedefDecl 0x10302dc60 <<invalid sloc>> <invalid sloc> implicit __uint128_t 'unsigned __int128'
|-TypedefDecl 0x10302e020 <<invalid sloc>> <invalid sloc> implicit __builtin_va_list '__va_list_tag [1]'
`-FunctionDecl 0x10302e0c0 <c1.cpp:1:1, line:6:1> line:1:5 main 'int (void)'
  `-CompoundStmt 0x10302e3d0 <line:2:1, line:6:1>
    |-DeclStmt 0x10302e248 <line:3:3, col:12>
    | `-VarDecl 0x10302e1d0 <col:3, col:11> col:7 used a 'int' cinit
    |   `-IntegerLiteral 0x10302e228 <col:11> 'int' 5
    `-DeclStmt 0x10302e3b8 <line:4:3, col:22>
      `-VarDecl 0x10302e270 <col:3, col:21> col:7 b 'int' cinit
        `-BinaryOperator 0x10302e390 <col:11, col:21> 'int' '*'
          |-ParenExpr 0x10302e350 <col:11, col:17> 'int'
          | `-BinaryOperator 0x10302e328 <col:12, col:16> 'int' '+'
          |   |-ImplicitCastExpr 0x10302e310 <col:12> 'int' <LValueToRValue>
          |   | `-DeclRefExpr 0x10302e2c8 <col:12> 'int' lvalue Var 0x10302e1d0 'a' 'int'
          |   `-IntegerLiteral 0x10302e2f0 <col:16> 'int' 2
          `-IntegerLiteral 0x10302e370 <col:21> 'int' 3


As you can see, C generates a more 'verbose' AST, but it can be converted almost directly into assembly. While Python's AST is more concise - it requires a massive runtime to support all of the entities.
Title: Re: New programming language for beginners: what could it look like?
Post by: YemSalat on Nov 15, 2015, 12:29 pm
A small sneak peek of what the error highlighting looks like in the IDE I am making:
(http://i.imgur.com/PX009o2.png)

Yet another update:

So far I am happy with how the project is progressing. I think it will take a little bit more time until the first release, as now I am not only building the compiler but also the IDE which comes with it.

Compiler progress:
I have built an evaluator (code analyzer), which can check whether the types are correct, if all variables have been declared, etc. It can also put additional info into the parse tree (e.g. types for inferred declarations)
It still needs a few tweaks, but overall its initial version is complete.


Right now the compiler consists of three main parts:

- The Parser - Checks the code syntax and generates the first parse tree

- The Evaluator - Analyzes code semantics and generates the second parse tree

- Code Generator - (I am yet to write this one) - it will produce C++ code for Arduino; As I said before I may need help from the community in the future in order to optimize it properly.


IDE progress:
The IDE is going well, especially since I added the analyzer - it can now point out errors like undefined variables, incorrect types, etc. Its really nice to have a compiler 'built-in' because we can check not only the syntax, but also the semantics of the code, and all that can be done 'live'

Besides that it allows for some other great functionality in the future, for example, we could develop an Arduino emulator which can be used for testing code before uploading to the actual device.

I also checked out some other similar projects like https://codebender.cc/ (https://codebender.cc/), but I think my approach has some benefits over what they are doing.

Other stuff:
Putting aside the compiler implementation details, here is a somewhat interesting 'quirk' that I stumbled upon while developing the languge.

As I mentioned before I wanted to add 'optional' types to the language, meaning that the type of the variable can be inferred from its value during assignment. E.g. ( b = 4 ) - here b is obviously an int
I also made semicolons optional as I think it can make the code look more 'approachable' to beginners

But take a look at this situation:
Code: [Select]
a =     // accidentally forgot to assign value
        // (real life example, I did this while testing the IDE)
b = 3


in a system with no declaration types and no semicolons this converts into:
Code: [Select]
int a = b = 3
Which can lead to some weird errors, and the compiler also would not be able to catch such an error.

I decided to fix this by not allowing multi-line assignments, so both left and right parts of the assignment must be on the same line. I think that this is a good enough fix, but this made me think whether optional types will bring more benefits then problems. I will leave them in for now, but this is something that we can discuss in the future.


Ideas:
I also have a couple ideas that I want to add to the language.

For example, 'short-hand functions' (or 'dynamic variables') which could be declared with -> operator, and are really just a shorter way to express a function with no arguments

So you can write something like this:
Code: [Select]
a = 1
b = 2
c -> a + b
serial_println( "Sum is " + c ) // Sum is 3

a = 3
serial_println( "Sum is " + c ) // Sum is 5



Also sub-string expressions, which are kind of like string 'templates' and act as an alias for sprinf() :
Code: [Select]
foo = "Hello"
serial_println( "{foo} World!" ) // Hello World!



Anyways, its important to focus on getting the basics right for now :) I will post a new update soon.

PS I updated the topic title since it has become more of a 'developer notes' kind of post.
Title: Re: Designing a new programming language for Arduino
Post by: GrooveFlotilla on Nov 15, 2015, 06:11 pm
Why not use the Pascal trick (inherited from ALGOL) of distinguishing an assignment ( x := y ) from a comparison ( if ( x = y) )?
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 15, 2015, 08:05 pm
Why not use the Pascal trick (inherited from ALGOL) of distinguishing an assignment ( x := y ) from a comparison ( if ( x = y) )?
or the C trick of distinguishing an assignment ( x = y ) from a comparison ( if ( x == y) )?

I'm sorry, but if a beginner cannot use = and == properly then they are unlikely to use := and = properly.
Title: Re: Designing a new programming language for Arduino
Post by: GrooveFlotilla on Nov 15, 2015, 08:50 pm
The thing is, at school you're taught that = is part of an equation ( left side of = sign is identical to right hand side), so in C, "x = x + 1;" is clearly arithmetically nonsensical.
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 15, 2015, 09:10 pm
Quote
in C, "x = x + 1;" is clearly arithmetically nonsensical.
True, but I am not convinced that beginners worry about such niceties.  Personally I have never had any problem understanding the x = x + 1 construction and have never given a single thought to the fact that both sides of the = are normally identical.  My brain simply says that as a result of the operation x will now equal its original value plus 1.

Quote
at school you're taught that = is part of an equation ( left side of = sign is identical to right hand side)
Quote
( if ( x = y) )
Here both sides of the = may or may not be identical.  Where does that leave the notion that both sides must be identical ?
Title: Re: Designing a new programming language for Arduino
Post by: GrooveFlotilla on Nov 15, 2015, 09:23 pm
Quote
Here both sides of the = may or may not be identical. 
You've just asked the question "Here both sides of the = may or may not be identical.  the left side is equal to the right".

Quote
Personally I have never had any problem understanding the x = x + 1 construction
It was one of the hardest  things I had to overcome, converting from ALGOL and Pascal.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 15, 2015, 09:46 pm
Quote
in a system with no declaration types and no semicolons this converts into:
Why wouldn't you treat the end-of-line as the end of a statement if there is no semi-colon?

It is possible to have continuation character for the odd occasions when code needs to run on to a second line.

Indeed, if semi-colons don't matter why not just run over the code and delete them and always use the EOL as the delimiter.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 16, 2015, 01:39 am
Why not use the Pascal trick (inherited from ALGOL) of distinguishing an assignment ( x := y ) from a comparison ( if ( x = y) )?
I have been thinking about this myself, I am not entirely convinced this is a good solution, I think it makes the language a little bit more 'cryptic'. I've been playing with Go language a couple month back, it is similar to Pascal in this respect, and guess which question comes up ever so often in Go community ( http://stackoverflow.com/questions/16521472/assignment-operator-in-go-language (http://stackoverflow.com/questions/16521472/assignment-operator-in-go-language) )

or the C trick of distinguishing an assignment ( x = y ) from a comparison ( if ( x == y) )?
This is what I am doing now. Overall I would prefer to keep the syntax more similar to C (since it is so popular among embedded systems programmers)


I remember I was using some program back in the day to generate simple animations, I can't remember what was the name of the program, but it had a simple scripting language built into it, and in that language the '=' sign was treated differently depending on the context.

So for example, this is assignment: a = 23
But this is equality check: if ( a = 17 ) - since you rarely want to actually assign something inside an if statement.

This makes sense when you translate the code into English and replace '=' with 'is', so you get:
a is 23
if a is 17


I remember that I never had problems with this concept, perhaps could do something similar? I am not a big fan of context-dependent behaviour though.


Why wouldn't you treat the end-of-line as the end of a statement if there is no semi-colon?

It is possible to have continuation character for the odd occasions when code needs to run on to a second line.

Indeed, if semi-colons don't matter why not just run over the code and delete them and always use the EOL as the delimiter.

...R
That's pretty much what is happening now, Assignment expressions were just one of the edge cases where this can actually break things. All other expressions seem to be 'multi-line safe'.
E.g. you can also define multiple variables like so:
Code: [Select]
int a = 1,
    b = 2,
    c = 3
Title: Re: Designing a new programming language for Arduino
Post by: CrossRoads on Nov 16, 2015, 01:56 am
Quote
Why not use the Pascal trick (inherited from ALGOL) of distinguishing an assignment ( x := y ) from a comparison ( if ( x = y) )?
Two symbols are two symbols, I have no problem with ==.
At the same time, context within an if( ) makes sense too, I've never written anything where I assigned a value within an if ( ) statement, that is just too convoluted for me. Write it as two lines, let the compiler deal with it.
Title: Re: Designing a new programming language for Arduino
Post by: CrossRoads on Nov 16, 2015, 01:57 am
Especially where : needs a Shift press too.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 16, 2015, 09:52 am
Especially where : needs a Shift press too.
Agree 100%. That's also why I hate underscore characters in code.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 16, 2015, 10:56 am
Ok, so far we have the following options:

1)
Code: [Select]
=          // assignment
==         // check

 

2)
Code: [Select]
:=         // or..
int x = .. // assignment

==         // check


3)
Code: [Select]
=          // both assignment and equality check

I personally prefer #1, as it is the most C-like. (And indeed does not require pressing additional keys)
I am not a huge fan of #3 because it would have to be excluded from expressions where its hard to tell whether its an assignment or a logical operation:
Code: [Select]
bool isPressed = (reading = 3) // or
bool isPressed = isOn = 1 // etc.
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Nov 16, 2015, 02:17 pm
Two symbols are two symbols, I have no problem with ==.
At the same time, context within an if( ) makes sense too, I've never written anything where I assigned a value within an if ( ) statement, that is just too convoluted for me. Write it as two lines, let the compiler deal with it.
Agreed, it is far easier to recognize what is happening with two lines, however both switch and if statements have an exception; initialization.


if( int valueNotZero = random(0,10)){
  //use valueNotZero if value not zero
}


switch( int resultOfComplexAlgo = ...){
  case whatever:
    //use resultOfComplexAlgo in case easily
  break;
}


Of course this is still subject to personal preferences, but a very tidy use none the less (keeps locals scoped to where they belong).
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 16, 2015, 03:01 pm
I would favour option 1, ie =/== as in C

The new programming language is unlikely to expand to become as feature rich as C++ unless something extraordinary happens, so making it as C like as possible to allow a smooth transition at a later date would seem to make sense.

What might make even more sense would be to make it a subset of C++ but with better possible error trapping at the time of input and more informative error messages.  For instance, assignment in an if statement is possible as has been shown but is relatively unlikely.  Maybe the preprocessor could query the use of a single = in an if statement and the use of a == in what is probably an assignment statement with an option to turn off the querying of such uses.
Title: Re: Designing a new programming language for Arduino
Post by: westfw on Nov 17, 2015, 06:40 am
I know!   For comparison, we should use something less symbolic.   Maybe "if a .eq. b then xxx" ?
Maybe for assignment, we can add an additional keyword to make that more obvious, say "let a = b"

 :-)  (some people should recognize this as a joke.)

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 17, 2015, 11:55 am
The thing that I want to see made more intuitive is * and & as used for pointers. I understand pointers but I just can't get my brain to interpret those symbols intuitively in the same way that it does with +, -  and = (or even ==).

...T
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 17, 2015, 12:33 pm
I'm with you on that Robin.
I can read and understand x = x + 1 etc until the cows come home but I cannot do the same for & and * as used for pointers.  I have to stop and think when I read code containing them.  I am getting better but I am not there yet.

However, as I have said before, we need to understand what this new language is for.  Is it going to be a "be all and end all" language or a stepping stone to C++ ?  If the latter then let's keep it as C like as possible but make the environment more helpful.
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Nov 17, 2015, 01:22 pm
Quote
The thing that I want to see made more intuitive is * and & as used for pointers. I understand pointers but I just can't get my brain to interpret those symbols intuitively
This new fan-dangled language might not even need to use pointers, for instance, c# has value types and reference types, these do not need explicit '&', its more a feature of whether something is a struct/intrinsic (value type) or class (reference type), or JavaScript where virtually everything is a reference and you just take it for granted.

The major factor with these languages is garbage collection (reference counting) which allows a lot of background work to be done with little user interaction. C/C++ was designed for places where these features may not be suitable (C++11 can implement GC however).


What is it you struggle with? Is it because the same symbol has different meanings in different contexts (& can = b/w and, declare a reference, get pointer of var), (* can = declare a pointer, dereference a pointer, multiplication).

Why not make yourself some defines that make these things easier (use words rather than symbols).

Quote
I have to stop and think when I read code containing them.
The one thing I can recommend is use them more, its the simplest way of pushing through a barrier (obvious I know, but its the only way I was able to understand operators like '->*' and '.*' explanations just did not cut it until put into practice).
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 17, 2015, 06:19 pm
Why not make yourself some defines that make these things easier (use words rather than symbols).
Unfortunately they would not make it easier to read another person's code

I guess I could write a parser that goes through the other person's code and replaces * and & with my #defines  :)   :)    (for avoidance of doubt, I am NOT going to try)

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 17, 2015, 07:03 pm
However, as I have said before, we need to understand what this new language is for.  Is it going to be a "be all and end all" language or a stepping stone to C++ ?  If the latter then let's keep it as C like as possible but make the environment more helpful.
Right now I think that the best approach is to make a semi-subset of C++ with lighter syntax and a couple built-in language constructs for some core Arduino development things (like basic time handling, interrupts, etc.)
So the language would have all the things like primitives/loops/functions/classes
It would also be good to allow for easy compatibility with current Arduino libraries and as you said - easier transition to C/C++, so keeping the syntax as C-like as possible is the way to go in my opinion.


This new fan-dangled language might not even need to use pointers, for instance, c# has value types and reference types, these do not need explicit '&', its more a feature of whether something is a struct/intrinsic (value type) or class (reference type), or JavaScript where virtually everything is a reference and you just take it for granted.
Well we definitely can't afford a proper GC on the Arduino, but I do think that some common tasks that use pointers can be abstracted away and use them behind the scenes to optimize things, while providing nicer syntax.

Why not make yourself some defines that make these things easier (use words rather than symbols).
I was actually thinking of adding a couple built-in operators, e.g. ref() - to create a pointer and valueOf() to get its value, etc.



A small update

Unfortunately I did not have much time to work on the language yesterday, I did a few tests and worked on the evaluator. Now it generates a kind of a "byte-code" (if you can call it that) that is later going to be consumed by the code generator. I still have not yet started on the generator itself, but I am hoping to start tomorrow and finish in 1-2 days depending on how busy I will be.

The "byte-code" looks like this: ..[INIT INT $1 $2][FOR_BEGIN][L_PAREN]...
, (pretty 'literal')

I also worked on the IDE a little bit and implemented a simple file handling system.
(http://i.imgur.com/2Kxp0mf.png)

Everything is stored locally on your computer (browser Localstorage allows for up to 5MB of space by default, that should be enough for most use cases, but the app can request more if required), so there is no downloading-uploading going on, you can however "download" any file as a text document or simply copy-paste it. Also, once the IDE is loaded - it becomes completely available offline (via appcache) so it will work even if you reload the page or reboot your computer.


As I already said I am hoping to have more spare time in the next few days to work on the compiler.
The C++ code generated by the first version will most likely be quite ugly, so please do not expect any great optimizations or elegant solutions in the first release.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 17, 2015, 11:27 pm
Keep going.

It is nice to see a man hard at work.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 20, 2015, 08:52 am
Another progress report


I worked on the compiler for the past few days and I gotta say it is starting to get a bit more complicated; in terms of compiler now having more code in order to account for language features.
Most of the complexities are introduced by interactions between types.

Initially was only checking types for literals and variables and wanted to throw part of the type checking work on the C compiler, but now I think that it would be much more useful to do a complete type check so we produce a completely type safe C code. So I had to introduce yet another stage in the compilation process, which is the Type Checker, it goes through the parse tree (post-order) and assigns types to every single expression and checks them all.

I also added special built-in type conversion functions: int(), float(), str(), etc.
For example to convert a string to an integer, you would do the following:
int a = int( "42" ) // 42

As I mentioned before, I would like to overload the '+' operator for strings, so you can concatenate two strings like so: "Hello" + " World"

But I am not sure whether its a good idea to overload '+' for concatenating strings with other types,
i.e. "hello" + 42

It would be good to have some input from the community on this.



So far I came up with the following rules for the overloaded '+' operator:
Strings:
String + (Int | Float | Bool) = String

Numbers:
Int + Float = Float, etc.. // same as C

Others:
Int + Bool, etc.. = ILLEGAL


I have not worked on the IDE since last update and am hoping to finish the compiler by tomorrow, most likely without concatenation of different types in the first version, but we'll see how it goes and what you have to say about this as well.
After the compiler is done - I'll just need a little bit of time to 'link' it with the IDE and it should be good for the first release.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 20, 2015, 10:53 am
As I mentioned before, I would like to overload the '+' operator for strings, so you can concatenate two strings like so: "Hello" + " World"
I presume you are talking about strings (small s) and not Strings (capital S).

Quote
But I am not sure whether its a good idea to overload '+' for concatenating strings with other types,
i.e. "hello" + 42
In a system that uses duck-typing this might make sense. In a system that requires typed variables I think it would just be confusing. However even Python requires "hello" + str(42) and Ruby (IIRC) requires "hello" + 42.str

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 20, 2015, 11:34 am
I presume you are talking about strings (small s) and not Strings (capital S).
Yep, sorry, just capitalized the first letters in the example, in the language all those entities are primitives.
In a system that uses duck-typing this might make sense. In a system that requires typed variables I think it would just be confusing. However even Python requires "hello" + str(42) and Ruby (IIRC) requires "hello" + 42.str
Yep, good point, perhaps its best to not include concatenation of different types.

Plus, I wanted to introduce sub-string expressions anyway.
So one can write: foo = "This is how a {{ variable }} can be represented inside a string"

Could also add 'filtering' functionality: {{ number | %d }}
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 21, 2015, 05:48 pm
Have spent some more time on the project and the evaluator is pretty much complete at this point :)
It can now do a full semantic analysis of all the language entities (respecting the scope), and point out various errors.

A couple screens of some tests I've done today (this is not the IDE UI):

[ View Full Size (http://i.imgur.com/XZuYkz4.png) ]
(http://i.imgur.com/XZuYkz4.png) (http://i.imgur.com/XZuYkz4.png)

*Underlined on the image above:
Semantic error on line 9 column 5
Error message: variable "foo" was already initialized as 42 on line 3


Some others (clickable):
(http://i.imgur.com/1nZerGu.png) (http://i.imgur.com/1nZerGu.png)   (http://i.imgur.com/3Nzgc6g.png) (http://i.imgur.com/3Nzgc6g.png)

I am about to add functions to the language, for now they will look the same as C functions (hope everybody is ok with that), but I do have plans on changing some things around in the future.

I am also thinking about adding an ability to the IDE for the users to generate and upload test suites. How I see it - one will be able to write some piece of code that produces some output, so you write the code and specify the output that it produces, then it gets added to the global database, so that new language versions can be tested against all the previous tests. Any thoughts on this?

I am proposing this as I am afraid I can miss some unobvious errors including the ones in language semantics.
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 22, 2015, 01:47 pm
A few more updates

IDE

I finished integrating the new evaluator into the IDE.
Had to spend a couple hours banging my head against the wall trying to figure out the timing for it (like delays after the code changed, etc.) and the fact that parser runs in a parallel process does not help at all.. But I figured it out in the end, so I am happy with the result.

Interesting fact:
I did some performance tests, evaluating a 2000 line file which contains all language entities.
Chrome and Firefox on my 3 year old core i5 laptop parse it within 250-300ms. I also tried IE11 on Windows 8, and surprisingly it got the best result of all - around 70-80ms.


The Language

I added functions.

Right now they are very much like C functions, but I also added docstrings like in Python, which allow you to annotate your functions with some documentation. The syntax for them - is just three slashes (/// text..), so they look like comments:

/// This function multiplies two integers
int multiply (int a, int b) {
 return a * b
}


This allows the IDE to show you tooltips containing the function description when you mouse over them in the code.
Docstring comments will also be kept when the code is transpiled to C++ (all other comments are ignored by the parser)


I am working on the code generator and to be honest it is a bit harder then I though, as I don't have too much experience with Arduino, I might need to create a topic or two in the Programming questions section soon :)

Anyways, we are getting really close to the release, I will be a bit busy with other stuff in the next couple days, not sure if I'll do any more updates before the demo, but we'll see how it goes.
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 22, 2015, 02:23 pm
Quote
I might need to create a topic or two in the Programming questions section soon :)
Questions from beginners are always welcome.  :)

Well done with progress so far.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 22, 2015, 02:40 pm
I might need to create a topic or two in the Programming questions section soon :)
I suggest you keep the discussion in this Thread at least until you publish a working version. I find it very difficult to follow a discussion over multiple Threads. (Can't walk and chew gum at the same time)

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 23, 2015, 04:48 am
OK, it probably IS best to keep everything in the same thread.


Just to get a couple small things out of the way:

1) Is there any particular reason for having the setup() {} routine in every sketch?

2) What is the preferred way of concatenating two strings? (meaning arrays of chars, do you think sprintf is a good fit for this?)


Thanks!
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 23, 2015, 08:19 am
Quote
Is there any particular reason for having the setup() {} routine in every sketch?
There will almost certainly be things that need to be done once in every program, such as setting up pinMode()s and starting the Serial interface, hence the setup() function.  It also suits the way in which the Arduino environment runs setup() once and then calls loop() repeatedly from the hidden main() function.

Quote
What is the preferred way of concatenating two strings? (meaning arrays of chars, do you think sprintf is a good fit for this?)
strcat() is the obvious solution whereas sprintf() is more powerful but uses more memory.  The standard Arduino implementation of sprintf() does not allow for the use of float variables although it can be added at the expense of more memory use.  snprintf() is safer to use than sprintf()
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 23, 2015, 10:04 am
1) Is there any particular reason for having the setup() {} routine in every sketch?
I think requiring it provides consistency that is very valuable for beginners.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 23, 2015, 02:34 pm
I think requiring it provides consistency that is very valuable for beginners.
...
strcat() is the obvious solution whereas sprintf() is more powerful but uses more memory.  The standard Arduino implementation of sprintf() does not allow for the use of float variables although it can be added at the expense of more memory use.  snprintf() is safer to use than sprintf()
Thanks for the explanations, I guess its best to stick with snprintf() for concatenation, at least for the first version. I am planning to revisit this later though, for example could do char array manipulations behind the scenes, perhaps some optimizations could even be done during the code evaluation phase.


Another small question, regarding object creation on Arduino. Just let me know if I get this right:

Object foo; - this statement means that the object will be created on the stack and cleaned up automatically (e.g inside a function)
Object* foo = new Foo(); - vs this, creating an object on the heap and having to delete it manually.

Is the above correct?
Title: Re: Designing a new programming language for Arduino
Post by: SurfingDude on Nov 23, 2015, 08:42 pm
It's good that each generation strives to re-invent the tools of a previous generation. I grew up on FORTRAN vs ASM vs COBOL, so that's my generation. I moved into ALGOL, SIMSCRIPT, PASCAL, and ADA, and then into C and C++. I've seen lots of language wars and fought on several sides.

The question for this thread is whether we are re-arranging the frosting on the cake or are we making a new kind of cake?

As to frosting, I think the conceptual bridge between the arduino world and the C/C++ world is something to keep. Many recipes exist to solve problems in in the semicolon-bounded world and I'd want to be able to copy and paste them into any new world, or take my arduino work and learning onto an industry platform.

As to the cake part, the thing that we lack most are effective ways to deal with concurrency. This is holding back the use of multi-core hardware and leads step-by-step programs into problems of semaphores and deadlocks, especially now when we try to use interrupts to multitask.

If there is to be a new language I'd like it to focus on improving the cake. Maybe with objects that communicate messages via queues, but do computation in the way that is already known and accepted. The standard frosting looks just fine.

Just my 2 cents.
Title: Re: Designing a new programming language for Arduino
Post by: CrossRoads on Nov 23, 2015, 09:35 pm
If you didn't have s setup() function, where would the stuff that only needs to occur once happen?

I didn't see the unsigned long type in there. Will that be coming to support the time elements?

Will bit manipulation be supported?
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 23, 2015, 11:40 pm
If there is to be a new language I'd like it to focus on improving the cake.
I like your analogies (or are they metaphors).

I think that @YemSalat is making new tastier frosting to make the Arduino more accessible to beginners. And IMHO that is A GOOD THING and should not be sidetracked by a wish to change from a victoria sponge to a fruit cake.

I don't think there is any intention to displace the usual C/C++ Arduino programming system - apart from the possibility that the new system (if it works) is so attractive that everyone evolves towards it.

And there is an opportunity for another enthusiast to create a different cake.

However I can't help feeling that you are expecting a lot from a 2K Arduino. Personally I like the fact that it carries no baggage even though that means I have to create my own. That way I know exactly what is in my cake (sorry for the mixed metaphors)

...R
Title: Re: Designing a new programming language for Arduino
Post by: CrossRoads on Nov 23, 2015, 11:50 pm
Well, it can be geared toward the 1284 then - 16K SRAM, dual serial port, 32 IO, all the stuff that will let beginners avoid having to add more hardware right off the bat.
Title: Re: Designing a new programming language for Arduino
Post by: SurfingDude on Nov 24, 2015, 12:37 am
However I can't help feeling that you are expecting a lot from a 2K Arduino.
Hello Robin2,

There is nothing wrong with developing new languages. I went to school with a fellow who designed a new language for each problem, then implemented it as a set of functions.

We are in an industry where the rate of change is frightening. I am not expecting much at all from the 8-bit Arduinos. Today belongs to the 32-bit architectures with lots of flash and RAM.

Tomorrow belongs to multi-core processors, which are already available. But aside from Parallax Propeller nobody has figured out how to make a go with them in the enthusiast space. This is an open area for language designers, and within a few years there will be multicore ARM solutions in the Arduino price range.


Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 24, 2015, 02:53 am
As to frosting, I think the conceptual bridge between the arduino world and the C/C++ world is something to keep. Many recipes exist to solve problems in in the semicolon-bounded world and I'd want to be able to copy and paste them into any new world, or take my arduino work and learning onto an industry platform.
At least syntax-wise the language is definitely going to resemble C/C++ as close as possible, exactly for the reasons you mentioned (compatibility with existing code and ability to transfer your projects 'back' to C if required)
I also want to keep the transpiled code close to the original (keep variable names, etc) and also format it in a nice and readable way.

What I am still 'on the fence' about though - are the C pointers, right now I have not implemented them in any way, and I am not a big fan of the original syntax (especially things like having the same '*' operator to declare and dereference a pointer) So if pointers will be kept in the language - they will almost certainly undergo some syntax changes. I know that pointers are super useful, especially on embedded systems, but I will need to think more about the best way to implement them before they appear in the language.

As to the cake part, the thing that we lack most are effective ways to deal with concurrency. This is holding back the use of multi-core hardware and leads step-by-step programs into problems of semaphores and deadlocks, especially now when we try to use interrupts to multitask.

If there is to be a new language I'd like it to focus on improving the cake. Maybe with objects that communicate messages via queues, but do computation in the way that is already known and accepted. The standard frosting looks just fine.
This is exactly what I have been thinking about myself. Better support for concurrency even on the level of language constructs would be a big win (note, when I say concurrency I don't mean threads)
The best idea I had so far, which would not require a complex runtime, is implementing a kind of an 'event loop' where objects would emit events and react to them. This is similar to message passing and would not change the code structure much.

Another idea I had - is being able to write asynchronous code in a synchronous way. For example, the delay construct could be limited to the current code block, so it does not stop other things from executing at the same time:

void spinMotor () {
   setPin(9, HIGH)
   delay 5s // <- will only delay the code inside the 'spinMotor' function
   setPin(9, LOW) // and won't block other code execution
}



But these are of course 'Big' ideas for the future. Right now I am focusing on making a good language for beginners that is easy to understand and building some helpful tools around it.

If you didn't have s setup() function, where would the stuff that only needs to occur once happen?

I didn't see the unsigned long type in there. Will that be coming to support the time elements?

Will bit manipulation be supported?
I decided that I will use setup() for what it was intended.

Sorry, I forgot to mention the unsigned types in the previous update, but I added the unsigned long and unsigned int in the form of: uint, ulong
All the time literals are assigned a 'ulong' type by default at the moment (this can be optimised in the future)

Regarding bit manipulation - yep, all C/C++ bitwise operations are supported (actually they are just mapped directly because the syntax is the same)

I will definitely publish a list of all the language features and entities that are supported once the demo is released.



[EDIT]
Sorry, I started writing this reply before the last 3 messages were posted.
For now I would like to focus on getting the minimal functionality into the language and make it a 'friendlier' version of C/C++ which is more geared towards Arduino development. However, I would definitely like to improve on some bits in the future versions (which does not mean dismissing the original concept). For example by providing an easier way to express what you want to do then you would in C/C++ (I already talked about this in my reply to @SurferDude's comment)
The best approach would of course be to keep the existing 'cake' untouched, but also add some additional flavours to it (and I don't mean 'overload' with features, we already have C++ for that)

I would also like to keep the UNO chips as the primary target at least for the nearest future (this is still by far the most popular version of the Arduino)

The more modern chips do provide some really cool possibilities, I think we can discuss this as time goes.
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 27, 2015, 04:17 am
Yet another update.


Unfortunately I did not have time to work on the project in the beginning of the week.
So I was trying to cram everything into the past two days.. I really want to release the first version over this weekend, but as usual the most tricky bugs show up right before the demo :)

Initially the IDE was just a prototype which I was planning to rewrite from scratch at some point.
But things did not go exactly as planned, so instead I was adding more and more code to the prototype.
I just HAD to refactor all of this, so I spent a good chunk of yesterday evening on that. Now all the code is much more tidy.

I also had a few cache issues with the IDE (those are the worst, aren;t they?) but I figured it out.

There are a couple things that I was planning to include in the first release, but will most likely have to postpone until future versions:
- I had to remove the 'save' button from the IDE, instead everything is auto-saved. It is just quicker to implement it this way, because a 'save' button means that I would need to keep track of the file states (saved/unsaved)

- I initially added syntax to support classes, so the standard 'library' could use them. But having classes means adding another step to the compilation, because class names can be used before they are declared, so I would need to first go through the code and collect all the class declarations. This step is inevitable though and I will be adding it at a later stage.
For now all the standard built-in functions - are just 'functions', so you have print() and println() for output, pinRead(), pinWrite(), pinMode() for pins, etc. (I will post a full list upon release)

Overall I am doing final polishing before the release , just wanted to give a quick update.

[EDIT] added a couple things to the post, fixed some typos
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 27, 2015, 09:25 am
Quote
everything is auto-saved.
I don't like that idea.

I often find myself making a stupid mess and it is great just to NOT save the file and re-load the original version.

I think if you must use auto-save it should have to a different file each time - with serially-numbered names. But that is very difficult with the Arduino IDE because of its requirement for the file name and directory name to match. I sometimes save a version with a name myProgram.ino.BK1 and the IDE just ignores it when it comes to compile time. (This is on a Linux PC).

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 28, 2015, 07:29 am
I think if you must use auto-save it should have to a different file each time - with serially-numbered names. But that is very difficult with the Arduino IDE because of its requirement for the file name and directory name to match. I sometimes save a version with a name myProgram.ino.BK1 and the IDE just ignores it when it comes to compile time. (This is on a Linux PC).
You will still be able to 'download' a latest copy of any file at any time.
What I mean by auto-saving is that there is no way to loose whatever changes you make to the currently open files.

Its a bit hard to explain, in the IDE you can create new 'files' in the sidebar. You can download a copy of any of those files at any time, both source and transpiled C++ code (this code is to be pasted into Arduino IDE). But the files within the IDE itself are always saved, you can still Ctrl+Z to undo last changes, but it does not ask you to save it when you close the file. Does this make sense? (sorry, I am not great at explaining things)



Also, another quick question.

Do you think I should name the pin control functions the same way as they are on Arduino?

Right now they are pinWrite, pinRead, PinMode, which are shorter to type and all start with 'pin', which sort of 'namespaces' them in a way.
On the other hand they could be named digitalWrite/Read - that way the code can be transferred between languages easier.

PS sorry about the delay so far, just can't get enough time to work on this, and don't want to showcase an incomplete project
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 29, 2015, 12:44 pm
But the files within the IDE itself are always saved,
That is the bit I don't like. Personally I only want it to save the changes when I say so. I am happy to take responsibility for my own failures. But I don't want my perfectly good working code automatically overwritten by some garbage that I typed and then thought better of. I do realize, of course, that the code needs to be saved before it can be compiled. Maybe I am misunderstanding when your program saves the code?

Quote
Right now they are pinWrite, pinRead, PinMode, which are shorter to type and all start with 'pin', which sort of 'namespaces' them in a way.
I rather like this style. It would be easy to have a few macros in regular Arduino code to make the same commands be converted to digitalWrite() etc.

It is probably a typo - but you have an uppercase P for your PinMode - be consistent (I prefer to start with lowercase).

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Nov 30, 2015, 06:30 am
That is the bit I don't like. Personally I only want it to save the changes when I say so. I am happy to take responsibility for my own failures. But I don't want my perfectly good working code automatically overwritten by some garbage that I typed and then thought better of. I do realize, of course, that the code needs to be saved before it can be compiled. Maybe I am misunderstanding when your program saves the code?
I think you got it right, and yes, as I have already discovered myself during testing that this can be a bit of a pain. (you can use Ctrl+Z to get the changes back, but if you reload the IDE - it won't help)
I wanted to auto-save everything just because it is quicker to implement it that way, but since I did not release the demo yet, I will probably just take a bit more time and do it properly.


I rather like this style. It would be easy to have a few macros in regular Arduino code to make the same commands be converted to digitalWrite() etc.
It is probably a typo - but you have an uppercase P for your PinMode - be consistent (I prefer to start with lowercase).
Oops, yep, that is definitely a lowercase. I was actually thinking of making it mandatory that all functions start with a lowercase ( and all class names start with uppercase ) and constants are all caps. I have not implemented this yet though, but I think it could improve the code style.


PS also, sorry again about missing the weekend release mark. This project obviously turned out longer then I initially planned (although I was not planning to make the IDE back then)

The compiler itself is complete (I blinked an LED for the first time with it yesterday :)
Right now I am just polishing all the stuff around it, so it does not feel too 'raw' when showcased.
I also want to make the project open source so the code needs to be cleaned up a bit (that's what I spent most of my time on in the last few days)

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Nov 30, 2015, 11:57 am
you can use Ctrl+Z to get the changes back,
Sorry if I am labouring this point too much ... but just to clarify further ... (and this may be just my own foolishness)

Sometimes I get an idea for a substantial reorganization of a piece of code and rush in to make changes. After 30 minutes or so I have got completely confused. At that stage Ctrl+Z isn't really any help - partly because it is a sequential process and I can no longer remember all the changes I have made, never mind the order in which I made them. In that situation starting over is often easiest -  so dump all the changes and go back to the original version.

Don't worry about a few days delay. Although we might lose patience after 3 months  :)

...R
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Nov 30, 2015, 06:15 pm
That sounds very familiar.
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 01, 2015, 06:17 am
Quote
That sounds very familiar.
Sometimes I get an idea for a substantial reorganization of a piece of code and rush in to make changes. After 30 minutes or so I have got completely confused. At that stage Ctrl+Z isn't really any help - partly because it is a sequential process and I can no longer remember all the changes I have made, never mind the order in which I made them. In that situation starting over is often easiest -  so dump all the changes and go back to the original version.
I don't think there is anything wrong with you, all developers I know are like that, that's why we use source control :)
This actually makes me think if it would be a good idea to add some sort of "Versioning" to the files, something that allows you to save the current version of the file - and then get back to it at any time. There can be multiple versions to one file. (This definitely won't be in the first release, but something to think about for the future)

Quote
Don't worry about a few days delay. Although we might lose patience after 3 months  :)
We are at 3 weeks and one day mark right now. (initially I was planing to be done in one week, hehe)
The first version of compiler is already done, now its all about making the IDE usable.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 01, 2015, 11:54 am
I confess (but don't tell anyone else) that I do (usually) make backup copies of programs before embarking on major changes but I have never managed to get my head around formal version control systems.

I can see how they would be mandatory in a multi-programmer environment, but even then I have never found a clear explanation of how they actually work to resolve conflicts between a change made by A and a different change to the same thing made by B. Or how you could take 75% of A's changes and 25% of B's changes.

I cut my teeth on a small DEC PDP/11 and on one of their operating systems it automatically saved files with successive version numbers. I recall attending a tutorial one day where I enjoyed watching the screen behind the tutor gradually creating so many backup copies that it would have run out of disk space if I had not drawn attention to it at the 11th hour. (Probably a 10MB disk).

...R
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Dec 01, 2015, 01:43 pm
Quote
on one of their operating systems it automatically saved files with successive version numbers.
Sounds like VMS
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 01, 2015, 02:13 pm
I can see how they would be mandatory in a multi-programmer environment, but even then I have never found a clear explanation of how they actually work to resolve conflicts between a change made by A and a different change to the same thing made by B. Or how you could take 75% of A's changes and 25% of B's changes.
Well, the exact way of how merging is done will of course depend on which version control system you are using.
I personally use git, and its real power comes with branches. For example you could have a separate branch for every feature in your project, and withing those branches will be separate commits. So when you merge two branches - you can merge only individual commits and not the entire branch.

Version control definitely makes more sense when multiple developers are involved, but I still find it quite useful when I work by myself on my own projects. It gives the project some 'order', and it is nice to be able to view its entire history and go back to any point.
For example when I work on my own - I will have the master branch with the stable version of the project and I will also have a few branches where I am developing new features - once the feature is complete and tested - I merge it into the master branch. This way all new features have a separate history and I don't have to remind myself what I was doing when I last looked at the code. ( Having a backup of all my work on github is pretty cool too )

I cut my teeth on a small DEC PDP/11 and on one of their operating systems it automatically saved files with successive version numbers. I recall attending a tutorial one day where I enjoyed watching the screen behind the tutor gradually creating so many backup copies that it would have run out of disk space if I had not drawn attention to it at the 11th hour. (Probably a 10MB disk).
That sounds a bit crazy to be honest ) I would definitely require the user action to create a new version of the file, and not do it automatically (so basically its like creating a separate 'backup' file)
In order to save space - it can store just the difference between new version and the previous one instead of saving the entire content of the files.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 01, 2015, 10:07 pm
I personally use git,
I know this is off-topic, but are you aware of any "tutorial" that explains the way to use Git. I could make no sense of their own website. It explains how to use the commands, but not the underlying concept.

...R
Title: Re: Designing a new programming language for Arduino
Post by: SurfingDude on Dec 01, 2015, 10:13 pm
Regarding Git, SparkFun has a pretty good tutorial: https://learn.sparkfun.com/tutorials/using-github
Title: Re: Designing a new programming language for Arduino
Post by: westfw on Dec 02, 2015, 07:03 am
Git/Github free online class: https://www.udacity.com/course/how-to-use-git-and-github--ud775 (https://www.udacity.com/course/how-to-use-git-and-github--ud775)
It includes quite a bit of introduction to the basic principles of revision control, and it's pretty good.  The "exercises" are well matched to the lecture material - a weak point in some other online classes I've taken :-(

For single-person projects, I've been using "rcs", one of the granddaddies of this class of programs.  I've used it "professionally", and it seems to be sufficient as long as only one person is modifying a particular file at one time.
"cvs" (which I've also used professionally) adds more complex "merge" capabilities and is useful for several people sharing the same workspace and sometimes colliding with each other.  The newer programs (svn, git, mercurial) add network-shared code repositories and are useful with bigger and more dispersed developer communities.  (there are other differences as well; slight tweaking of philosophies and such.  But that's how I think of things.)

VMS (on vax) and TOPS20 (on PDP10) had "file versions" that could be used; very handy (and I'm surprised that none of the PC operating systems have done anything similar.)  There were settings for controlling the maximum number of versions of a file to keep.  Sigh.  One of the "lost capabilities" in the transition from mainframes to micros.

(RSX-11 apparently also had a versioning file system. https://en.wikipedia.org/wiki/Versioning_file_system (https://en.wikipedia.org/wiki/Versioning_file_system))


Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 02, 2015, 10:14 am
Thanks for the links Guys. I have bookmarked this for follow-up reading over the next few days.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 02, 2015, 03:13 pm
Just to add to the previous comments, I gotta say that I personally had to learn git the 'hard way', I got hired in a company where they were already using git. To be honest it took me some time to completely understand what's going on (I was mostly using svn before), but now I don't think there is a better source control system at the moment.

I just had a quick look around on google, and this tutorial seems to be pretty straigth to the point: http://rogerdudler.github.io/git-guide/ (http://rogerdudler.github.io/git-guide/) (title says it all hehe)

Overall, if you want to just get started quickly - cd to your project directory (as far as I understood you are on linux)
1) Run 'git init' to initialize the repository.
2) Then run 'git add --all' - adds all uncomitted changes to index
3) Finally run 'git commit -m "COMMIT MESSAGE"'

Then whenever you make changes that you want to save - just run #2 and #3
Commit messages are usually things like "Implemented feature X" or "Closed bug #777", etc.

I would also use some visual tool to view the commit history (like DAG or git cola) - it can give you a good visual representation of the repository.
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 02, 2015, 04:23 pm
Now to bring it a bit more on topic :)

Here is a brief intro into the language syntax
I will definitely publish a more comprehensive documentation after the release, but I thought it would be good to give a quick summary of all the features that are currently implemented.


Variable initialization - inferred types, (also - optional semicolons)
foo = 10 // int
bar = "Hello" // str


Parenthesis in if and for statements are optional, but the curly brackets are mandatory.
if a < 5 { ... }

Alias keywords for common operators - and, or, xor
if bar < 500 and foo > 100 { ... }


I also added alias for equality operators - is and is not (convert to - '==' and '!=')
if reading is not 0 { ... }
if ledPin is HIGH { ... }


Loops - there are only for loops in the language.
They can take just the testing condition:
for i < 10 { ... }

Otherwise they work the same way as regular for loops:
for i = 0; i < 10; i++ { ... }


You can use break and continue statement,
there is also an alias for 'continue' - skip

As I mentioned before - functions can have docstrings:
/// This is the function description
int readPin ( int number ) { ... }


There are aliases for the built in functions:
println() - Serial.println()
pinRead() - digitalRead()
now()     - millis()

(Please let me know in case I forgot anything)


Also, here is a listing of all errors that are currently caught by the compiler:
(Sorry about the format, it is pretty much straight from the source)
Code: [Select]
    VARIABLE_STATEMENT   
      type_mismatch       'variable "{0}" does not match type "{1}"',
      already_exists      'variable "{0}" was already initialized as {1} on line {2}',
      does_not_exist      'variable "{0}" is not defined'

    VARIABLE_DECLARATOR   
      already_initialized 'variable "{0}" was already initialized as {1} on line {2}',
      already_declared    'variable "{0}" was already declared on line {1}'

    FUNCTION_DECLARATION 
      already_exists      'function "{0}" was already declared on line {1}',
      no_return           'function "{0}" must return {1}',

    PARAM_DECLARATOR     
      duplicate_param     'duplicate parameter "{0}"'

    RETURN_STATEMENT     
      return_outside      'return statement outside function declaration',
      type_mismatch       'return statement type does not match function "{0}"'

    IDENTIFIER           
      does_not_exist      'variable "{0}" does not exist'

    BINARY_EXPRESSION     
      type_mismatch       '"{0}" operation types do not match, can\'t {operator: 0} "{1}" and "{2}"'

    LOGICAL_EXPRESSION   
      type_mismatch       'can\'t compare "{0}" and "{1}"',
      cant_compare        'can\'t compare "{0}" and "{1}"',
      not_a_number        'can use "{0}" operator only on numbers'

    ASSIGNMENT_STATEMENT 
      type_mismatch       'assignment types mismatch, can\'t assign "{0}" to "{1}"'

    UPDATE_EXPRESSION     
      not_a_number        'expression is not a number'

    UNARY_EXPRESSION     
      not_a_bool          'unary expression must be a boolean'

    IF_STATEMENT         
      not_a_bool          'IF condition must be a boolean'

    FOR_STATEMENT         
      not_a_bool          'FOR test expression must be a boolean'

    BREAK_STATEMENT       
      outside_loop        'BREAK outside FOR loop'

    CONTINUE_STATEMENT   
      outside_loop        '{loopStatement: 0} outside FOR loop'



Title: Re: Designing a new programming language for Arduino
Post by: MorganS on Dec 02, 2015, 07:59 pm
So you haven't got to the point of having an error message for the original problem: unmatched parentheses.

This is not a hard problem. It's unsolvable. No imaginable quantum-computing AI can solve this. A god who can examine the mind of the programmer might be able to do it. How many times have you seen a newbie post some code here (after being told to use code tags) and the answer is "We think you meant to put a closing brace <here> but that may not do what you wanted. What did you want it to do?"

The thing I love about the Arduino is that it is approachable for high school and even primary students but it's not a walled garden. If you absolutely must put in some clever code that saves one clock cycle in an ISR, then you can do it and you haven't broken the walls of the garden so that the simple delay() function stops working. You can write 99% of your code with standard Arduino functions and your main program logic will be easy to understand 6 months from now when you have to look at your own code again and ask "Who wrote this crap?"

Delays that only delay inside one code block? We already have that on the Due with the Scheduler library. It's a standard part of Arduino. It uses more memory to maintain different contexts and time is lost switching, which is why it's not used on the smaller Arduinos.

The classic computer-science test for a new language is "Can you write its own compiler in the language?" I think that's a big hurdle and probably not appropriate here. I would ask, "Can I write a library for a sensor in this new language?" Can I take an accelerometer or CO2 sensor or whatever and write some code in this new language which will enable it to be used as simply as a readPin() function? If I have to write the library code in C++ and then switch to this different language to write the "real program" I'm working on, then you've lost. I'm not going to switch languages in the middle of a project.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 02, 2015, 10:43 pm
Parenthesis in if and for statements are optional, but the curly brackets are mandatory.
I suspect it would be less confusing if the parenthesis are also mandatory


If I have to write the library code in C++ and then switch to this different language to write the "real program" I'm working on, then you've lost. I'm not going to switch languages in the middle of a project.
I don't think this is the intended market for @YemSalat's system.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 03, 2015, 02:57 pm
So you haven't got to the point of having an error message for the original problem: unmatched parentheses.
Sorry, I should have mentioned that the errors that I posted - are semantic errors.
Missing parentheses is a syntax error, and it would be caught before the semantic analyzer runs:

(http://i.imgur.com/5SbT0AB.png)

Quote
This is not a hard problem. It's unsolvable. No imaginable quantum-computing AI can solve this. A god who can examine the mind of the programmer might be able to do it. How many times have you seen a newbie post some code here (after being told to use code tags) and the answer is "We think you meant to put a closing brace <here> but that may not do what you wanted. What did you want it to do?"
The key here - is catching errors early and providing more meaningful error messages. Semantic check is also very important, it definitely won't automatically solve all developers' problems, but it will help prevent the more 'technical' ones.


Quote
Delays that only delay inside one code block? We already have that on the Due with the Scheduler library. It's a standard part of Arduino. It uses more memory to maintain different contexts and time is lost switching, which is why it's not used on the smaller Arduinos.
To be honest I did not know about the Scheduler library, but I would imagine something like that exists, because it is a useful feature to have.
But if you take a look at a simple LED blinking example with the said lirary: https://www.arduino.cc/en/Tutorial/MultipleBlinks (https://www.arduino.cc/en/Tutorial/MultipleBlinks)
You'll see that there is plenty of code that is not related to the main goal (delaying function execution) - you gotta do Scheduler.startloop() for each task, also add a yield() at the end. The loop is added by default, but the rest of the functions are not, so you need to remember that as well.
Regarding performance - a lot can be done by re-structuring the resulting code in a certain way, this is a real benefit of a transpiler over libraries. Some things will definitely require some sort of runtime, but all of these are just plans for the future at the moment.


Quote
Can I take an accelerometer or CO2 sensor or whatever and write some code in this new language which will enable it to be used as simply as a readPin() function? If I have to write the library code in C++ and then switch to this different language to write the "real program" I'm working on, then you've lost. I'm not going to switch languages in the middle of a project.
As @Robin noted - I am not making a replacement for C/C++, my goal as I explained before - is to create a language with a 'lighter' syntax that would be more tailored to the Arduino environment and take care of some things (like string manipulations) behind the scenes. I am also putting a big accent on the tools - the IDE with semantic analyzer and syntax checker. Having said that - I don't see any reason why you would not be able to read accelerometer pins, in fact it is already possible.




I suspect it would be less confusing if the parenthesis are also mandatory
I thought about this, but in the end I decided to go with Ruby / Python style for the condition expressions.
We'll see how it goes and I might make parens mandatory, but I think with appropriate syntax highlighting - the code looks a bit 'cleaner' without them.
Title: Re: Designing a new programming language for Arduino
Post by: MarkT on Dec 04, 2015, 08:02 pm
Have you seen the talk on the Sonic Pi system by Sam Aaron, https://www.youtube.com/watch?v=e_CQpFaTGyw (https://www.youtube.com/watch?v=e_CQpFaTGyw)?

Its all about how to make a notation that's easy to use for music coding, but the principles
would apply to more general microcontroller programming.
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 05, 2015, 10:36 am
Its all about how to make a notation that's easy to use for music coding, but the principles
would apply to more general microcontroller programming.
Thanks for the link, the language he is showcasing does seem interesting and I agree that somewhat similar principles could be applied in microcontrollers.

Since we are sharing video links.. Here is a series of videos from Stanford University and Open Course, its about compiler design, it is a bit more technical in a sense that they teach more about the compiler design rather than language design.

https://www.youtube.com/watch?v=sm0QQO-WZlM&list=PLFB9EC7B8FE963EB8 (https://www.youtube.com/watch?v=sm0QQO-WZlM&list=PLFB9EC7B8FE963EB8&index=1)

I wish I started watching these earlier, could have probably saved some time on writing the compiler )


A quick update on the IDE.
I am working on syntax highlighting, I think it can really help distinguish between language entities. Ace editor that I am using has an api for adding custom highlighters. Since the syntax is compatible with C - I just forked the default C highlighter and modified it a bit.

I also settled on the two main themes for the IDE - dark (monokai) and light (tomorrow):

(http://i.imgur.com/cxUMB7j.png) (http://i.imgur.com/ODV7RWN.png)


The IDE UI depends on the selected theme, so for example when the editor theme is dark - the UI around it is dark as well.
Ace comes with a few other themes for syntax highlighting (like eclipse, gihub, etc.) - and its not gonna be hard to add more themes in the future.

I also fixed a few small bugs in the code generator.
I am still not entirely sure that I got the C/C++ automatic type conversion for things like float/int/bool exactly right, will definitely need to do some thorough testing on it.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 05, 2015, 08:29 pm
I hope the highlighting will work with Geany (http://www.geany.org/)

Edit to add ...
Perhaps a more general comment would be that your system will have limited appeal if it is tied to a particular editor.


...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 06, 2015, 01:52 am
Perhaps a more general comment would be that your system will have limited appeal if it is tied to a particular editor.
It does not have to be tied to any particular editor at all.

The compiler is written in node, so you can just run it from command line on any operating system. It is just easier for me to add features like on-the-fly code analyzer, if I have complete control of the environment.

If the language catches up - I will be happy to create highlighters for other editors.

I hope the highlighting will work with Geany (http://www.geany.org/)
To be honest I have not heard of that one before, but judging by their api docs - its gonna be quite trivial to add a new syntax highlighting: http://geany.org/manual/dev/#creating-a-custom-filetype-from-an-existing-filetype (http://geany.org/manual/dev/#creating-a-custom-filetype-from-an-existing-filetype)
(its mostly about copying C/C++ highlighter and adding a few more keywords and structures)
Title: Re: Designing a new programming language for Arduino
Post by: westfw on Dec 06, 2015, 03:18 am
So...  Could this hypothetical language pre-processor parse and convert alternate syntaxes?  For example, the "C" for-loop structure is particularly confusing to beginers (and many other languages aren't much better), while I've always found the ancient BASIC for-loop syntax to be unusually obvious.  It seems that a parser ought to be able to detect:
Code: [Select]
   for i = 1 to 10 step 2
distinct from a C for-loop:
Code: [Select]
   for (i=0; i < 10; i++)
And still easily emit "real C" code for each...
While you're at it you you might as well accept Ada/Ruby syntax as well:
Code: [Select]
   for i in 1..5

It could be a general concept: as long as the parser can uniquely recognize the statement, accept the syntax of several popular languages that the user might be familiar with.  (This is in contradiction to most language design, I think, which try to simplify each basic program structure to a single syntax.)

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 06, 2015, 12:47 pm
but judging by their api docs - its gonna be quite trivial to add a new syntax highlighting:
In that case I may volunteer to help.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 13, 2015, 01:18 am
@westfw, I am really sorry about such a delayed reply, I did not notice your message before (the forum scrolls right to the last one when I click the topic title, my bad!)

while I've always found the ancient BASIC for-loop syntax to be unusually obvious.  It seems that a parser ought to be able to detect:
Code: [Select]
   for i = 1 to 10 step 2
I do like this syntax for its simplicity too, but I think there should be other ways of simplifying loops besides additional having keywords (or would be good to find other uses for these keywords).

While you're at it you you might as well accept Ada/Ruby syntax as well:
Code: [Select]
   for i in 1..5
I am actually thinking of having this syntax not only for loops, but maybe as a 'range' construct (similar to Python ranges) But I have not figured out a good way to implement it yet.

It could be a general concept: as long as the parser can uniquely recognize the statement, accept the syntax of several popular languages that the user might be familiar with.  (This is in contradiction to most language design, I think, which try to simplify each basic program structure to a single syntax.)
To be honest I don't really want to put to much 'syntax' into the language and I think that such a feature (combination of syntaxes) may actually do more harm then good (imagine reading through the code in all possible languages at once).
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 13, 2015, 01:32 am
[ Progress Update ]

Sorry everybody for the lack of updates for the past week, I was really busy at work, considering its Christmas in just a couple of weeks and I did not have too much spare time to work on the project.

I did some more refactoring, and even submitted a patch to pegjs - the parser generator that I am using because I need a specific feature for the language - https://github.com/pegjs/pegjs/pull/391 (https://github.com/pegjs/pegjs/pull/391) - they seem pretty cool in terms of accepting pull requests o hopefully it will get into the library soon.

And I have kind of big news: I decided to spend more time on the IDE and release the compiler separately.
So I made the compiler Open Source!

It is now available on GitHub: https://github.com/YemSalat/spark-lang (https://github.com/YemSalat/spark-lang)

The compiler is fully-functional from the command line, but I only had a chance to test it on Linux so far.
I will not urge you to go ahead and test it right now, especially if you are not very familiar with the node environment.
But overall it is quite easy and comes down to running sparc filename.sprk -o output.cpp
If you don't specify the output- it will just print C++ code to command line.

Later today or tomorrow - I will upload an online version which will not require instalation and will be much easier to test.


I also had to pick a name for the language, I was refering to it as 'Spark' during development, but there are a couple 'complications' with this name.
The compiler is named sparc and in the official grammar spec I refer to the language as 'Sparklang'. So as you can see its a bit of a mess right now :) Any naming suggestions are welcome!

PS This is not the 'BIG' release yet, I will still be working on the IDE, and I will upload an online testing environment over the rest of the weekend

PPS feel free to post suggestions and issues on GitHub as well
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 13, 2015, 12:32 pm
I am currently putting up a quick online testing environment for the language.
My main objective at the moment is to make sure that when IDE is out it has as little issues as possible.

In the testing environment - you can input spark code on the left, press 'Eval' - and it outputs C++ on the right.

(http://i.imgur.com/LRXbMji.png)
full size (http://i.imgur.com/LRXbMji.png)

It also outputs a 'prettified' parse tree, so one can make sure that code is interpreted correctly.

I will update the post once I upload the testing environment (PS you can subscribe on github for updates as well).
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 13, 2015, 12:53 pm
Great progress!

Quote
Any naming suggestions are welcome!
To start the ball rolling, what about

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 13, 2015, 03:48 pm
Thanks!
To start the ball rolling, what about
  • ArduLang
  • ArduEasy
  • InoLang
  • InoEasy

I actually really like 'ArduLang', but I do not want to tie it too close to the Arduino brand (especially considering stuff that has been happening in the past couple months), 'Ino..' seems a bit ambiguous in terms of
 pronunciation (is it '[ I-no ]' or '[ Ee-no ]' ?).

I also got this idea a couple days ago that it would be great to try and expand the language to other platforms (e.g. Raspberry PI). I made a small remote controlled robot with RPi about a year ago, I wrote it in Python and node. And I gotta tell you - 8bit Arduino running C++ completely beats RPi running Python in terms of IO performance.

Things have improved since the release of RPi2, but there are still plenty of old boards around and overall I think it would be nice to be able to have the same code working across different platforms, using the same APIs. Especially since projects that involve multiple platforms are becoming more and more frequent ( This is just something to think about in the future of course, I am only focusing on Arduino implementation at the moment )
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 13, 2015, 06:29 pm
I like the idea of cross-platform stuff - especially the idea that I can write a program on this Linux PC and give to to the neighbour's Granny on a USB stick and she just plugs it into any other PC and clicks the filename with the mouse and the program runs. You can pretty much do that with Python. And you can do that with a JVM based program if Granny has the JVM installed (but programming in Java is the pits).

I can't see the same functionality being possible for any system (like the Arduino) that is designed to interface with specialized hardware.

My instinct is to focus on a name that includes the word "easy" or a synonym because then it is simple to explain the difference between your system and regular C/C++ and it will avoid upsetting the purists. And avoiding criticism from that body of diehards could be important in the drive to gain market share.

If your idea is to use much the same program language on different platforms without expecting a program to be transferrable between (say) an Arduino and an RPi, then I think there is merit in having (for example) both ArduEasy and RPiEasy.

Just my 2 cents

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 14, 2015, 05:06 pm
I can't see the same functionality being possible for any system (like the Arduino) that is designed to interface with specialized hardware.
I think its a question of whether it is possible to come up with flexible enough abstractions for specific hardware.
There is no way to cover everything of course.

Quote
If your idea is to use much the same program language on different platforms without expecting a program to be transferrable between (say) an Arduino and an RPi, then I think there is merit in having (for example) both ArduEasy and RPiEasy.
Yep, sort of like that, basically having the same language that can be compiled to different platforms.
So, for example println() is printing to Serial on Arduino, but on RPi it can print to stdout instead - the benefit is having exactly the same APIs, but the source itself will have to be recompiled.

[UPDATE]

Just released a quick preview of the test environment.
You can check check it out here: Language TEST - http://webcloudtools.com/sprk/lang-test/ (http://webcloudtools.com/sprk/lang-test/)

Note: I am changing the way type checking works a little bit, so I had to disable type checking for functions for now and also 'unsigned' number types (will enable once refactoring is complete)

I know there is not much to see there at the moment but it would be great if you could give it a shot. I am mostly worried if I got automatic conversion for number types right. Would really appreciate if you could let me know if there are any issues with it (or any other stuff as well)

I will be updating with more functionality tomorrow.

PS as I said before, this is not the actual IDE yet, just want to make sure the basic stuff works right
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 15, 2015, 11:05 am
Hi everybody, I am getting a bit stuck on trying to write a string concatenation function that would work properly on Arduino.
As far as I know it is best to avoid malloc() calls on avr (is that true btw?)

@UkHeliBob suggested using snprintf for concatenating two strings, here is my best shot at it:
Code: [Select]

    st1 = "Hello ";
    st2 = "World";

    hello = st1 + st2;


Will be converted into:
Code: [Select]

    char st1[] = "Hello ";
    char st2[] = "World";

    size_t strSize = sizeof st1 + sizeof st2;
    char hello[strSize];

    snprintf(hello, strSize, "%s%s", st1, st2);


Do you reckon this implementation is OK? (I have not tested it on Arduino yet, but I will in the next couple days)

PS Quick update - I fixed a couple bugs in the test environment and will upload the new typing system later today. Test URL is the same: http://webcloudtools.com/sprk/lang-test/ (http://webcloudtools.com/sprk/lang-test/)

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 15, 2015, 05:48 pm
I think (due to the limited Arduino memory) I would prefer a system in which the size of the array to receive the two strings had to be decided beforehand by the programmer and then the concatenation would put as much of the two strings as would fit into it. Something like

Code: [Select]
st1 = "Hello ";
st2 = "World";

stCombined[10];

stCombined = st1 + st2; // which would give "HelloWorl\0"


Otherwise I think you are creating the same problem that the String class suffers from.

------
I can't remember if you have mentioned it already, but how are you dealing with the scope of your variables?

...R
 
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 16, 2015, 01:48 pm
I think (due to the limited Arduino memory) I would prefer a system in which the size of the array to receive the two strings had to be decided beforehand by the programmer and then the concatenation would put as much of the two strings as would fit into it. Something like
That's actually a pretty good idea, being able to set the string size limit like that.
I will add it to the TODO list: https://github.com/YemSalat/spark-lang/blob/master/TODO.md (https://github.com/YemSalat/spark-lang/blob/master/TODO.md) (by the way, feel free to contribute to it)

------
Quote
I can't remember if you have mentioned it already, but how are you dealing with the scope of your variables?
I don't think I have, right now the scope is static and is pretty much the same as C/C++; Each block creates its own scope.


For example:
(http://i.imgur.com/VDP8cen.png)
Here each declaration is inside its own block scope.

But, if we move it outside - it will trigger a 'double-declaration' error:
(http://i.imgur.com/p7WVlvB.png)
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 16, 2015, 02:30 pm
That's actually a pretty good idea, being able to set the string size limit like that.
Just to be clear, my suggestion is that pre-defined size limits should be the ONLY option.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 16, 2015, 03:12 pm
I would still like to be able to save user the hassle of having to calculate sizes when the new string must contain both substrings completely.
Regarding the same problems as the original String class - I do want to avoid that kind of stuff, but everywhere I read it seems like snprintf is a really safe option.
I am not exactly sure but from what I understand, the main problem with the String class is that it uses a buggy version of free() function?

The great thing about the transpiler - is that it can analyze source and output just the required code, so for example, if you are not using variable length strings - it will only output the code for simple string manipulations (like in your example).
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 16, 2015, 08:38 pm
I would still like to be able to save user the hassle of having to calculate sizes when the new string must contain both substrings completely.
Regarding the same problems as the original String class - I do want to avoid that kind of stuff, but everywhere I read it seems like snprintf is a really safe option.
I have not studied the code  behind the String class but my understanding is that there are, potentially, two types of problem. One is that it frequently creates new copies of a string while it is doing its work and can therefore use up a lot of memory that the user cannot be aware of. The other is that it may not release all the unused memory in an adequate fashion.

My strong sense is that what you say in the first sentence I have quoted will have a similar effect of consuming memory that the user has not allocated, or made allowance for in his/her program design. It matters little that it might be a tidier user of memory than the String class - that just postpones the crash.

I also think it is psychologically a good thing to force users to be aware of the memory limitations rather than hide it all from them. Hiding the C functions such as strcpy() will be sufficient.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 17, 2015, 01:09 pm
Quote
I also think it is psychologically a good thing to force users to be aware of the memory limitations rather than hide it all from them. Hiding the C functions such as strcpy() will be sufficient.
I agree with you on this, I'll probably leave this idea for later and will implement simple string concatenations for now (the ones with specified length limits)

I might create a separate thread later in the Programming section to get more info on the subject of strings on Arduino (I think this topic might be too big to cover within this thread)
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 17, 2015, 01:28 pm
[UPDATE]

A quick update on the compiler progress. Link to test: Test Environment (http://webcloudtools.com/sprk/lang-test/#functions)

I am finally done with refactoring the function declarations, so now they also get evaluated by the semantic analyzer:
(http://i.imgur.com/9n4m87z.png)
(full image) (http://i.imgur.com/O1Eco0c.png)

You can have functions with the same name, but different signatures (signatures must be unique).

The analyzer currently runs the following checks on each function declaration:

- Check if function was already declared
- Check if function contains duplicate parameters
- Check if function returns a value (this is skipped for void functions)
- Check if all function return statements have the same type as the function declaration

Please let me know if you find any errors in the implementation.
Thanks in advance!

PS I am also thinking of adding short-hand function declarations, using -> operator
So, for example, instead of writing:
int add (int a, int b) {
  return a + b
}


You could write:
int add (int a, int b) -> a + b
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 17, 2015, 07:22 pm
PS I am also thinking of adding short-hand function declarations, using -> operator
So, for example, instead of writing:
int add (int a, int b) {
  return a + b
}


You could write:
int add (int a, int b) -> a + b
May I respectfully suggest that it is now time to STOP.

That concept would require users to learn a new idiom that has no parallels in regular Arduino programming.

Ruby-on-Rails was a great idea just before they moved on to Version 2 (dunno where they are now) but then it just got more and more complex because IMHO the developers could find nothing more interesting to occupy them.  :)

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 18, 2015, 05:32 am
May I respectfully suggest that it is now time to STOP.
Yep, fair enough, hehe, I can get carried away sometimes. I agree that its probably best not to introduce any 'brand-new' syntax (at least at this stage) I might save this idea for later.

I am working on evaluating function calls right now and will be uploading more updates later today.


PS
I never really got into RoR (played a bit with one of the earlier versions), I think it could be because I had some bad experiences in the past trying to install a couple apps that were written in it. But your 'analogy' actually made me curious to go and check it out, maybe save some refactoring time in the future :)
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 18, 2015, 11:13 am
I had some bad experiences in the past trying to install a couple apps that were written in it.
One of the fascinating things is that they developed a really clever system for simpifying web development but completely neglected the "infrastructure" needed to make it convenient to use it. Eventually some other guy developed "Passenger" aka "modRails" for easy deployment with Apache - as PHP could do all along.

For similar reasons I finally gave up Ruby/JRuby in favour of Python. But I had abandoned RoR before that in favour of Sinatra. The equivalent for Python is Bottle.

...R

Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Dec 19, 2015, 04:23 am
Yep, fair enough, hehe, I can get carried away sometimes. I agree that its probably best not to introduce any 'brand-new' syntax (at least at this stage) I might save this idea for later.
Well actually, its not as new or unrelated of a concept in C++ as you might think. Your simple example would easily translate to a lambda.


auto func = [](int a, int b){ return a + b; };

//...

int sum = func(4, 5);


They are quite useful with code that requires lots of callbacks which may do very little, they can be used as nested/sub-routines (defined inside another function, not just globally) and can be completely inline.

Code: [Select]
template< typename Func > void longTaskWithCallback( Func func );

void setup() {

  Serial.begin(9600);

  //Call longTask.. and do something when complete.

  longTaskWithCallback([](){
    Serial.print( "done" );
  });

}

void loop(){}

template< typename Func > void longTaskWithCallback( Func func ){

    //Do something long

    //Run callback function
    func();
}


Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 19, 2015, 09:37 am
Your simple example would easily translate to a lambda.
I don't think it would help to expose beginners to that sort of complexity - and the purpose of @YemSalat's work is to make life easier for beginners.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 20, 2015, 05:34 am
Well actually, its not as new or unrelated of a concept in C++ as you might think. Your simple example would easily translate to a lambda.
I will definitely need to look more into it. I have never really used lambdas in C++, but they do seem very useful (as you said, perfect for short callbacks, etc.). However, it seems like they have only been added in c++11? Do you know if all Arduino IDE versions support it?

As @Robin mentioned, I am currently focusing on making the language beginner-friendly, so it is probably best not to introduce lambdas at this point. If I ever do in the future - I will also need to make sure that the syntax is simple enough and does not conflict with other language constructs.
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Dec 20, 2015, 06:25 am
I will definitely need to look more into it. I have never really used lambdas in C++, but they do seem very useful (as you said, perfect for short callbacks, etc.). However, it seems like they have only been added in c++11? Do you know if all Arduino IDE versions support it?

As @Robin mentioned, I am currently focusing on making the language beginner-friendly, so it is probably best not to introduce lambdas at this point. If I ever do in the future - I will also need to make sure that the syntax is simple enough and does not conflict with other language constructs.
IDE 1.6.6 and above have it enabled by default.

I noticed you are interested in ranged for loops. These exist in C++11 also. For example:

Code: [Select]
float data[10];

for( float &element : data ){

  //use the element
  Serial.print( element, 3 );
  element = 3.14f;
}

//you could simply replace the ':' with the word 'in' and use a reference by default for anything larger than a char
// (and maybe 'for' to 'foreach' << would be easier to do the grammar maybe):

foreach element in data {

}


Although I do not think it is a good idea having the brackets optional, and the curly brackets mandatory. Just enforce both.

All ranged for loops in C++ can be equivalently written as a standard for loop in the older C++98 which the previous IDE's used.

There is also an std template (or I have my own) that will allow you to do numerical ranges, mine will also allow stepping > 1 and reverse iteration. Similar to the boost counting range: http://www.boost.org/doc/libs/1_47_0/libs/range/doc/html/range/reference/ranges/counting_range.html (http://www.boost.org/doc/libs/1_47_0/libs/range/doc/html/range/reference/ranges/counting_range.html)

Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 20, 2015, 07:33 am
IDE 1.6.6 and above have it enabled by default.
Good to know, thanks.
And yes, I would like to add ranged loops, I have also been thinking about having a 'for .. in ..'  syntax to express them.
Its nice that C11 supports them out of the box, but it probably makes sense to generate C++98 as you said.
In terms of writing the grammar - there is really not much difference between having a 'for' or 'foreach' to indicate a ranged loop, but I would really like to get away with just using 'for' (it currently also replaces 'do' and 'while' loops)
PS I'd be interested to see your example as well if you don't mind?

Quote
Although I do not think it is a good idea having the brackets optional, and the curly brackets mandatory. Just enforce both.
To be honest, after a couple weeks of constantly testing the language - I just got really used to omitting the brackets. But you are not the first one to mention this, so I will probably change this in the nearest version.



[ A quick Update ]

I spent some more time on function evaluation. There are still a couple things to tweak, but overall I am quite happy with it.

I also added default function argument values. It uses a well-established syntax (e.g. like Python)
int multiply (int a, int b = 2) {
  return a * b
}


Now if you call:
multiply(7) - with just one parameter,
the compiler will automatically add the default one:
multiply( 7, 2 )

Right now the default values can only be literals (strings, numbers, etc.) but I am planning to allow using constss as well.
All default params - must be declared last, so you can't have this:
int multiply(int a=1, int b) {} // (compiler will throw an error)

What I really like about this feature is that all the default parameters are resolved during compilation, so there is no overhead for using them. It is of course mostly just a developer convenience.

You can check the latest version here http://webcloudtools.com/sprk/lang-test/#functions (http://webcloudtools.com/sprk/lang-test/#functions)
(I updated the functions example)

Next step is gonna be adding arrays and working on string concatenation.
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Dec 20, 2015, 07:52 am
Good to know, thanks.
And yes, I would like to add ranged loops, I have also been thinking about having a 'for .. in ..'  syntax to express them.
Its nice that C11 supports them out of the box, but it probably makes sense to generate C++98 as you said.
In terms of writing the grammar - there is really not much difference between having a 'for' or 'foreach' to indicate a ranged loop, but I would really like to get away with just using 'for' (it currently also replaces 'do' and 'while' loops)
PS I'd be interested to see your example as well if you don't mind?
I just mentioned that it was translatable to C++98, however I do not recommend it, no need to go backwards. The newer compiler has far more optimisations (and the changes in the standard allow for more) which are going to help, especially with generated code.

In the newer IDE you can make your own toolchain (just copy and modify the AVR one). Then you can run your own compilation recipe and run your process and pass the finished C++ source directly to the compiler. No need for a new IDE, just add in your own features that people can download from the board manager.

I can upload my lib to GitHub for you later. For now there is a few versions of forward iterators here (only forward & step of 1):

http://stackoverflow.com/questions/7185437/is-there-a-range-class-in-c11-for-use-with-range-based-for-loops (http://stackoverflow.com/questions/7185437/is-there-a-range-class-in-c11-for-use-with-range-based-for-loops)

I used these in the past before I went further and made my own with more functionality.


Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 20, 2015, 09:12 am
The newer compiler has far more optimisations (and the changes in the standard allow for more) which are going to help, especially with generated code.
I think you are right, I'll start looking more in that direction.

In the newer IDE you can make your own toolchain (just copy and modify the AVR one). Then you can run your own compilation recipe and run your process and pass the finished C++ source directly to the compiler. No need for a new IDE, just add in your own features that people can download from the board manager.
That's a good idea actually ( I did not know about the official IDE update when I started ), plus it will be quicker to do than finishing my own IDE since I already have the compiler binary. I'll look into it when I have time.

I would still like to release my own IDE as well though, because it allows me to add more 'UX' features that aid development.
Like real-time error checking, jumping to definitions, etc. It is just quicker to add these to my own IDE, rather then developing separate plugins for the official one; I might do that at a later stage.
The last feature that I was working on for the IDE is actually quite useful - its a side panel that, whenever you place the cursor anywhere in the code - shows you the complete scope ( all the variables that are 'visible' at that location ). It also sorts them by scope 'depth' and allows jumping to the locations where each variable was declared which can be really helpful.

By the way, please don't be mislead by the link I posted above (http://webcloudtools.com/sprk/lang-test/ (http://webcloudtools.com/sprk/lang-test/)) - this is not the actual IDE that I am working on, its just a quick way to review some of the language features.

I will be releasing the IDE once the language is more complete (most likely after new year)


PS thanks for the link, would be good if you could upload yours later, but I think I got the general idea.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 20, 2015, 11:01 am
@YemSalat, please don't force people to use the latest Arduino IDE. I mostly use 1.5.6 but I have some stuff that I need to use 1.0.6 for - I have not yet figured why.

Also, be aware that the Arduino development folks don't seem to bother at all with backwards compatibility.

...R
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Dec 20, 2015, 11:30 am
@YemSalat, please don't force people to use the latest Arduino IDE. I mostly use 1.5.6 but I have some stuff that I need to use 1.0.6 for - I have not yet figured why.

Also, be aware that the Arduino development folks don't seem to bother at all with backwards compatibility.

...R
Unfortunately every new board will require it and there is little reason to stay in the dark ages. That compiler is over 10 years old.

What is the problem? It can't be made compatible with the older IDE's. You will have to be willing to copy the generated code into the IDE and compile it... with every change... yeah right!

You'll still need another IDE to be remotely useful, so it might as well as be one that (eventually) most will have.

On a side note: If you have a problem with the new IDE, start a thread and we can help you knuckle out a solution...

@YemSalatOn the contrary to robins argument if you force people to use a separate IDE, you are going to significantly reduce the number of people that can/will use your language. Think schools/universities that are quite selective in what they install on their machines.

Hanging onto the past is why internet technologies are so bloated like writing CSS. Adobe flash is still around because people refuse to stop using it (http://occupyflash.org/), while HTML5 makes it completely unnecessary.

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 20, 2015, 02:39 pm
Unfortunately every new board will require it and there is little reason to stay in the dark ages.

....SNIP....

On the contrary to robins argument if you force people to use a separate IDE,
I think we are at cross purposes.

I was only trying to say that @YemSalat's program should work with various IDEs - not that it should be tied to any particular one, either new or old.

...R
Title: Re: Designing a new programming language for Arduino
Post by: pYro_65 on Dec 20, 2015, 02:55 pm
I think we are at cross purposes.

I was only trying to say that @YemSalat's program should work with various IDEs - not that it should be tied to any particular one, either new or old.

...R
Yeah I know what you're putting forward, however the old IDE isn't compatible, plain and simple. Whereas the new one is. It is not practical to copy source into the old IDE every time you need to compile... might as well use C++ which it supports directly. And requiring a new (non-arduino) IDE is not going to be as beneficial especially if it is an Arduino tool!

1.6.x has added the capability to fully control how the source is used, and its great I can use the ESP8266, Zero, and next year we will probably have the PINE64 using it.

You can run two different versions of the IDE if you do not want to upgrade your code that does not work, but either way, YemSalat's compiler chain will not work with old versions of the IDE; so it might as well use the latest and greatest (regardless of whether the code is C++98 or C++11 compatible).

Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 20, 2015, 05:13 pm
It is not practical to copy source into the old IDE every time you need to compile...

...SNIP...

but either way, YemSalat's compiler chain will not work with old versions of the IDE; so it might as well use the latest and greatest
I had not realized there was any need to copy source code ... so maybe I have not quite got my head around how the new system will work.

There may still be the problem that it won't be compatible with a future version of the IDE.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 20, 2015, 09:11 pm
We might be getting a bit confused here.

Here are all the options that I currently see for uploading the new language to Arduino:

1) Adding a custom mode to the board manager (this is the option that pYro_65 has suggested, it is only supported on newer versions of the IDE) - this is the simplest and one of the most effective. All I need to do is add one more additional step to the Arduino toolchain and after that users will be able to just select the language from the board manager - and type it directly in the Arduino IDE. (You should also be able to use any custom editor with the official IDE)

2) Using my own IDE. Since version 1.5.5 - Arduino has command line tools (I am not sure how well they are  supported though) - I can run an agent that can 'talk' to my IDE and then use Arduino CLI to compile and upload the sketch.

3) Using the binary compiler of the language and the CLI tools. One could just use any custom editor and set up something like a makefile for compiling and uploading sketches.

4) Using Arduino IDE just as an uploading tool. So basically copy-pasting, even though it is not an option for 'professional' use, I think for many people (especially beginners) - copy-pasting can be a simpler option then setting up board managers, etc. Once they are more comfortable with the tools - they can switch to using other options.

Overall I think it is better to move towards the newer versions, however I would still like to do testing in the older ones as well.

Quote
Also, be aware that the Arduino development folks don't seem to bother at all with backwards compatibility.
Do you mean the IDE versions, or the libraries?
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 20, 2015, 10:50 pm
4) Using Arduino IDE just as an uploading tool.
I use the Geany editor to create my code and just use the IDE to compile and upload. I did figure out how I could call the IDE from Geany but I don't actually do it that way - mainly because it was too much trouble to incorporate all the options that the IDE offers.

I guess, without really thinking hard, I had assumed your system would work in much the same way - i.e produce code that the IDE would consume.

There is no way I could use the IDE as an editor ... UGH

Also, I suspect it would be less confusing for a newcomer to use our editor for your language and use the Arduino editor for C/C++. But I can see that this is very much an area of personal preference.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 22, 2015, 11:41 am
I use the Geany editor to create my code and just use the IDE to compile and upload. I did figure out how I could call the IDE from Geany but I don't actually do it that way - mainly because it was too much trouble to incorporate all the options that the IDE offers.
I coded all my Arduino projects the same way so far, only used the IDE as compilation/uploading tool. (I use Sublime as my editor most of the time)

Quote
I guess, without really thinking hard, I had assumed your system would work in much the same way - i.e produce code that the IDE would consume.
That was the original idea, however pYro_65 brings up a good point that apart from this method - I could also add my own compilation recipe to the board manager, so people who do want to write code directly in the Arduino IDE - will be able to do so.


Quick update:
I updated the testing environment a bit: http://webcloudtools.com/sprk/lang-test/#hello-world (http://webcloudtools.com/sprk/lang-test/#hello-world)
Nothing major, mostly refactored the default function parameters and updated the examples.
Added 'fancier' errors for number type overflows:
(http:////i.imgur.com/pLYdnHd.png)


I am now working on arrays implementation.
I need to decide which is a better notation for arrays.

- having the array limit next to array type:
int[10] foo = [0..9]

- or having the limit next to the identifier:
int foo[10] = [0..9]

The second way is the same as it is in C++, which is a strong point towards it.
However first way seems to work better with implicit types, so one could write:
int[10] foo = [0..9]
or
foo = [0..9] - to let the compiler count the array length for you.

It also seems to be the way it is done in most other strong-typed languages (C#, Java, Go, etc.)
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Dec 22, 2015, 02:17 pm
Quote
I need to decide which is a better notation for arrays.
That decision, as well as many others, should be influenced by what the purpose of the new programming language is. 

If it is to be stand alone and fully capable of doing anything that say C++ can do, just in a more logical and friendly way, then by all means implement the best and easiest syntax possible.  If, however, it is to be an introductory language that allows a subset of C++ functionality to be used with the aim of the user moving on to C++ when they reach a point that they need its functionality then surely it makes sense for the syntax to match C++ as far as possible whilst providing meaningful error messages and possibly context sensitive help, command completion, automatic insertion of brackets, braces, etc where they are used.

I for one am confused as to the purpose of the new language.

Go back and read your original post in this thread.  Have you strayed outside of your original objectives and maybe let feature creep start to occur ?
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 22, 2015, 02:50 pm
I suggested STOP in Reply #108.

I respectfully suggest that the ONLY work you should do NOW is on things that are ESSENTIAL before publishing the product so that people can try it out.

Don't worry about anything that "could be improved" or even about any shortcomings that don't actually prevent it working.

When you have published the first version and got some user feedback you can then think about what changes might justify producing a second version.

...R
Title: Re: Designing a new programming language for Arduino
Post by: YemSalat on Dec 23, 2015, 04:19 pm
I for one am confused as to the purpose of the new language.

Go back and read your original post in this thread.  Have you strayed outside of your original objectives and maybe let feature creep start to occur ?
I agree that I have deviated from some of the ideas that I layed out in the first post (I should probably update it)
Most approaches that are currently in the language have been chosen for reasons of better compatibility with exisitng Arduino environment.

I am hoping to avoid the 'feature creep' (with some help from the community at times :) )

I think its best to do what @Robin is suggesting now - I will finish the first version of the project (IDE + current version of the language w/o any more modifications) - then get some user feedback and decide what is the best direction to take the project next.

PS
I probably won't be posting for the next couple days, hope everybody has great holidays!
Title: Re: Designing a new programming language for Arduino
Post by: UKHeliBob on Dec 23, 2015, 06:24 pm
Quote
I agree that I have deviated from some of the ideas that I layed out in the first post (I should probably update it)
If you do decide to update it please do so by adding to it explaining why the objectives have changed rather than changing the stated objectives.
Title: Re: Designing a new programming language for Arduino
Post by: Robin2 on Dec 23, 2015, 09:09 pm
@YemSalat, one lesson that I have learned has been somewhat painful - and it happens every time. Whenever I display something to users they are ALWAYS interested in some aspect that I have not been bothered with. Most recently the handheld plastic box I put the demo model in was too big - they took the functionality for granted!

Getting user feedback is vital.

It is wise to be aware of the bicycle shed syndrome (https://en.wikipedia.org/wiki/Parkinson's_law_of_triviality).


...R