Designing a new programming language for Arduino

SurfingDude:
As to frosting, I think the conceptual bridge between the arduino world and the C/C++ world is something to keep. Many recipes exist to solve problems in in the semicolon-bounded world and I'd want to be able to copy and paste them into any new world, or take my arduino work and learning onto an industry platform.

At least syntax-wise the language is definitely going to resemble C/C++ as close as possible, exactly for the reasons you mentioned (compatibility with existing code and ability to transfer your projects 'back' to C if required)
I also want to keep the transpiled code close to the original (keep variable names, etc) and also format it in a nice and readable way.

What I am still 'on the fence' about though - are the C pointers, right now I have not implemented them in any way, and I am not a big fan of the original syntax (especially things like having the same '*' operator to declare and dereference a pointer) So if pointers will be kept in the language - they will almost certainly undergo some syntax changes. I know that pointers are super useful, especially on embedded systems, but I will need to think more about the best way to implement them before they appear in the language.

SurfingDude:
As to the cake part, the thing that we lack most are effective ways to deal with concurrency. This is holding back the use of multi-core hardware and leads step-by-step programs into problems of semaphores and deadlocks, especially now when we try to use interrupts to multitask.

If there is to be a new language I'd like it to focus on improving the cake. Maybe with objects that communicate messages via queues, but do computation in the way that is already known and accepted. The standard frosting looks just fine.

This is exactly what I have been thinking about myself. Better support for concurrency even on the level of language constructs would be a big win (note, when I say concurrency I don't mean threads)
The best idea I had so far, which would not require a complex runtime, is implementing a kind of an 'event loop' where objects would emit events and react to them. This is similar to message passing and would not change the code structure much.

Another idea I had - is being able to write asynchronous code in a synchronous way. For example, the delay construct could be limited to the current code block, so it does not stop other things from executing at the same time:

void [b]spinMotor[/b] () {
	setPin(9, HIGH)
	[b]delay[/b] [i]5s[/i] [color=gray]// <- will only delay the code inside the 'spinMotor' function [/color]
	setPin(9, LOW) [color=gray]// and won't block other code execution[/color]
}

But these are of course 'Big' ideas for the future. Right now I am focusing on making a good language for beginners that is easy to understand and building some helpful tools around it.

CrossRoads:
If you didn't have s setup() function, where would the stuff that only needs to occur once happen?

I didn't see the unsigned long type in there. Will that be coming to support the time elements?

Will bit manipulation be supported?

I decided that I will use setup() for what it was intended.

Sorry, I forgot to mention the unsigned types in the previous update, but I added the unsigned long and unsigned int in the form of: uint, ulong
All the time literals are assigned a 'ulong' type by default at the moment (this can be optimised in the future)

Regarding bit manipulation - yep, all C/C++ bitwise operations are supported (actually they are just mapped directly because the syntax is the same)

I will definitely publish a list of all the language features and entities that are supported once the demo is released.

[EDIT]
Sorry, I started writing this reply before the last 3 messages were posted.
For now I would like to focus on getting the minimal functionality into the language and make it a 'friendlier' version of C/C++ which is more geared towards Arduino development. However, I would definitely like to improve on some bits in the future versions (which does not mean dismissing the original concept). For example by providing an easier way to express what you want to do then you would in C/C++ (I already talked about this in my reply to @SurferDude's comment)
The best approach would of course be to keep the existing 'cake' untouched, but also add some additional flavours to it (and I don't mean 'overload' with features, we already have C++ for that)

I would also like to keep the UNO chips as the primary target at least for the nearest future (this is still by far the most popular version of the Arduino)

The more modern chips do provide some really cool possibilities, I think we can discuss this as time goes.

Yet another update.

Unfortunately I did not have time to work on the project in the beginning of the week.
So I was trying to cram everything into the past two days.. I really want to release the first version over this weekend, but as usual the most tricky bugs show up right before the demo :slight_smile:

Initially the IDE was just a prototype which I was planning to rewrite from scratch at some point.
But things did not go exactly as planned, so instead I was adding more and more code to the prototype.
I just HAD to refactor all of this, so I spent a good chunk of yesterday evening on that. Now all the code is much more tidy.

I also had a few cache issues with the IDE (those are the worst, aren;t they?) but I figured it out.

There are a couple things that I was planning to include in the first release, but will most likely have to postpone until future versions:

  • I had to remove the 'save' button from the IDE, instead everything is auto-saved. It is just quicker to implement it this way, because a 'save' button means that I would need to keep track of the file states (saved/unsaved)

  • I initially added syntax to support classes, so the standard 'library' could use them. But having classes means adding another step to the compilation, because class names can be used before they are declared, so I would need to first go through the code and collect all the class declarations. This step is inevitable though and I will be adding it at a later stage.
    For now all the standard built-in functions - are just 'functions', so you have print() and println() for output, pinRead(), pinWrite(), pinMode() for pins, etc. (I will post a full list upon release)

Overall I am doing final polishing before the release , just wanted to give a quick update.

[EDIT] added a couple things to the post, fixed some typos

everything is auto-saved.

I don't like that idea.

I often find myself making a stupid mess and it is great just to NOT save the file and re-load the original version.

I think if you must use auto-save it should have to a different file each time - with serially-numbered names. But that is very difficult with the Arduino IDE because of its requirement for the file name and directory name to match. I sometimes save a version with a name myProgram.ino.BK1 and the IDE just ignores it when it comes to compile time. (This is on a Linux PC).

...R

Robin2:
I think if you must use auto-save it should have to a different file each time - with serially-numbered names. But that is very difficult with the Arduino IDE because of its requirement for the file name and directory name to match. I sometimes save a version with a name myProgram.ino.BK1 and the IDE just ignores it when it comes to compile time. (This is on a Linux PC).

You will still be able to 'download' a latest copy of any file at any time.
What I mean by auto-saving is that there is no way to loose whatever changes you make to the currently open files.

Its a bit hard to explain, in the IDE you can create new 'files' in the sidebar. You can download a copy of any of those files at any time, both source and transpiled C++ code (this code is to be pasted into Arduino IDE). But the files within the IDE itself are always saved, you can still Ctrl+Z to undo last changes, but it does not ask you to save it when you close the file. Does this make sense? (sorry, I am not great at explaining things)

Also, another quick question.

Do you think I should name the pin control functions the same way as they are on Arduino?

Right now they are pinWrite, pinRead, PinMode, which are shorter to type and all start with 'pin', which sort of 'namespaces' them in a way.
On the other hand they could be named digitalWrite/Read - that way the code can be transferred between languages easier.

PS sorry about the delay so far, just can't get enough time to work on this, and don't want to showcase an incomplete project

YemSalat:
But the files within the IDE itself are always saved,

That is the bit I don't like. Personally I only want it to save the changes when I say so. I am happy to take responsibility for my own failures. But I don't want my perfectly good working code automatically overwritten by some garbage that I typed and then thought better of. I do realize, of course, that the code needs to be saved before it can be compiled. Maybe I am misunderstanding when your program saves the code?

Right now they are pinWrite, pinRead, PinMode, which are shorter to type and all start with 'pin', which sort of 'namespaces' them in a way.

I rather like this style. It would be easy to have a few macros in regular Arduino code to make the same commands be converted to digitalWrite() etc.

It is probably a typo - but you have an uppercase P for your PinMode - be consistent (I prefer to start with lowercase).

...R

Robin2:
That is the bit I don't like. Personally I only want it to save the changes when I say so. I am happy to take responsibility for my own failures. But I don't want my perfectly good working code automatically overwritten by some garbage that I typed and then thought better of. I do realize, of course, that the code needs to be saved before it can be compiled. Maybe I am misunderstanding when your program saves the code?

I think you got it right, and yes, as I have already discovered myself during testing that this can be a bit of a pain. (you can use Ctrl+Z to get the changes back, but if you reload the IDE - it won't help)
I wanted to auto-save everything just because it is quicker to implement it that way, but since I did not release the demo yet, I will probably just take a bit more time and do it properly.

Robin2:
I rather like this style. It would be easy to have a few macros in regular Arduino code to make the same commands be converted to digitalWrite() etc.
It is probably a typo - but you have an uppercase P for your PinMode - be consistent (I prefer to start with lowercase).

Oops, yep, that is definitely a lowercase. I was actually thinking of making it mandatory that all functions start with a lowercase ( and all class names start with uppercase ) and constants are all caps. I have not implemented this yet though, but I think it could improve the code style.

PS also, sorry again about missing the weekend release mark. This project obviously turned out longer then I initially planned (although I was not planning to make the IDE back then)

The compiler itself is complete (I blinked an LED for the first time with it yesterday :slight_smile:
Right now I am just polishing all the stuff around it, so it does not feel too 'raw' when showcased.
I also want to make the project open source so the code needs to be cleaned up a bit (that's what I spent most of my time on in the last few days)

YemSalat:
you can use Ctrl+Z to get the changes back,

Sorry if I am labouring this point too much ... but just to clarify further ... (and this may be just my own foolishness)

Sometimes I get an idea for a substantial reorganization of a piece of code and rush in to make changes. After 30 minutes or so I have got completely confused. At that stage Ctrl+Z isn't really any help - partly because it is a sequential process and I can no longer remember all the changes I have made, never mind the order in which I made them. In that situation starting over is often easiest - so dump all the changes and go back to the original version.

Don't worry about a few days delay. Although we might lose patience after 3 months :slight_smile:

...R

That sounds very familiar.

That sounds very familiar.

Robin2:
Sometimes I get an idea for a substantial reorganization of a piece of code and rush in to make changes. After 30 minutes or so I have got completely confused. At that stage Ctrl+Z isn't really any help - partly because it is a sequential process and I can no longer remember all the changes I have made, never mind the order in which I made them. In that situation starting over is often easiest - so dump all the changes and go back to the original version.

I don't think there is anything wrong with you, all developers I know are like that, that's why we use source control :slight_smile:
This actually makes me think if it would be a good idea to add some sort of "Versioning" to the files, something that allows you to save the current version of the file - and then get back to it at any time. There can be multiple versions to one file. (This definitely won't be in the first release, but something to think about for the future)

Don't worry about a few days delay. Although we might lose patience after 3 months :slight_smile:

We are at 3 weeks and one day mark right now. (initially I was planing to be done in one week, hehe)
The first version of compiler is already done, now its all about making the IDE usable.

I confess (but don't tell anyone else) that I do (usually) make backup copies of programs before embarking on major changes but I have never managed to get my head around formal version control systems.

I can see how they would be mandatory in a multi-programmer environment, but even then I have never found a clear explanation of how they actually work to resolve conflicts between a change made by A and a different change to the same thing made by B. Or how you could take 75% of A's changes and 25% of B's changes.

I cut my teeth on a small DEC PDP/11 and on one of their operating systems it automatically saved files with successive version numbers. I recall attending a tutorial one day where I enjoyed watching the screen behind the tutor gradually creating so many backup copies that it would have run out of disk space if I had not drawn attention to it at the 11th hour. (Probably a 10MB disk).

...R

on one of their operating systems it automatically saved files with successive version numbers.

Sounds like VMS

Robin2:
I can see how they would be mandatory in a multi-programmer environment, but even then I have never found a clear explanation of how they actually work to resolve conflicts between a change made by A and a different change to the same thing made by B. Or how you could take 75% of A's changes and 25% of B's changes.

Well, the exact way of how merging is done will of course depend on which version control system you are using.
I personally use git, and its real power comes with branches. For example you could have a separate branch for every feature in your project, and withing those branches will be separate commits. So when you merge two branches - you can merge only individual commits and not the entire branch.

Version control definitely makes more sense when multiple developers are involved, but I still find it quite useful when I work by myself on my own projects. It gives the project some 'order', and it is nice to be able to view its entire history and go back to any point.
For example when I work on my own - I will have the master branch with the stable version of the project and I will also have a few branches where I am developing new features - once the feature is complete and tested - I merge it into the master branch. This way all new features have a separate history and I don't have to remind myself what I was doing when I last looked at the code. ( Having a backup of all my work on github is pretty cool too )

Robin2:
I cut my teeth on a small DEC PDP/11 and on one of their operating systems it automatically saved files with successive version numbers. I recall attending a tutorial one day where I enjoyed watching the screen behind the tutor gradually creating so many backup copies that it would have run out of disk space if I had not drawn attention to it at the 11th hour. (Probably a 10MB disk).

That sounds a bit crazy to be honest ) I would definitely require the user action to create a new version of the file, and not do it automatically (so basically its like creating a separate 'backup' file)
In order to save space - it can store just the difference between new version and the previous one instead of saving the entire content of the files.

YemSalat:
I personally use git,

I know this is off-topic, but are you aware of any "tutorial" that explains the way to use Git. I could make no sense of their own website. It explains how to use the commands, but not the underlying concept.

...R

Regarding Git, SparkFun has a pretty good tutorial: Using GitHub - SparkFun Learn

Git/Github free online class: Version Control with Git | Free Courses | Udacity
It includes quite a bit of introduction to the basic principles of revision control, and it's pretty good. The "exercises" are well matched to the lecture material - a weak point in some other online classes I've taken :frowning:

For single-person projects, I've been using "rcs", one of the granddaddies of this class of programs. I've used it "professionally", and it seems to be sufficient as long as only one person is modifying a particular file at one time.
"cvs" (which I've also used professionally) adds more complex "merge" capabilities and is useful for several people sharing the same workspace and sometimes colliding with each other. The newer programs (svn, git, mercurial) add network-shared code repositories and are useful with bigger and more dispersed developer communities. (there are other differences as well; slight tweaking of philosophies and such. But that's how I think of things.)

VMS (on vax) and TOPS20 (on PDP10) had "file versions" that could be used; very handy (and I'm surprised that none of the PC operating systems have done anything similar.) There were settings for controlling the maximum number of versions of a file to keep. Sigh. One of the "lost capabilities" in the transition from mainframes to micros.

(RSX-11 apparently also had a versioning file system. Versioning file system - Wikipedia)

Thanks for the links Guys. I have bookmarked this for follow-up reading over the next few days.

...R

Just to add to the previous comments, I gotta say that I personally had to learn git the 'hard way', I got hired in a company where they were already using git. To be honest it took me some time to completely understand what's going on (I was mostly using svn before), but now I don't think there is a better source control system at the moment.

I just had a quick look around on google, and this tutorial seems to be pretty straigth to the point: git - the simple guide - no deep shit! (title says it all hehe)

Overall, if you want to just get started quickly - cd to your project directory (as far as I understood you are on linux)

  1. Run 'git init' to initialize the repository.
  2. Then run 'git add --all' - adds all uncomitted changes to index
  3. Finally run 'git commit -m "COMMIT MESSAGE"'

Then whenever you make changes that you want to save - just run #2 and #3
Commit messages are usually things like "Implemented feature X" or "Closed bug #777", etc.

I would also use some visual tool to view the commit history (like DAG or git cola) - it can give you a good visual representation of the repository.

Now to bring it a bit more on topic :slight_smile:

Here is a brief intro into the language syntax
I will definitely publish a more comprehensive documentation after the release, but I thought it would be good to give a quick summary of all the features that are currently implemented.

Variable initialization - inferred types, (also - optional semicolons)

foo = 10 [color=gray]// int[/color]
bar = "Hello" [color=gray]// str[/color]

Parenthesis in if and for statements are optional, but the curly brackets are mandatory.

if a < 5 { ... }

Alias keywords for common operators - and, or, xor

if bar < 500 [b]and[/b] foo > 100 { ... }

I also added alias for equality operators - is and is not (convert to - '==' and '!=')

if reading [b]is not[/b] 0 { ... }
if ledPin [b]is[/b] HIGH { ... }

Loops - there are only for loops in the language.
They can take just the testing condition:
for i < 10 { ... }

Otherwise they work the same way as regular for loops:

for i = 0; i < 10; i++ { ... }

You can use break and continue statement,
there is also an alias for 'continue' - skip

As I mentioned before - functions can have docstrings:

[color=gray]/// This is the function description[/color]
int readPin ( int number ) { ... }

There are aliases for the built in functions:
println() - Serial.println()
pinRead() - digitalRead()
now() - millis()

(Please let me know in case I forgot anything)

Also, here is a listing of all errors that are currently caught by the compiler:
(Sorry about the format, it is pretty much straight from the source)

    VARIABLE_STATEMENT    
      type_mismatch       'variable "{0}" does not match type "{1}"',
      already_exists      'variable "{0}" was already initialized as {1} on line {2}',
      does_not_exist      'variable "{0}" is not defined'

    VARIABLE_DECLARATOR   
      already_initialized 'variable "{0}" was already initialized as {1} on line {2}',
      already_declared    'variable "{0}" was already declared on line {1}'

    FUNCTION_DECLARATION  
      already_exists      'function "{0}" was already declared on line {1}',
      no_return           'function "{0}" must return {1}',

    PARAM_DECLARATOR      
      duplicate_param     'duplicate parameter "{0}"'

    RETURN_STATEMENT      
      return_outside      'return statement outside function declaration',
      type_mismatch       'return statement type does not match function "{0}"'

    IDENTIFIER            
      does_not_exist      'variable "{0}" does not exist'

    BINARY_EXPRESSION     
      type_mismatch       '"{0}" operation types do not match, can\'t {operator: 0} "{1}" and "{2}"'

    LOGICAL_EXPRESSION    
      type_mismatch       'can\'t compare "{0}" and "{1}"',
      cant_compare        'can\'t compare "{0}" and "{1}"',
      not_a_number        'can use "{0}" operator only on numbers'

    ASSIGNMENT_STATEMENT  
      type_mismatch       'assignment types mismatch, can\'t assign "{0}" to "{1}"'

    UPDATE_EXPRESSION     
      not_a_number        'expression is not a number'

    UNARY_EXPRESSION      
      not_a_bool          'unary expression must be a boolean'

    IF_STATEMENT          
      not_a_bool          'IF condition must be a boolean'

    FOR_STATEMENT         
      not_a_bool          'FOR test expression must be a boolean'

    BREAK_STATEMENT       
      outside_loop        'BREAK outside FOR loop'

    CONTINUE_STATEMENT    
      outside_loop        '{loopStatement: 0} outside FOR loop'

So you haven't got to the point of having an error message for the original problem: unmatched parentheses.

This is not a hard problem. It's unsolvable. No imaginable quantum-computing AI can solve this. A god who can examine the mind of the programmer might be able to do it. How many times have you seen a newbie post some code here (after being told to use code tags) and the answer is "We think you meant to put a closing brace but that may not do what you wanted. What did you want it to do?"

The thing I love about the Arduino is that it is approachable for high school and even primary students but it's not a walled garden. If you absolutely must put in some clever code that saves one clock cycle in an ISR, then you can do it and you haven't broken the walls of the garden so that the simple delay() function stops working. You can write 99% of your code with standard Arduino functions and your main program logic will be easy to understand 6 months from now when you have to look at your own code again and ask "Who wrote this crap?"

Delays that only delay inside one code block? We already have that on the Due with the Scheduler library. It's a standard part of Arduino. It uses more memory to maintain different contexts and time is lost switching, which is why it's not used on the smaller Arduinos.

The classic computer-science test for a new language is "Can you write its own compiler in the language?" I think that's a big hurdle and probably not appropriate here. I would ask, "Can I write a library for a sensor in this new language?" Can I take an accelerometer or CO2 sensor or whatever and write some code in this new language which will enable it to be used as simply as a readPin() function? If I have to write the library code in C++ and then switch to this different language to write the "real program" I'm working on, then you've lost. I'm not going to switch languages in the middle of a project.

YemSalat:
Parenthesis in if and for statements are optional, but the curly brackets are mandatory.

I suspect it would be less confusing if the parenthesis are also mandatory

MorganS:
If I have to write the library code in C++ and then switch to this different language to write the "real program" I'm working on, then you've lost. I'm not going to switch languages in the middle of a project.

I don't think this is the intended market for @YemSalat's system.

...R