Release the mcu nodes to npmjs?

[colinl]

@phoddie have you considered releasing the mcu node to npmjs?

The current system has some problems, particularly when used with node-red-mcu-plugin.
Firstly it is a bit tortuous to install the node as one has to do
npm install node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu/nodes/mcu
but more serious is the fact that, after installing the mcu node, if one uninstalls node-red-mcu-plugin then npm gets in a state where nothing can be done, due to the fact that, in package.json, in the dependencies, is
"@moddable-node-red/mcu": "file:node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu/nodes/mcu",
but that file no longer exists.
If one tries to do anything with npm then it errors with
npm ERR! Cannot set properties of null (setting 'peer')
and the only way that I have found to recover from this is to manually remove the offending line from package.json

[bartbutenaers]

Hi @colinl,

That would indeed be very useful.

1. I have to install new versions of Ralph's sidebar plugin quite often, and every time it indeed fails (like two minutes ago again...):

   

   And then indeed I just need to remove that single line:

   

   And then I can install Ralph's sidebar without problem.

2. And like you say, installing the node via npm install node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu/nodes/mcu is completely different to how all the other nodes work. Would be very handy if these nodes could be installed the same way as normal nodes. That would be a great step for convincing other users to start experimenting with this...

[colinl]

@bartbutenaers it can be made a little easier for the moment by making a backup copy of package.json, re-installing the plugin, restoring the backup package.json and then running npm install, which will install the mcu node again.

[phoddie]

@phoddie have you considered releasing the mcu node to npmjs?

@colinl – the title of this issue refers to "mcu nodes" (plural) by which I understand you to mean things like the MCU Sensor, MCU Clock, I2C, etc nodes that are part of this (node-red-mcu) repository. But the first sentence says "mcu node" (singular) followed by some discussion about the MCU Plug-in path. As a result, I'm a little confused. Would you help?

[colinl]

Yes, sorry, too much overloading of the word node. There is a tendency in the node red universe to use the word node for what are, more correctly, npm packages.
So what I am talking about is anything that is installed using npm install. If the nodes contained within the nodes/mcu set were published to npm as a package, then instead of using
npm install node_modules/@ralphwetzel/node-red-mcu-plugin/node-red-mcu/nodes/mcu
one would just do something like
npm install node-red-mcu-mcu
though that naming does not seem ideal.
I even wonder whether node-red-mcu could itself be made available as an npm package. Currently when it is installed by installing the plugin there is no way to control which release of node-red-mcu one gets. Maybe there are already plans to address that, but if not then it is something that should be thought about, as it will become an issue at some point in the future.

[Edit] Actually the install would more likely be something like
npm install @node-red-mcu/mcu

[phoddie]

Thanks for the clarification. Now I understand. Please keep in mind that the current state of things is one step on a journey. As I noted in my presentation at Node-RED Con:

What you describe would indeed address the immediate issue that you and @bartbutenaers describe. The direction I'm pursuing is to allow users to install the nodes for the MCU from the usual Node-RED palette without having to use npm directly. That will, of course, require having the nodes available through npm.

Those MCU nodes could be part of the core suite provided by Node-RED MCU Edition itself or contributed nodes.

[colinl]

Please keep in mind that the current state of things is one step on a journey.

Fully understood. Provided the plan is for it to happen eventually. I am sure we can live with it as it is for the moment. The most annoying problem is that of how to update the plugin without npm getting its knickers in a twist.

[colinl]

@bartbutenaers (and anyone else interested) I have found a workflow that allows easy updating of the plugin without falling down the npm black hole.

• Clone node-red-mcu to a convenient folder.
• git clone https://github.com/phoddie/node-red-mcu
• npm remove the mcu nodes. To do that, from the .node-red folder run
• npm remove @moddable-node-red/mcu
• Install it from the clone folder instead, so in my case
• npm install ~/node-red-mcu/nodes/mcu
• Now the plugin can safely be upgraded. To pick up the latest mcu nodes go to the clone folder and run
• git pull
• and restart node-red, which will pick up the new version.

[bartbutenaers]

@colinl,
Thanks for sharing your workaround.
That indeed might be a good alternative for us to renew frequently our nodes, withouth fighting with npm.

But of course it would be very useful if - one day in the future - these node(s) could simply be installed via the Node-RED palette menu, similar to all other nodes. To allow less technically skilled people to use this, from within their flow editor.

Until then I will use your workaround ;-)

[phoddie]

But of course it would be very useful if - one day in the future - these node(s) could simply be installed via the Node-RED palette menu, similar to all other nodes. To allow less technically skilled people to use this, from within their flow editor.

That's the goal. Perhaps this is a good opportunity for some experienced Node-RED developers to help this Node-RED rookie out. ;) At this point we should have all the pieces together to achieve the goal. Perhaps we could start with the MCU node suite. It already even has a package.json to help out the MCU plugin.

[colinl]

I think all you need do is to add a node-red version spec to package.json, so it reads something like

"node-red"     : {
    "version": ">=3.0.0",
    "nodes": {
			"mcu_analog": "analog/mcu_analog.js",
			"mcu_clock": "clock/mcu_clock.js",
			"mcu_digital": "digital/mcu_digital.js",
			"mcu_i2c": "i2c/mcu_i2c.js",
			"mcu_neopixels": "neopixels/mcu_neopixels.js",
			"mcu_pulsecount": "pulsecount/mcu_pulsecount.js",
			"mcu_pulsewidth": "pulsewidth/mcu_pulsewidth.js",
			"mcu_pwm": "pwm/mcu_pwm.js",
			"mcu_sensor": "sensor/mcu_sensor.js"
    }
}

If 3.0.0 is what is needed of course.
Then it should be possible to publish it to npmjs by going into the mcu folder and running
npm publish ./

Do you want to publish the package version at 1.0.0? The convention is to use 0.x.x. until it is stable.

It is good practice, though, to have a git release at the same version as the node, so that release can be checked out to get the code of that npm version, and change logs can be used to record the changes. That works better if the package is in a git repo of its own, rather than buried inside a bigger one.

[bartbutenaers]

@phoddie,
Good question.

• In the left search bar: I thought that you can search on the node name, or your node label: see this pull-request.

• In the palette manager install menu on the upper right top corner: I thought that some time ago they had expanded the search capabilities to also be able to search on the keywords specified in your package.json file. Can't find the pull request anymore that implemented that, so I might be make a wrong assumption. But I see you don't have specified any keywords in your json. You can find an example here in the current node I am working on. It is very weird that it can find your node, because the node-red keyword is mandatory (see documentation). There were many discussions in the past on discourse about people not being able to find their node, and the solution was always: add the 'node-red' keyword. However perhaps that was because Node-RED was automatically scanning new packages for that keyword, and then they were automatically available in the palette. However since that failed a lot (due to problems in the npm cluster), they have stopped automatically searching and now you have to add your node and its updates manually on flows.node-red.org (so therefore that keyword is not mandatory anymore, but the docs perhaps obsolete...)

[bartbutenaers]

However if I type "coral" it doesn't find my node, although it is in my keyword list:

Unless that search feature was added to a recent Node-RED version. I am testing on Node-RED v3.0.2
Perhaps @colinl can try, if he has a recent version. Not for my node of course because that is not on npm yet, but any other node.

[ralphwetzel]

Annoyingly, if I type "mcu restart" or "restart mcu" it doesn't find it. Is there some keyword magic I should be doing?

The reason for this might be that the search algorithm used is just not as fancy as we are (meanwhile) accustomed to. I haven't checked the source code, but it looks like a standard RegEx comparison. Thus I have the impression, there's nothing you can do to create magic ... in this case!

[bartbutenaers]

Ah this so much more user friendly. Since I had already installed your node previously, I now can automatically update it:

First remark: The info panel is rather short:

I assume you just had a lack of free time to implement this. Just sharing some ideas: it is useful to add at least what the input message should look like. Because otherwise users have to start digging on Github, to search for how to use your node. Which again increases the threshold for a lot of users to get started with this...

Don't know if you have seen their guide about this? As you can see they added some time ago the ability to use markdown syntax. I have never used it, because most of my nodes were developed before that. But for you it might be usefull, because then you perhaps can reuse it perhaps in your readme pages on github to lower the maintenance work of your nodes. I assume it is what you prefer most. Don't know...

[phoddie]

@bartbutenaers and @ralphwetzel, thank you for the notes. Very helpful.

I noticed that my keywords were sparse. I added a few more. Perhaps it will help, if not now, in the future.

The restart node is very simple so there's not much to document. But, in the spirit of my own learning and trying to be helpful to developers, I added some more text. I switched over to using the same markdown in both the readme and Node-RED help. That is certainly more convenient.

The formatting of the Inputs, unfortunately, isn't correct when rendered by GitHub or the Node-RED Library page. Maybe that is expected?

[phoddie]

The MCU nodes have been published to npmjs! You can install them into Node-RED in the usual way:

cd ~/.nodered
npm install @moddable-node-red/mcu

All the MCU nodes now set the moddable_manifest property to be explicit about where to locate their implementation.

There's work to do on the documentation in order to be ready to publish to the Node-RED library. I hope to finish that up this week.

Please give this a try and share how it goes. Please make sure you are running the latest Moddable SDK (including rebuilt tools) and Node-RED MCU Edition. Thank you.

[mtoko]

Thanks for this
The flow with the example OTA and restart nodes now builds without errors and the flow is running in the MCU. I can see MQTT messages received in another Node Red
A question first.
If I am using the plugin in the Node Red editor and I now want to change my flow and create a new bin file that can be uploaded OTA then the build will not fail because there is no device connected to the USB port. I have tried this with no MCU on the USB and a new bin file is created.
I have used the example curl file in the example and only changed the ip address of my MCU but get the following error. I have checked that this MCU address is reachable on the network and port 80 is open
darryl@darryl-Zorin:/.node-red$ cd
darryl@darryl-Zorin:$ cd $MODDABLE
darryl@darryl-Zorin:/Projects/moddable$ cd build
darryl@darryl-Zorin:/Projects/moddable/build$ cd bin
darryl@darryl-Zorin:/Projects/moddable/build/bin$ cd esp32
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32$ cd nodemcu
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32/nodemcu$ cd debug
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32/nodemcu/debug$ cd e3bbkcwfqq
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32/nodemcu/debug/e3bbkcwfqq$ ls
bootloader.bin ota_data_initial.bin partition-table.bin xs_esp32.a xs_esp32.bin xs_esp32.elf xs_esp32.map xs_esp32.sym
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32/nodemcu/debug/e3bbkcwfqq$ curl http://192.168.1.144/ota/firmware -X PUT -H "Content-Type: application/octet-stream" --data-binary '@xs_esp32.bin'
curl: (28) Failed to connect to 192.168.1.144 port 80: Connection timed ou

[phoddie]

I have a couple of questions:

1. How did you confirm that port 80 is open on the the MCU
2. Did you set the Push Endpoint in the node editor (shown below)

[mtoko]

Thank you for your advice

1.) I have a network scanner which scans the network and shows which ports are open on all connected devices
2.) The Push Endpoint is correct

Yay! it is working
When I carefully checked the network the MCU seemed to be going offline after a period of time which could have resulted in the error. I restarted EVERYTHING (network etc) and the OTA update worked. Just to prove that this was really working I disconnected the MCU from the USB port and powered from another source. I then changed the flow to add a second MQTT out with a new topic, created a new bin file and then did an OTA update. This worked and I could observe both MQTT messages arriving at another instance of Node Red.
I think the challenge now is to document this so the normal users (like me), can use this and be successful before their heads explode.
Thank you for all the hard work

[phoddie]

Yay! it is working

Woohoo!

Thanks for double checking everything. Great to have all the pieces working together.

I think the challenge now is to document this so the normal users (like me), can use this and be successful

Agreed. I think we've worked through a bunch of issues recently that help:

• It works. ;)
• OTA Update & MCU Restart node can be installed within Node-RED
• No manual manifest editing required to add OTA Update & MCU Restart nodes to Moddable build manifest
• ESP32 flash automatically partitioned to support OTA
• Extensive documentation on the OTA Update node itself (thanks to encouragement from @bartbutenaers!)

The workflow of building and applying updates happens outside the flow on the MCU. There are many options for how to do that, but we should document a common, simple approach. It seems possible that there might be some light updates to the MCU Plugin to help there too, since is the front-end for many developers using Node-RED MCU Edition.

[mtoko]

So, just to clarify the procedure that worked for me.

I updated everything
1.) Moddable sdk and Node Red MCU
2.) I created a first flow with the OTA and restart nodes from the examples
3.) I built this with the Node Red MCU plugin using the Node Red editor and it was downloaded to the ESP32 through the connected usb port. This only needs to happen once to get the OTA and restart nodes into the ESP32
4.) I then disconnected the ESP 32 and just powered this from a dedicated supply but on the same network
5.) I modified the flow and built again to create a new bin file
6.) I navigated to the directory that contained the new bin file, for me this was
darryl@darryl-Zorin:/Projects/moddable/build/bin/esp32/nodemcu/debug/e3bbkcwfqq$
7.) I then ran the example curl command in this directory
curl http://192.168.1.144/ota/firmware -X PUT -H "Content-Type: application/octet-stream" --data-binary '@xs_esp32.bin'

The flow was updated OTA

Easy!

[bartbutenaers]

@mtoko:
Thanks a lot for the detailed instructions!

Question about the last step (i.e. step 7): did you find perhaps any "easy" way to get the ip address (or mac address) of your MCU? I have quite a large list in my DHCP server, and unfortunately it is not always very clear which ip address corresponds to which device...

Don't see an (ip or mac) address in the console log of my sidebar. Thought that perhaps an address would be displayed, e.g. when connecting via xsbug...
Bart

[phoddie]

@bartbutenaers – when connecting with both Wi-Fi and Ethernet, the IP address is traced to the debug console at start-up.

Looking a little bit longer term, I really like mDNS as a solution here. A device can be assigned a name (pump1 and accessed through that pump1.local, no matter what IP address is assigned. I looked into supporting that in Node-RED MCU Edition a while back, but didn't find an mDNS node that looked like a good model. Do you have any experience with mDNS in Node-RED?

[bartbutenaers]

@phoddie,

when connecting with both Wi-Fi and Ethernet, the IP address is traced to the debug console at start-up.

Aaah, I had forgotten to power off/on my wt32-eth01 after starting xsbug-log. Done it some many times, and I forget it every time again...

And then indeed yes, there it is:

>> IDF_PYTHON_ENV_PATH: /home/pi/.espressif/python_env/idf4.4_py3.9_env
>> mcconfig -d -x localhost:5004 -l -m -p esp32/wt32_eth01 -t xsbug
mcconfig: ESP32 factory app partition divided into OTA partitions of size 0x1e0000 and 0x1fe000
mcconfig: file storage partition of size 0x10000 created from factory app partition
/home/pi/esp-idf/tools/idf.py
# starting xsbug
#xsbug-log listening on port 5002. ^C to exit.
#xsbug-log connected to "main"
Skipping Ethernet setup because Wi-Fi SSID is configured.
Wi-Fi connected to "xxxx"
IP address 192.168.1.31
...

Thanks for you patience, and sorry for the noise!!

[bartbutenaers]

@phoddie,
Using curl I got the ota update running. This a very nice feature!!!
I also like the progress indicator in the status of the Ota-update node.

I also tried to use a http-request node (instead of curl) however that failed. Since the http-request node doesn't give me any feedback about what is going wrong, I have no clue at all what is incorrect in my input. Unfortunately I don't have time to start debugging the http-request node, so I will just use curl for this.

Keep up the good work!!

[phoddie]

@bartbutenaers – that's great! Thanks for taking the time to try it out.

I also like the progress indicator in the status of the Ota-update node.

Glad you noticed that detail. I'm slowly learning what features are expected in a node's implementation. ;)

Since the http-request node doesn't give me any feedback about what is going wrong, I have no clue at all what is incorrect in my input.

If you have a test flow to share, I could take a look. It could be configuration of the HTTP Request node or mismatched expectations with what the OTA Update node wants. Either way, I'll learn something. ;)

[phoddie]

A quick update on the effort to the MCU node into npm.

There was a delay because of a problem with the help text. In the reorganization to support the MCU plugin, the help and icons for most nodes was broken. It took a while to figure out why (some subtle behavior of the Node-RED Editor), but that's now taken care of. I've migrated the help to markdown to make it easier to maintain and get the formatting consistent. I also updated all the nodes' help text. The last step is to update the readme, which is a bit dated and focused on the Sensor node That should be quick.

I know from experience that help text always invites feedback. Now that it is in markdown, it is much easier to update so please feel free to propose improvements.

[phoddie]

Done!