Moddable SDK manifests in flows

[phoddie]

I've implemented the ability for any Node-RED node to include a Moddable SDK manifest in its exported flow configuration. The primarily motivation is to allow the MCU Sensor and MCU Clock nodes to automatically add the required Sensor or Clock driver to the project. Prior to these changes, developers had to manually edit manifest files to add the drivers. That's been a little confusing and a bit of a headache.

The database used by the MCU Clock and MCU Sensor nodes has contained the driver path and the Clock and Sensor nodes have displayed the driver path. That was in anticipation of making this automatic.

The way the feature works is that that a node includes a moddable_manifest property with the manifest it requires. For the sensor node, this is just an include:

{
    "id": "39f01371482a60fb",
    "type": "sensor",
    "z": "fd7d965ef27a87e2",
    "moddable_manifest": {
        "include": [
            "$(MODDABLE)/modules/drivers/sensors/tmp117/manifest.json"
        ]
    }
}

The Moddable SDK build tool (mcconfig) processes all the manifests embedded in the Node-RED flows.json as part of a build.

Thanks to @ralphwetzel for suggesting the use of an unambiguously unique property name for this. That allows it to work with any node.

This feature has quite a bit of power. The MCU Clock and MCU Sensor nodes just scratch the surface of what is possible. I hope to have another example of what is possible in the coming days.

This property may also be a good way for a Node-RED contributed node to signal that it is compatible with Node-RED MCU Edition. The presence of modadble_manifest can be considered an indication that the author intends for it to be used with the MCU, and also allows the author to explicitly indicate which source files should be included, module specifies to use, etc.

To try this out you need to update just-about-everything:

• Get the latest Moddable SDK. Rebuild tools.
• Get the latest Node-RED MCU Edition
• Reinstall the MCU node suite into Node-RED
• Restart Node-RED

Note: If you have MCU Sensor or Clock nodes created BEFORE these changes, they won't include the new moddable_manifest property (obviously!). There are two solutions for that: delete and re-create the nodes or open the nodes in the Node-RED Editor and edit a property to force a re-save. You can use the Node-RED Editor Export feature to confirm that the exported JSON for the node contains the moddable_manifest property.

If you have a chance to try this out, please let me know how it goes. I'm interested to hear that it is working for you and happy to help try to resolve any issues that come up.

[colinl]

For those who don't know how to add the property into the node's properties (me for example), in the node's html file, in the defaults section, add

            moddable_manifest: {
              value: {
                include: [
                  "path/to/manifest.json"
                ]
              }
            }

So I surmounted that obstacle, and checked it was looking for the manifest by not providing one, and I get a build error saying it can't find it.
I assume I should add the manifest file into the node's source folder, but I don't know what path to use in the property. I tried ./manifest.json but that didn't find it.

[phoddie]

@colinl – I thought this might interest you. Thanks for taking a look. What you show is correct for providing a default manifest. I believe the problem you are bumping into is one reason the the docs note:

Nodes providing manifests should not use relative paths because the root for the relative path is the location of the flows.json file, which is not consistent

It does work with an absolute path to a local file:

moddable_manifest: {
	value: {
		include: [
			"/Users/example_user/Projects/moddable/modules/data/wavreader/manifest.json"
		]
	}
}

I think there a couple of problems with that though:

1. Determining the local path of your node is not obvious (editor code runs in the browser, so doesn't have access to Node.js globals like __dirname).
2. Even if you can determine the path, it isn't portable or stable, and so doesn't belong in the exported JSON data.

I have an idea how to address that, but need to build a prototype to prove it out. I'll try to do that tomorrow.

[colinl]

Yes, those are exactly the problems I was seeing.

[phoddie]

@colinl, I put together a quick example. It works as I hoped. Unfortunately, I don't have the time today to document it fully. Check out the lower-case-mcu example.

• The only change to the HTML is to include the moddable_manifest property in the defaults.
• There is a separate implementation of the node, lower-case-mcu, which is optimized for the Node-RED MCU Edition runtime. This avoids using the compatibility node. This isn't required, but it made the example more interesting.
• Install the lower-case-mcu node into the Node-RED Editor as usual with npm install.
• No manifest editing required.
• An example flow is below.

[
    {
        "id": "fd7d965ef27a87e2",
        "type": "tab",
        "label": "Flow 4",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "581ac525ad83fbde",
        "type": "lower-case",
        "z": "fd7d965ef27a87e2",
        "name": "",
        "moddable_manifest": {
            "include": [
                {
                    "git": "https://github.com/phoddie/lower-case-mcu.git"
                }
            ]
        },
        "x": 490,
        "y": 220,
        "wires": [
            [
                "adbaf5c92b7434ec"
            ]
        ]
    },
    {
        "id": "7b0fd3f3169c3264",
        "type": "inject",
        "z": "fd7d965ef27a87e2",
        "name": "",
        "props": [
            {
                "p": "payload"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": true,
        "onceDelay": 0.1,
        "topic": "",
        "payload": "THIS IS A TEST",
        "payloadType": "str",
        "x": 260,
        "y": 120,
        "wires": [
            [
                "581ac525ad83fbde",
                "adbaf5c92b7434ec"
            ]
        ]
    },
    {
        "id": "adbaf5c92b7434ec",
        "type": "debug",
        "z": "fd7d965ef27a87e2",
        "name": "",
        "active": true,
        "tosidebar": false,
        "console": true,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 670,
        "y": 120,
        "wires": []
    }
]

[colinl]

I am a bit confused by the example. I thought it would be possible to put the manifest file in the node's source folder, but I see the modable_manifest property is pointing to the node's git repository.

[phoddie]

@colinl – I agree: that would be nice. But, it is impractical because, as noted above, an absolute local file path is a bad idea and there is no obvious way to reliably map the source code of a node identifier ("sensor") to its local sources, especially running outside of the Node-RED environment as the nodere2mcu tools does. The MCU plug-in tries to do this mapping and it is challenging, as @ralphwetzel has patiently explained to me. For example, we made significant organizational changes to this repository just to help the MCU plug-in.

On the other hand, pointing to the repository works transparently for users of the node. Importantly, it allows the Node-RED Editor to use one implementation of the node and the MCU runtime to use a completely separate one (which is common enough already) -- again without the user having to take any additional steps. This creates a path to seamlessly and non-disruptively integrate MCU support for existing nodes. It also allows builds of exported flows succeed even on machines that do not have Node-RED installed, which is great for automated builds and updates.

The primary inconvenience is for developers of nodes. Fortunately, it is a small one. During development they will have moddable_manifest include from a local file, and for deployment from their repository. In time, perhaps someone will propose a solution to simplify this too.

[colinl]

In the html file, where one has something like

moddable_manifest: {value: {
				include: [
					{
						"git": "https://github.com/phoddie/lower-case-mcu.git"
					}
				]
			}}

will that allow the specification of a particular release of the repository?

There has been discussion about this in the node-red forum here where @ralphwetzel has pointed out that if I include a manifest file in the package's root folder then the plugin will automatically include it, so in that case there is no need to provide it via the route proposed here, so at the moment I am confused about what I should do.

[ralphwetzel]

I took some time to reflect about the proposal of @phoddie.
In fact, this it not an either / or - but it's an and ... which I consider a valuable progress (meanwhile!)

Peter was a bit short in explaining what's happening when you include the repository link in the manifest.
In the end, mcconfig pulls the whole repository and operates on this data. You can even fine-tune the pull spec - which is quite powerful.

Thus you simply put the manifest.json in the packages root folder... and ... provide the link to the repo.

If you use the plugin (and no repo is defined), it will care to put the files where they belong locally.
Once you define the repo - and you run without plugin - the repo will be pulled and those files be used.

There's a case I have to make up my mind for: Running with the plugin and having a repo defined; as it is now, this will include two manifest.jsons. I'm yet convinced we'll find a solution for this...

[phoddie]

will that allow the specification of a particular release of the repository?

It will, as described in the documentation.

...if I include a manifest file in the package's root folder then the plugin will automatically include it...

...then the plugin will automatically include it, so in that case there is no need to provide it via the route proposed here, so at the moment I am confused about what I should do.

If you are willing to limit yourself to working with the plug-in, you can depend on its features. My goal is to design solutions that work solely with the exported data from Node-RED to support a broad ecosystem of solutions. The MCU plug-in is great, but there are other equally valid workflows. Fortunately, as @ralphwetzel notes, the plug-in here is effectively an optimization so your node can include the moddable_manifest property, and the plug-in can choose to override that to use Node-RED's local copy.

[colinl]

Thanks, that has clarified it a lot.
So, for the moment, until the duplicate inclusion issue is fully worked out, and since I am using the plugin for these particular experiments, I think I am probably best just putting it with the package source, and don't provide a link to the repo.