fixed module path

[krismarc]

fixed module path
reported here: michaelarnauts/home-assistant-comfoconnect#77

[michaelarnauts]

@krismarc Why does it needs to have another nested aicomfoconnect/protobuf path for the nanopb?

I'm looking for a solution where I can keep using the original .proto files (as they come from the Zehnder APK). It seems this might be related: grpc/grpc#9575 (comment)

[krismarc]

@michaelarnauts It doesn't need. I had a hope it would put generated module directly in expected directory within your project (so files locations remains the same). However, it seems to produce the same structure in the destination directory so I had to manually place generated files in aicomfoconnect/protobuf anyway.

The idea was to just get rid of dependency to nanopb module (the one you have packed into this project was never in use) and use your own instead without modyfing it after compilation. Home assistant has bumped nanopb to 0.4.9 which has dependency to protobuf 5.+ and that's why it can't find runtime_version module.

[krismarc]

So the main goal is to stop using nanopb_pb2.py from nanopb module and tell zehnder_pb2.py to import one from your project module. Later it calls protobuf modules which are yet 4.+.

The breaking change is nanopb's upgrade from 0.4.8 (which has nanopb_pb2.py protobuf's 4.+ compliant) to 0.4.9 (which has newer nanopb_pb2.py generated by 5.+ and it requires protobuf's 5.+).

It's weird to me. IMO nanopb's shouldn't be involved at all. Eventually, both nanopb and protobuf should be upgraded and then Zehender modules generated by protos 5.+

[krismarc]

ah sorry! It needs to be like this! This is the way how we tell protoc compiler to build the import path. So by doing this we achieve

from aiocomfoconnect.protobuf import nanopb_pb2 as aiocomfoconnect_dot_protobuf_dot_nanopb__pb2

https://github.com/michaelarnauts/aiocomfoconnect/pull/42/files#diff-11cdd0c45a310a93f85ec32c805d256961c873f858d326d4c658c1ff1e11de8bR15

if there's no nested directory it considers it as specific module and pulls the module from nanopb package:

import nanopb.generator.proto.nanopb_pb2 as nanopb__pb2

[krismarc]

I'm looking for a solution where I can keep using the original .proto files (as they come from the Zehnder APK). It seems this might be related: grpc/grpc#9575 (comment)

isn't it what I've actually done?

[krismarc]

I'd be not surpised that Zehnder apk is also not aware that they do relay on nanopb module instead their own version ;)

[michaelarnauts]

I'm not sure what's going on...

I initially also had a dependency on nanopb, not sure why. You would think that I could get rid of the nanopb.proto file then, and that the import in the zehnder.proto would pick up the file from the nanopb package, but that doesn't happen. (Looks like nanopb/nanopb#877), so I leave it anyway. The nanopb.proto file in this repo is the one from the Zehnder APK, and the upstream one has a few extra lines.

Also, when I remove the import nanopb_pb2 as nanopb__pb2 line from zehnder_pb2.py, python complains ModuleNotFoundError: No module named 'nanopb_pb2', so it really doesn't pickup anything from the nanopb package. But then how can an upgrade of the nanopb package trigger these issues?

So... so far the only fix I can see is your original suggestion by modifying the zehnder_pb2.py file so it does a import aiocomfoconnect.protobuf.nanopb_pb2 as nanopb__pb2.

I also have removed the nanopb dependency and upgraded to protobuf = "^5.28.2" and grpcio-tools = "^1.66.1", and regenerated the files.

However, looking at Home Assistant, it seems they still pin protobuf to 4.25.1.

homeassistant/package_constraints.txt:protobuf==4.25.4

So I'm not sure again why it works, while we are using protobuf 5.28.2 now, and Home Assistant is still using Protobuf 4.25.4

[krismarc]

Exactly, we will face the same issue once they upgrade to 5.+

[krismarc]

Also, when I remove the import nanopb_pb2 as nanopb__pb2 line from zehnder_pb2.py, python complains ModuleNotFoundError: No module named 'nanopb_pb2', so it really doesn't pickup anything from the nanopb package. But then how can an upgrade of the nanopb package trigger these issues?

because, nanopb 0.4.9 has nanopb_pb2, however generated by protoc 5.+ which requires protobuf 5.+ dependencies like runtime_version.py :) and it's not found in protobuf 4.+

[michaelarnauts]

Also, when I remove the import nanopb_pb2 as nanopb__pb2 line from zehnder_pb2.py, python complains ModuleNotFoundError: No module named 'nanopb_pb2', so it really doesn't pickup anything from the nanopb package. But then how can an upgrade of the nanopb package trigger these issues?

because, nanopb 0.4.9 has nanopb_pb2, however generated by protoc 5.+ which requires protobuf 5.+ dependencies like runtime_version.py :) and it's not found in protobuf 4.+

Okay, but then upgrading to protobuf 5.+ and using nanopb would fix it, but that doesnt.

Also, are you sure home assistant is using nanopb 0.4.9? I can't find anything in the git repo as a dependency, and protobuf is pinned to 4.25.4. I wonder how protobuf 5 comes into play then.

[krismarc]

yes, that's from HA's python:
michaelarnauts/home-assistant-comfoconnect#77 (comment)

IMO, nanopb and protobuf are seperate projects and they lost track at some point.

Let me try to run it using nanopb 0.4.9 and protobuf 5.+, at the moment I think this worked for me yesterday.

[krismarc]

nanopb 0.4.9 should have requirement for protobuf 5+ and I think it's not there, not sure why HA made upgrade only of nanopb alone, I think it's worth letting them know :)

[michaelarnauts]

It seems Home Assistant is still using a pinned protobuf == 4.25.4

First attempt to go to go to 5.26.0 (March 14) has been closed.

• Bump protobuf to 5.26.0 home-assistant/core#113380

Then, they upgraded to 4.25.4 on Aug 12. This has been merged:

• Bump protobuf to 4.25.4 home-assistant/core#123675

Finally, a recent draft is open to upgrade to 5.28.0.

• Bump protobuf to 5.28.2 home-assistant/core#124936

So I assume they want to upgrade to 5.28, but they can't yet.

[krismarc]

not sure how nanopb comes into play here, how it appears in HA's python modules. Maybe it's a dependency of something's else? I just checked locally difference of running your project using older and newer nanopb. With new one the problem is reproducible, while with 0.4.8 it works fine and it's internal nanopb_pb2 has no dependency to eg. runtime_version. Basically, it calls protobuf 4.+

[krismarc]

at the end, it doesn't really matter, even if they would remove nanopb considering it as not in use, you would face missing nanopb_pb2 then, unless you use your own.

[krismarc]

let's say, you were simply lucky till now, that nanopb was there out of the box :D

[krismarc]

I assume, dependency to nanopb_pb2 could be replaced with some internal protobuf methods if I would know this API 😄
..nanopb's seems to be just simplified protobuf for embedded systems. and I assume this would be the right approach. Probably Zehnder's developer found this way and used it without considering potential side-effects.

[michaelarnauts]

I'm back to the original error now when trying to use Protobuf 5 generated files with a Protobuf 4 in Home Assistant.

I think I will have to generate Protobuf 4 files, and use them with Home Assistant, and once Home Assistant uses Protobuf 5, we can (and will have) to upgrade to Protobuf 5 also.

[michaelarnauts]

@krismarc I've made a new branch here without the changing of the .proto files. I initially tried to push to your branch, but I wasn't allowed.

Thanks a lot for your help!

[krismarc]

@krismarc I've made a new branch here without the changing of the .proto files. I initially tried to push to your branch, but I wasn't allowed.

Thanks a lot for your help!

does it mean, you have edited generated module manually after generation? I mean the import path.

[krismarc]

or it's smart enough and this part wasn't really needed?
https://github.com/michaelarnauts/aiocomfoconnect/pull/42/files#diff-0be108a4c49f824c3d933250f17d02ca4b97f959a01c850ec18d5f60f6691b48R5