Redis: WRONGPASS invalid username-password pair

[satyam1990]

Hi Team,

I am writing a new device service using EdgeX C SDK. I was following the documentation for running the sample random number service.
Initially I faced issue with secrets-token.json not found that got created when I added my service name in docker-compose.yml file under below attribute:
ADD_SECRETSTORE_TOKENS: 'device-mqtt, device-modbus, device-rest, device-template'

After doing above change I was able to run my sample device-template service and issue requests to it via the EdgeX Command microservice. (i.e. I can successfully call the REST API http://EDGE-IP:59882/api/v2/device/name/RandNum-Device01/RandomNumber)

Then I noticed there is below error coming on the stdout when I execute the service:
level=ERROR ts=2022-10-26T10:47:51Z app=device-template msg="vault: GET http://localhost:8200/v1/secret/edgex/device-template/redisdb failed"
level=ERROR ts=2022-10-26T10:47:51Z app=device-template msg="vault: get secrets request failed"

In order to resolve above error I see addon-service documentation and hence added below line in my docker compose:
ADD_KNOWN_SECRETS: redisdb[app-rules-engine], redisdb[device-mqtt], redisdb[device-modbus], redisdb[device-rest], redisdb[device-template]

After I made above change now my device service immediately exits after throwing below error:
level=INFO ts=2022-10-26T10:38:36Z app=device-template msg="Event data will be sent through Redis streams at localhost:6379"
level=ERROR ts=2022-10-26T10:38:36Z app=device-template msg="Error authenticating with Redis: WRONGPASS invalid username-password pair or user is disabled."
Error: 15: Remote server unresponsive
level=DEBUG ts=2022-10-26T10:38:36Z app=device-template msg="Scheduler thread terminating"
level=DEBUG ts=2022-10-26T10:38:37Z app=device-template msg="Thread exiting"

I could not figure out why username-password is invalid since both are auto generated and nothing is manually done by me.

1. Any help on how to resolve this redis issue would be great.

2. Also if someone could explain a bit about these secret tokens on how they work? Like when we add a device service name in ADD_SECRETSTORE_TOKENS list then does it create something like public private keys. Private key gets stored with the device-service and public gets distributed with other services, not sure what happens but it would be great if someone could explain a simple use-case, on the use of secret-tokens. I went through security docs but somehow not getting it.

3. How is my sample device service "device-template" will be making use of redis, why do we want to connect to redis, does my sample device service send auto-events data to redis or redis is used for some other purpose.

Version details:
I am using:
EdgeX Version: 2.1.1
EdgeX C SDK version: 2.2.0

Additional Info on how I am running my sample device service "device-template":

• I am not running my device service in docker, instead I am just running it on the host machine natively.

• Since I am not running it in docker then existing EdgeX services like Command Service was unable to reach to a service binded on host machine hence I had to modify the bind IP to the IP of the docker interface:
  3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
  link/ether 02:42:a4:bb:3f:c3 brd ff:ff:ff:ff:ff:ff
  inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0

I did this by modifying configuration.toml file:
Host = '172.17.0.1' # set this to the IP address of the system host
ServerBindAddr = '172.17.0.1' # blank value defaults to Service.Host value

• Another thing I had to change in the configuration.toml file is the service name because the default configuration.toml provided in the documentation has device name not matching the name of the device service in the source code. So I just changed "device-simple" to "device-template" since in source code we using device-template.

With this bind IP and name change my service works fine, I can use command microservice to get value of resource attributes, now only issue is that when I add redisdb[device-template] in ADD_KNOWN_SECRETS then things wont work as explained above.

[lenny-goodell]

Please attach you docker compose file where you have added the above changes.

[bnevis-i]

Also if someone could explain a bit about these secret tokens on how they work? Like when we add a device service name in ADD_SECRETSTORE_TOKENS list then does it create something like public private keys. Private key gets stored with the device-service and public gets distributed with other services, not sure what happens but it would be great if someone could explain a simple use-case, on the use of secret-tokens. I went through security docs but somehow not getting it.

ADD_SECRETSTORE_TOKENS in your example will result in a file, secrets-token.json being created at /tmp/edgex/secrets/device-template (default location). This file contains an authentication token to the EdgeX secret store. the ADD_KNOWN_SECRETS variable says to copy the redis password to edgex/device-template/redisdb in the secret store. The library is trying to access it via the URL http://localhost:8200/v1/secret/edgex/device-template/redisdb

The error

level=ERROR ts=2022-10-26T10:47:51Z app=device-template msg="vault: get secrets request failed"

Suggests that the secret store token could not be accessed by the service, was invalid, or some other error. Curiously, this should have been fatal but it attempted to access the database anyway, which of course did not work.

[satyam1990]

Hi @bnevis-i

This error "vault: get secrets request failed" I was able to correct as I have mentioned in my post by adding redisdb[device-template] in ADD_KNOWN_SECRETS: section.

Now the error I am facing is "Error authenticating with Redis: WRONGPASS invalid username-password pair or user is disabled."

If I run my service under strace like:
strace -s 256 ./device-example-c

Then I could see my device-service is trying to send below as username and password to redis:
sendto(3, "*3\r\n$4\r\nAUTH\r\n$6\r\nredis5\r\n$44\r\noosZfw7r3UnGW2y1/QXbsEzKMb2c8KtAgDKDfyVaUj8W\r\n", 77, 0, NULL, 0) = 77
Username: redis5
Password: oosZfw7r3UnGW2y1/QXbsEzKMb2c8KtAgDKDfyVaUj8W

I assume my sample device service is able to fetch user name password from vault that is why it is able to send the same but somehow it is not accepted by redis.

@lenny-intel Attaching my docker-compose file, please see if I made some mistake.
docker-compose.txt

[bnevis-i]

Username: redis5
Password: oosZfw7r3UnGW2y1/QXbsEzKMb2c8KtAgDKDfyVaUj8W

That's an easy one. It was a bug. edgexfoundry/edgex-go#4031

Username was a feature added in Redis5. The existing go code had only ever used the AUTH <password> form, which is an implied default user.

Unfortunately, this fix didn't make the 2.2.0 release. Good news is that you won't have to wait much longer for the fix, which will be in the 2.3.0 release.

It should be possible to also manually patch the secretstore with the fix, though it could be a little involved to do so.

[satyam1990]

@bnevis-i thanks for the quick response, just wanted to ask that you mean bug is in the EdgeX Security microservice or in EdgeX C SDK.

In continuation to above so do i need to update my EdgeX C SDK to 2.3.0 or the Security or other core microservices i need to update to 2.3.0 once the fix is released.

From the code changes i feel like it's the security microservice secretstore i will have to update to 2.3.

Moreover when you release new build i guess then new docker-compose yaml gets released and all core microservices will be at same release version in that

[bnevis-i]

@bnevis-i thanks for the quick response, just wanted to ask that you mean bug is in the EdgeX Security microservice or in EdgeX C SDK.

The C SDK is correct. the security-secretstore-setup component lied and put the wrong username in the secret store.

In continuation to above so do i need to update my EdgeX C SDK to 2.3.0 or the Security or other core microservices i need to update to 2.3.0 once the fix is released.
From the code changes i feel like it's the security microservice secretstore i will have to update to 2.3.

If you are using the docker version, you could switch to nexus3.edgexfoundry.org:10004/security-secretstore-setup:latest and everything should magically start working (need to clean first to reset secret store). This is an unreleased, untagged image for nightly regression purposes, so you can't use it in production.

Once Levski release is out, you can switch to edgexfoundry/security-secretstore-setup:2.3.0 and it should also have the fix that would work with the old SDK. I don't think we have any automated tests that mix SDK versions, so when 2.3.0 is out, if you wanted to continue using the old C SDK, you might have to do your own validation test.

Moreover when you release new build i guess then new docker-compose yaml gets released and all core microservices will be at same release version in that

Unless anything unexpected happens, we should have new github tags and docker images in the back half of next week. The docker-compose files will also be updated to reference those new tagged release images.

[satyam1990]

Hi @bnevis-i

Just a follow up query, when I started using the image nexus3.edgexfoundry.org:10004/security-secretstore-setup:latest, the security-secretstore-setup microservice is continuously restarting and from the docker logs on this service I can see below error:

level=INFO ts=2022-11-09T14:21:08.428368795Z app=security-secretstore-setup source=init.go:587 msg="Starting the Kong Admin API config file creation"
level=ERROR ts=2022-11-09T14:21:08.428427336Z app=security-secretstore-setup source=init.go:595 msg="failed to configure the Kong Admin API: [build-kong-admin][error] Kong configuration file path (.paths.config) could not be created"
level=INFO ts=2022-11-09T14:21:08.428457293Z app=security-secretstore-setup source=tokenmaintenance.go:84 msg="revoking token-issuing-token"
level=INFO ts=2022-11-09T14:21:08.431530566Z app=security-secretstore-setup source=request.go:97 msg="successfully made request to revoke self token"
level=INFO ts=2022-11-09T14:21:08.431575831Z app=security-secretstore-setup source=init.go:276 msg="revoking temporary root token"
level=INFO ts=2022-11-09T14:21:08.433680732Z app=security-secretstore-setup source=request.go:97 msg="successfully made request to revoke self token"

We are not using Kong maybe thats why, is it possible to workaround this error.

[bnevis-i]

@satyam1990 it shouldn't matter whether you are using Kong or not. All it does is write a configuration file.

Full explanation here:
https://github.com/edgexfoundry/edgex-go/blob/main/internal/security/secretstore/init.go#L568

[satyam1990]

@bnevis-i But since it could not create that configuration file, secretstore-setup microservice is unable to start. It keeps on restarting.

Due to that redis is somehow not working as I see in redis logs:
level=INFO ts=2022-11-09T16:43:04.494846184Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2022-11-09T16:43:04.494932098Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-secretstore-setup:54322] with timeout: [1m0s]"
level=INFO ts=2022-11-09T16:43:04.577274199Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-secretstore-setup on 127.0.0.11:53: no such host. Sleeping 1s"

It seems to be looking for secretstore-setup which is restarting continously.

Hence even though now my device service have right credentials it is unable to connect:
level=ERROR ts=2022-11-09T16:43:32Z app=device-template msg="Error authenticating with Redis: Server closed the connection"
Error: 15: Remote server unresponsive

[bnevis-i]

I suggest putting this on hold a day or two and testing with the new release when it is out.

[bnevis-i]

@satyam1990 Levski (2.3) release is out today.

[satyam1990]

@bnevis-i Tried with levski image: nexus3.edgexfoundry.org:10004/security-secretstore-setup:2.3.0

Still same issue I am facing as mentioned before security-secretstore-setup keeps on restrating.
I have hardcoded the redis username in EdgeX C SDK for now just to make redis connection a success.

[bnevis-i]

@satyam1990 Another thing to do is try to update security-bootstrapper as well and see if that makes a difference. Bootstrapper writes out some config files, that maybe some of the downstream components are expecting.

[satyam1990]

Still having same issue even after upgrading both services to levski:
image: nexus3.edgexfoundry.org:10004/security-bootstrapper:2.3.0
image: nexus3.edgexfoundry.org:10004/security-secretstore-setup:2.3.0

level=ERROR ts=2022-11-28T10:12:51.640025037Z app=security-secretstore-setup source=init.go:595 msg="failed to configure the Kong Admin API: [build-kong-admin][error] Kong configuration file path (.paths.config) could not be created"

Just for info:
For now I have updated the EdgeX C SDK and hardcoded the username which works well for me.

[bnevis-i]

@satyam1990 that works for me. I think the kong error has something to do with not being able to create /tmp/kong folder to write out a temporary config file.