FD sharing protocol on localhost #82

TheMutta · 2024-07-01T10:34:05Z

This update creates a new interface to stream over files and a new protocol that allows for hosts where both the Server and the Client instances of XLink are running on the same device to simply send over the FD and its ownership instead of just streaming the file data over. This allows for sharing of memory buffers without the need for copying, i.e. zerocopy.

An automatic discovery for the protocol is setup for TCP/IP.

It is to be noted that the packet size for XLink has been altered by adding a new FD attribute, meaning that older versions of XLink won't be able to parse the packets correctly.

…nctions.

This reverts commit 8aa655b.

This reverts commit 3faa07c.

…nkWriteFd

…d is called on a protocol different from X_LINK_LOCAL_SHDMEM

…CP_IP

themarpe

Lets change the server to be "dual server" with the TCP/IP mechanism.

Instead of connect/disconnect/reconnect, lets spin up both TCP & SHDMEM, by the following mechanism (C++):

Create two threads and return the fd's for the server sockets back to main context
cond_var to which completes sooner
close the other that did not complete yet
join both threads
continue & return which protocol

For this dual option, we might create a separate protocol entry to signify going down this path. (so TCP_IP would just be TCP, SHDMEM just shdmem and TCP_IP_OR_LOCAL_SHDMEM would be the grouped one)

For the connecting client, it would be similar, if TCP_IP_OR_LOCAL_SHDMEM is picked, then following occurs:

First try connecting via local shdmem
if fails, go TCP/IP route

So no need for the threading on the client side

TheMutta · 2024-07-01T11:49:35Z

Instead of connect/disconnect/reconnect, lets spin up both TCP & SHDMEM, by the following mechanism (C++):

Create two threads and return the fd's for the server sockets back to main context

cond_var to which completes sooner

close the other that did not complete yet

join both threads

continue & return which protocol

Ok that sounds doable, but i have some thoughts.
If we're localhost and the TCP/IP connection resolves faster than the SHDMEM connection, which could definitely be possible, since the TCP/IP thread could be the first to be executed and therefore the first to complete the task, then we'd lose all the benefits of zerocopy.
Apart from that, we also risk that on the server side the TCP/IP thread completes first, while on the client side the SHDMEM could complete first, which is not to be excluded on a multicore system. And if we're single core, the first one to connect would absolutely be arbitrary. And that would be detremental.

For this dual option, we might create a separate protocol entry to signify going down this path. (so TCP_IP would just be TCP, SHDMEM just shdmem and TCP_IP_OR_LOCAL_SHDMEM would be the grouped one)

For the connecting client, it would be similar, if TCP_IP_OR_LOCAL_SHDMEM is picked, then following occurs:

First try connecting via local shdmem
if fails, go TCP/IP route
So no need for the threading on the client side

I'd prefer a similar approach to this, and like the one i put in, which would only require some minor changes to depthai and would avoid all the shenaningans that multithreading adds to a multicore system. Maybe adding a timeout for SHDMEM connection of, for example, 5 seconds, would be perfect, since also the SHDMEM connection can happen only after TCP/IP has been connected, meaning we have both a server and a client already in place.

themarpe · 2024-07-01T12:10:24Z

If we're localhost and the TCP/IP connection resolves faster than the SHDMEM connection, which could definitely be possible, since the TCP/IP thread could be the first to be executed and therefore the first to complete the task, then we'd lose all the benefits of zerocopy.
Apart from that, we also risk that on the server side the TCP/IP thread completes first, while on the client side the SHDMEM could complete first, which is not to be excluded on a multicore system. And if we're single core, the first one to connect would absolutely be arbitrary. And that would be detremental.

So, WRT the server, it always waits for both - the the timing doesn't matter, just the sequence.
The client will try SHDMEM first, and if that succeeds, then the server will wrap up and not wait for TCPIP.

I'd prefer a similar approach to this, and like the one i put in, which would only require some minor changes to depthai and would avoid all the shenaningans that multithreading adds to a multicore system. Maybe adding a timeout for SHDMEM connection of, for example, 5 seconds, would be perfect, since also the SHDMEM connection can happen only after TCP/IP has been connected, meaning we have both a server and a client already in place.

Above case does cover that - only one protocol at once.
Also, SHDMEM protocol fully supports all read/write ops right? (there is no need to have the TCP alongside it, correct)

TheMutta · 2024-07-01T12:37:36Z

Also, SHDMEM protocol fully supports all read/write ops right? (there is no need to have the TCP alongside it, correct)

Yes exactly, it's a completely indipendent protocol, and read/writes work perfectly. The only use for TCP/IP is to, well, determine if we're connecting to localhost and as a backup if SHDMEM connection doesn't work, meaning the connection is immediate if we're not connecting to localhost, and just takes the minimum time to open a socket if we want to connect through SHDMEM.

So, WRT the server, it always waits for both - the the timing doesn't matter, just the sequence.
The client will try SHDMEM first, and if that succeeds, then the server will wrap up and not wait for TCPIP.

Ok sure, that seems doable. It's a bit more complicated than the current system, and would function similarly. I still think the present system may be less of a hassle, with the multithreaded example we'll have to think about cleanup of open sockets that have not been connected to. In any case, perfectly doable.

TheMutta · 2024-07-01T13:36:06Z

So this is what i came up with:

typedef struct {
    bool lock;
    const char **devPathRead;
    const char **devPathWrite;
    XLinkProtocol_t **protocol;
    void **fd;
} args_t;

static void *tcpipServer(args_t *args) {
    if(tcpipPlatformServer(*(args->protocol), *(args->devPathRead), *(args->devPathWrite), args->fd) == X_LINK_SUCCESS) {
	args->lock = true;
    }

    return NULL;
}

static void *shdmemServer(args_t *args) {
    if(shdmemPlatformServer(SHDMEM_DEFAULT_SOCKET, SHDMEM_DEFAULT_SOCKET, args->fd) == X_LINK_SUCCESS) {
	args->lock = true;
	shdmemSetProtocol(*(args->protocol), *(args->devPathRead), *(args->devPathWrite));

    }
    return NULL;
}

xLinkPlatformErrorCode_t XLinkPlatformServer(const char* devPathRead, const char* devPathWrite, XLinkProtocol_t *protocol, void** fd)
{
    switch (*protocol) {
        case X_LINK_TCP_IP: {
	    pthread_t shdmemThread;
	    pthread_t tcpipThread;

	    args_t args;
	    args.lock = false;
	    args.devPathRead = &devPathRead;
	    args.devPathWrite = &devPathWrite;
	    args.protocol = &protocol;
	    args.fd = fd;

	    pthread_create(&shdmemThread, NULL, shdmemServer, (void*)&args);
	    pthread_create(&tcpipThread, NULL, tcpipServer, (void*)&args);

	    while(!*((volatile bool*)&args.lock)) {
	    }

	    if (*protocol == X_LINK_TCP_IP) {
		pthread_join(tcpipThread, NULL);
		pthread_cancel(shdmemThread);

	        return X_LINK_SUCCESS;
	    } else if (*protocol == X_LINK_LOCAL_SHDMEM) {
	        pthread_join(shdmemThread, NULL);
		pthread_cancel(tcpipThread);

		return X_LINK_SUCCESS;
	    } else {
		return X_LINK_PLATFORM_ERROR;
	    }

	    }
	    ....

The only issue would be cleaning up the remaining open sockets. This can only be done with cleanup handlers inside the tcpipPlatformServer and the shdmemPlatformServer functions. The fact is, being a mixed C and C++ codebase, this absolutely makes the pthreads function throw out bogus errors like error: expected ‘while’ before ‘listen’, which i couldn't believe at first. Turns out that pthreads are bugged in a mixed C/C++ context and, therefore, the cleanup functions can't be used.

Therefore, our multithreading approach seems more trouble than it's worth.

themarpe

Don't cancel the thread, but "close" the fd and just join both threads as is. Also, might be simpler to do this in C++ context due to thread, mutex & conditional_variable, etc...

Otherwise if really PITA, we can leave as is - but the "sleep" should atleast be "magic'd out" & add to comments as "future impl." posiblity, that we can go to spinning both protocols up in parallel

Also, this is Posix only I assume, modify CMakeLists.txt to only activate if Posix & be optionally disable-able as well

& add to CI & tests (which will make sure that we don't compile it in for Windows as well, etc...)

themarpe · 2024-07-01T15:23:55Z

include/XLink/XLinkPrivateDefines.h

-    XLINK_READ_REL_SPEC_REQ,
-    XLINK_READ_REL_SPEC_RESP,


Why removed?

Also, all new events MUST GO to the end of enum, otherwise backward compatibility is broken

I moved them before XLINK_LAST_REQ and XLINK_LAST_RESP, otherwise they will cause this assert to fail:

int handleIncomingEvent(xLinkEvent_t* event, XLinkTimespec treceive) { //this function will be dependent whether this is a client or a Remote //specific actions to this peer mvLog(MVLOG_DEBUG, "%s, size %u, streamId %u.\n", TypeToStr(event->header.type), event->header.size, event->header.streamId); ASSERT_XLINK(event->header.type >= XLINK_WRITE_REQ && event->header.type != XLINK_REQUEST_LAST && event->header.type < XLINK_RESP_LAST);

Hmm - do we need a special type of an event?
This will likely not work due to it being shared with (fixed) Bootloader on RVC2 (eg communicating with it will cause issues).

So lets try to either:

Remove the need for custom xLinkEventType entry

reorder them to the end & fix assert (eg, to check between two ranges)

Side note: xLinkEventHeader_t is also a structure that should not change (or "reserved bits" should be used under bitField)

TheMutta · 2024-07-01T17:28:47Z

Don't cancel the thread, but "close" the fd and just join both threads as is. Also, might be simpler to do this in C++ context due to thread, mutex & conditional_variable, etc...

I think i misunderstood, you mean that we'd spin up a separate thread waiting for the SHDMEM connection and, if successful, we just close the TCP/IP connection, create a new platform FD and switch to that protocol?

Otherwise if really PITA, we can leave as is - but the "sleep" should atleast be "magic'd out" & add to comments as "future impl." posiblity, that we can go to spinning both protocols up in parallel

It probably is more of a PITA that it's worth, at least imo. For the sleep, i think we could hide it in the shdmem protocol, with multiple retries delayed by a fixed amounts of ms to wait in between in connect().

Also, this is Posix only I assume, modify CMakeLists.txt to only activate if Posix & be optionally disable-able as well
& add to CI & tests (which will make sure that we don't compile it in for Windows as well, etc...)

Yes, i guess a simple preprocessor directive like #ifdef(__unix__)) will do the trick, since the code doesn't work in any other platform anyway.

TheMutta · 2024-07-02T09:58:55Z

Don't cancel the thread, but "close" the fd and just join both threads as is. Also, might be simpler to do this in C++ context due to thread, mutex & conditional_variable, etc...

So, that has been completed. It was indeed far easier to do in C++, i just had to add some minor modification to make sure the socket is closed properly and the unused thread is detached.

Does it look good?

themarpe · 2024-07-03T19:35:50Z

include/XLink/XLinkPrivateDefines.h

-    XLINK_READ_REL_SPEC_REQ,
-    XLINK_READ_REL_SPEC_RESP,


Hmm - do we need a special type of an event?
This will likely not work due to it being shared with (fixed) Bootloader on RVC2 (eg communicating with it will cause issues).

So lets try to either:

Remove the need for custom xLinkEventType entry

reorder them to the end & fix assert (eg, to check between two ranges)

themarpe · 2024-07-03T19:37:23Z

include/XLink/XLinkPrivateDefines.h

-    XLINK_READ_REL_SPEC_REQ,
-    XLINK_READ_REL_SPEC_RESP,


Side note: xLinkEventHeader_t is also a structure that should not change (or "reserved bits" should be used under bitField)

themarpe · 2024-07-03T19:40:22Z

src/pc/protocols/tcpip_host.h

 xLinkPlatformErrorCode_t tcpipPlatformBootBootloader(const char *name);
 int tcpipPlatformDeviceFdDown(void *fd);
 int tcpipPlatformClose(void *fd);
 int tcpipPlatformBootFirmware(const deviceDesc_t* deviceDesc, const char* firmware, size_t length);

+bool tcpipIsLocalhost(const char *ip);


No need for this anymore

themarpe · 2024-07-03T19:40:29Z

src/pc/protocols/tcpip_host.cpp

+// Checks whether ip is localhost
+bool tcpipIsLocalhost(const char *ip) {
+    if(strncmp(ip, "127.0.0.1", strlen("127.0.0.1")) == 0 ||
+       strncmp(ip, "0.0.0.0", strlen("0.0.0.0")) == 0) {
+	return true;
+    }
+
+    return false;
+}
+


No need for this anymore

themarpe · 2024-07-03T19:44:48Z

src/pc/protocols/tcpip_host.cpp

@@ -1163,6 +1233,16 @@ int tcpipPlatformBootFirmware(const deviceDesc_t* deviceDesc, const char* firmwa
    return -1;
 }


Retain tcpip_host.cpp and local_memshd.cpp as much as is, and move this code elsewhere.

Final goal being able to have this "special case" only under TCP_IP_OR_LOCAL_SHDMEM protocol, and the new code calls tpcServer + cancel & memshdServer + cancel with threads (some state will have to be shared, so some changes to the tcpip & local shdmem will have to be done, but let those be rather minimal - that code is already super finicky & its good that it remains that way, with this change essentially just "closing" & joining the thread whose server didn't got any connections)

Also, no detaches - just joins() - if we cannot join, we have a problem.

And - move the thread & vars into func scope or struct. should look something like this in the end:

tcpipOrLocalShdmemServer(*protocol) { if(protocol == nullptr) return; std::mutex mtx; std::condition_variable cv; bool isTcpIpThreadFinished = false; bool isShdmemThreadFinished = false; int retTcpIp = -1, retShdmem = -1; void *fdTcpIp = nullptr, *fdShdmem = nullptr; auto thread_tcpip = std::thread([&mtx, &cv, &isTcpIpThreadFinished, &retTcpIp, &fdTcpIp](){ auto ret = tcpipPlatformServer(..., &fdTcpIp); { std::unique_lock<std::mutex> l(mtx); isTcpIpThreadFinished = true; retTcpIp = ret; } cv.notify_one(); }); auto thread_shdmem = std::thread([&mtx, &cv, &isShdmemThreadFinished, &retSdhmem, &fdShdmem](){ auto ret = shdmemPlatformServer(..., &fdShdmem); { std::unique_lock<std::mutex> l(mtx); isShdmemThreadFinished = true; retSdhmem = ret; } cv.notify_one(); }); { std::unique_lock<std::mutex> lock(mtx); cv.wait(lock, [&isShdmemThreadFinished, &isTcpIpThreadFinished]{ return isShdmemThreadFinished || isTcpIpThreadFinished; }); // As soon as either one finishes, the other can be cleaned // Use signals, as "accept" cannot be unblocked by "close"ing the underlying socket if(!isShdmemThreadFinished) pthread_kill(thread_shdmem.native(), SIGINT); if(!isTcpIpThreadFinished) pthread_kill(thread_tcpip.native(), SIGINT); } // Wait for both threads to wrap up if(thread_tcpip.joinable()) thread_tcpip.join(); if(thread_shdmem.joinable()) thread_shdmem.join(); // Asign the final protocol (once both threads finalize) if(retTcpIp == 0) *protocol = X_LINK_TCP_IP; if(retShdmem == 0) *protocol = X_LINK_LOCAL_SHDMEM; // if both connected, close TCP_IP if(retTcpIp == 0 && retShdmem == 0) { tcpipPlatformClose(fdTcpIp); } // None of the connections were successful if(retTcpIp || retShdmem) return -1; ... }

That is if "signaling" works as expected with "accept". Otherwise you might have to switch to select in the servers, for which might be easiest to modify funcs with additional parameter with a fd that one can "unblock" (instead the pthread_kill calls)

Signaling will work, but there's an issue: we don't clean up the socket instantiated with the socket() syscall. To stop the accept call, it is sufficent to use this procedure:

shutdown(sock, SHUT_RDWR); #if defined(SO_LINGER) const int set = 0; setsockopt(sock, SOL_SOCKET, SO_LINGER, (const char*)&set, sizeof(set)); #endif tcpip_close_socket(sock);

Since shutdown blocks any data transfer on the socket and setting SO_LINGER to zero means that when calling close() the socket will just be forced closed.
This has the added benefit of the thread ending gracefully and not needing a SIGINT.
That's the reason detach() was called, because the thread ends normally with no need of intervention.

@TheMutta

AFAIK (as per reading yesterday), accept cannot be unblocked with shutdown
We'd have to switch to select

If above works however, I'm up for that path - rather than signals

That's the reason detach() was called, because the thread ends normally with no need of intervention.

lets join if thread is joinable - rather than detach (if thread actually finished as expected)

The issue seems to be that signals cause issues, they don't work, while the shutdown(),setsockopt() and close() methods in succession do work. While shutdown() isn't the one closing the socket, it's instead the combination of setting SO_LINGER to 0 and then calling close() which forces an immediate closure of the socket, it's sort of a security measure to make sure the socket is neither sending or receiving data and is in the middle of something (if there's a pending connection request and the socket, for example, we don't want it to block in the middle, we can call shutdown and let it complete that request and then just close the socket).

lets join if thread is joinable - rather than detach (if thread actually finished as expected)

Gotcha.

Noted - lets create a duplicate of "tcpipServer" function for this, where we have this (mostly) same logic flow as the original function, but with these changes (to not cause potential issues for the regular TCP_IP flow)

Thanks!

Changes have been pushed

themarpe · 2024-07-03T20:42:55Z

Also, WRT the crosscompilation - any "unix" only things, especially if whole files, can be added either to:

CMake - condiitioanlly compiled
if def'd in/out

To fix the CI

… into shared-memory-on-localhost

TheMutta · 2024-07-04T17:30:28Z

Hmm - do we need a special type of an event?
This will likely not work due to it being shared with (fixed) Bootloader on RVC2 (eg communicating with it will cause issues).

So lets try to either:

Remove the need for custom xLinkEventType entry
reorder them to the end & fix assert (eg, to check between two ranges)

Hmm, I understand the issue. I think option 2 is the most convinient. That said, will the fact that the streamPacketDesc_t structure has been expanded to add the fd field cause issues with the RVC2 bootloader?

…and instead new event types will go at the end of the enum

TheMutta · 2024-07-05T09:40:09Z

I've moved the requests in a way that the first few will remain constant and the new requests can be added without causing any issues for the new ones.

The detection protocol has been fixed with threads, and the conditional compilation has been put in place.

The remaining fail CI's seem to be because of the _findfirst64i32 or the ftruncate function redifiniton, none of which are in the source code.

themarpe

Apart form this LGTM if all works well integrated

themarpe · 2024-07-05T13:03:46Z

examples/CMakeLists.txt

+# Local shared memory example
+add_example(xlink_server_local xlink_server_local.cpp)
+add_example(xlink_client_local xlink_client_local.cpp)


Instead of doing "if unix" in source code, conditionally add these as examples IFF running on UNIX
Apply this elsewhere as well where needed (where we ifdef out a whole file - is better to make it "platform specific compiliation wise")

themarpe · 2024-07-05T13:05:03Z

That said, will the fact that the streamPacketDesc_t structure has been expanded to add the fd field cause issues with the RVC2 bootloader?

No, no issues - just the xLinkEventHeader_t structure and all the downstream types (enums, etc...)

The remaining fail CI's seem to be because of the _findfirst64i32 or the ftruncate function redifiniton, none of which are in the source code.

Might be a bug in the mingw unix "translation" helpers, though I wasn't able to repro on my machine. Lets leave as is, we don't support mingw officially anyway

TheMutta · 2024-07-05T15:13:47Z

Merging the new shared memory zerocopy protocol with the mainline branch

TheMutta added 30 commits June 11, 2024 19:49

Added X_LINK_LOCAL_SHDMEM protocol skeleton

9f17ea0

Examples added.

c4d4c45

Added checks for shdmem functions to compile only in Linux system

03539a3

Removed #warning directive.

283bbdc

Using correct FD for socket and not including headears on non-unix fu…

5562d45

…nctions.

Added XLINK_READ_FD and XLINK_WRITE_FD operations

3faa07c

Removed binary files.

8aa655b

Revert "Removed binary files."

bdbc176

This reverts commit 8aa655b.

Revert "Added XLINK_READ_FD and XLINK_WRITE_FD operations"

4cc25ab

This reverts commit 3faa07c.

Added XLINK_WRITE_FD_REQ with XLinkWriteFd function to send over FDs

4603388

Adding include files with definitions for XLINK_WRITE_FD_REQ with XLi…

685245b

…nkWriteFd

Working examples

43e39d6

Fixed naming convention in the examples.

011eab6

Added examples to CMakeLists.txt

706eea4

Added XLINK_WRITE_FD_RESP

c3f6f3c

Finalized examples.

204f070

SHDMEM protocol output reworked and finalized.

4ce854b

Transferring fd through streamPacket

c082504

Preparations for streaming files over the connection when XLinkWriteF…

0fb3d9a

…d is called on a protocol different from X_LINK_LOCAL_SHDMEM

Updated examples.

3a3c15a

Implemented file streaming on XLinkPlatformWriteFd

ffbda08

Fixed broken CMakeLists.txt

43d2ced

Automatic localhost detection and change to LOCAL_SHDMEM when using T…

8c33a71

…CP_IP

Updated examples

aa45e40

Fixed typo in XLinkPlatformConnect

48b6af8

Set fixed size for fd in streamPacketDesc_t at uint32_t

9fb7537

Implemented XLinkWriteFdData, akin to XLinkWriteData2

0878b3b

Changed fd to int32_t to allow for negative error values

13a02ab

Fixed typo with dataBuffer instead of dataSize.

0a3444a

shmem functions now returning byte count.

fa9bfd9

TheMutta self-assigned this Jul 1, 2024

themarpe requested changes Jul 1, 2024

View reviewed changes

TheMutta added 2 commits July 2, 2024 11:54

Using multiple threads and cond var to connect to TCP/IP or SHDMEM

629691f

Added conditional compilation for shdmemPlatform close

ec429c3

TheMutta added 2 commits July 2, 2024 17:00

Fixed flags for Windows.

89a4c13

Made certain example UNIX only

4e77fc1

themarpe requested changes Jul 3, 2024

View reviewed changes

TheMutta added 4 commits July 4, 2024 10:37

Removed tcpipIsLocalhost function from tcpip_host.cpp

741aff6

Removed tcpipIsLocalhost function from tcpip_host.h

a0b71c4

Added the custom X_LINK_TCP_IP_OR_LOCAL_SHDMEM protocol

fe25ef9

Merge branch 'shared-memory-on-localhost' of github.com:luxonis/XLink…

f017c1a

… into shared-memory-on-localhost

Made sure not to alter the first few event types for legacy systems, …

5e89479

…and instead new event types will go at the end of the enum

themarpe approved these changes Jul 5, 2024

View reviewed changes

Conditional example compilation only for unix

62909dc

TheMutta merged commit ddfc42c into develop_server Jul 5, 2024
10 of 16 checks passed

TheMutta deleted the shared-memory-on-localhost branch July 5, 2024 15:13

TheMutta restored the shared-memory-on-localhost branch July 5, 2024 15:14

TheMutta deleted the shared-memory-on-localhost branch July 12, 2024 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FD sharing protocol on localhost #82

FD sharing protocol on localhost #82

TheMutta commented Jul 1, 2024

themarpe left a comment

TheMutta commented Jul 1, 2024

themarpe commented Jul 1, 2024

TheMutta commented Jul 1, 2024 •

edited

Loading

TheMutta commented Jul 1, 2024

themarpe left a comment

themarpe Jul 1, 2024

TheMutta Jul 1, 2024

themarpe Jul 3, 2024

themarpe Jul 3, 2024

TheMutta commented Jul 1, 2024 •

edited

Loading

TheMutta commented Jul 2, 2024

themarpe Jul 3, 2024

themarpe Jul 3, 2024

themarpe Jul 3, 2024

TheMutta Jul 4, 2024

themarpe Jul 3, 2024

TheMutta Jul 4, 2024

themarpe Jul 3, 2024

themarpe Jul 3, 2024

TheMutta Jul 4, 2024 •

edited

Loading

themarpe Jul 4, 2024

TheMutta Jul 4, 2024

themarpe Jul 4, 2024

TheMutta Jul 4, 2024

themarpe commented Jul 3, 2024

TheMutta commented Jul 4, 2024

TheMutta commented Jul 5, 2024

themarpe left a comment

themarpe Jul 5, 2024

themarpe commented Jul 5, 2024

TheMutta commented Jul 5, 2024

		@@ -1163,6 +1233,16 @@ int tcpipPlatformBootFirmware(const deviceDesc_t* deviceDesc, const char* firmwa
		return -1;
		}

FD sharing protocol on localhost #82

FD sharing protocol on localhost #82

Conversation

TheMutta commented Jul 1, 2024

themarpe left a comment

Choose a reason for hiding this comment

TheMutta commented Jul 1, 2024

themarpe commented Jul 1, 2024

TheMutta commented Jul 1, 2024 • edited Loading

TheMutta commented Jul 1, 2024

themarpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheMutta commented Jul 1, 2024 • edited Loading

TheMutta commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheMutta Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themarpe commented Jul 3, 2024

TheMutta commented Jul 4, 2024

TheMutta commented Jul 5, 2024

themarpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themarpe commented Jul 5, 2024

TheMutta commented Jul 5, 2024

TheMutta commented Jul 1, 2024 •

edited

Loading

TheMutta commented Jul 1, 2024 •

edited

Loading

TheMutta Jul 4, 2024 •

edited

Loading