From 8ecc915cbe32ad0f00e5c6c4d444e9cc7ba45502 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Wed, 8 Jan 2025 23:48:11 -0500 Subject: [PATCH 1/7] add cluster documentation --- docs/install/on-prem/cluster.md | 275 ++++++++++++++++++++++++++++++++ docs/install/on-prem/index.rst | 1 + 2 files changed, 276 insertions(+) create mode 100644 docs/install/on-prem/cluster.md diff --git a/docs/install/on-prem/cluster.md b/docs/install/on-prem/cluster.md new file mode 100644 index 0000000..75a582e --- /dev/null +++ b/docs/install/on-prem/cluster.md @@ -0,0 +1,275 @@ +# Multinode Deployment with Docker Compose + +This document offers step-by-step instructions for deploying **Graphistry** in a multinode environment using Docker Compose. In this architecture, a **leader** node handles dataset ingestion and manages the single PostgreSQL instance, while **follower** nodes can visualize graphs too using the shared datasets. Currently, only the leader node has permission to upload datasets and files (data ingestion), but future updates will allow follower nodes to also perform dataset and file uploads (data ingestion). + +The leader and followers will share datasets using a **Distributed File System**, for example, using the Network File System (NFS) protocol. This setup allows all nodes to access the same dataset directory. This configuration ensures that **Graphistry** can be deployed across multiple machines, each with different GPU configuration profiles (some with more powerful GPUs, enabling multi-GPU on multinode setups), while keeping the dataset storage centralized and synchronized. + + +### Cluster Configuration Overview + +1. **Leader Node**: Handles the ingestion of datasets, PostgreSQL write operations, and exposes the required PostgreSQL ports. +2. **Follower Nodes**: Connect to the PostgreSQL instance on the leader and can visualize graphs using the shared datasets. However, they do not have their own attached PostgreSQL instance. +3. **Shared Dataset**: All nodes will access the dataset directory using a **Distributed File System**. This ensures that the leader and followers use the same dataset, maintaining consistency across all nodes. +4. **PostgreSQL**: The PostgreSQL instance on the leader node is used by all nodes in the cluster for querying. The **Nexus** service, which provides the main dashboard for Graphistry, on the **Leader** node is responsible for managing access to the PostgreSQL database. The **Nexus** services on the **follower** nodes will use the PostgreSQL instance of the **Leader**. + +### Configuration File: `cluster.env` + +The configuration for multinode deployment is managed through the environment file `cluster.env`. This file will vary depending on whether the node is a **leader** or a **follower** in the deployment. It defines key settings such as the node type (`leader` or `follower`), the shared dataset directory, and the single shared PostgreSQL connection. + +```bash +# Graphistry Cluster Configuration +# +# https://github.com/graphistry/graphistry-cli/blob/master/docs/cluster.md + + +# ENABLE_CLUSTER_MODE: true | false (default false) +# All nodes will use the same LOCAL_DATASET_CACHE_DIR and postgres instance (the leader one). +ENABLE_CLUSTER_MODE=false + +# NODE_TYPE: leader | follower (default leader) +# If ENABLE_CLUSTER_MODE=true and NODE_TYPE=leader it exposes the postgres ports to the host (LAN). +# If ENABLE_CLUSTER_MODE=true and NODE_TYPE=follower the follower node won't start the postgres service. +NODE_TYPE=leader + +# LOCAL_DATASET_CACHE_DIR=/host/path/ (default: /opt/graphistry/data) +# This environment variable can be defined in other configuration files, +# but the value here will take precedence and override those settings (e.g., custom.env). +# Examples: +# When NODE_TYPE=leader, LOCAL_DATASET_CACHE_DIR may point to an NFS path for local data storage. +# When NODE_TYPE=follower, LOCAL_DATASET_CACHE_DIR may point to a mount path that references the leader's network file system (e.g. the NFS path). +LOCAL_DATASET_CACHE_DIR=/opt/graphistry/data + +# POSTGRES_HOST=ip | host_url (default: postgres) +# If NODE_TYPE=follower it will use this env var to setup the postgres remote conn string. +# This environment variable can be defined in other configuration files, +# but the value here will take precedence and override those settings (e.g., custom.env). +# Examples: +# When NODE_TYPE=leader, POSTGRES_HOST may be postgres (.i.e. the internal postgres service that Graphistry deploys). +# When NODE_TYPE=follower, POSTGRES_HOST may point to the host where the leader is running. +POSTGRES_HOST=postgres +``` + +## Setup Instructions + +For this setup example, we will use the **Network File System (NFS)**, but any **Distributed File System** can be used to achieve the same goal. The file system must ensure that all nodes in the cluster can access the shared dataset directory. We will use **Ubuntu 22.04** on both the follower and leader nodes, with the follower having the IP address **192.168.18.8** and the leader **192.168.18.13**. + +### Step 1: Configure the NFS Shared Directory + +NFS will be used to share the dataset directory between nodes. Follow the steps below to set up NFS on both the leader and follower machines. + +#### On the Leader Node (Main Machine) + +1. **Create directories for PostgreSQL data and backups**: + + ```bash + mkdir -p /mnt/data/shared/postgresql_data + mkdir -p /mnt/data/shared/postgres_backups + ``` + + These directories will hold the PostgreSQL data and backups, which will be shared with follower nodes. + +2. **Install NFS server**: + + On the leader node, install the NFS server software: + + ```bash + sudo apt install nfs-kernel-server + ``` + + This will install the necessary software for serving NFS shares to the follower nodes. + +3. **Configure NFS exports**: + + Edit the `/etc/exports` file to specify which directories should be shared and with what permissions. The following configuration allows the follower node (with IP `192.168.18.8`) to mount the shared directory with read/write permissions. + + ```bash + sudo nano /etc/exports + ``` + + Add the following line to export the shared dataset directory: + + ```bash + /mnt/data/shared/ 192.168.18.8(rw,sync,no_subtree_check) + ``` + + - `rw`: Allows read and write access. + - `sync`: Ensures that changes are written to disk before responding to the client. + - `no_subtree_check`: Disables subtree checking to improve performance. + +4. **Export the NFS share** and restart the NFS server to apply the changes: + + ```bash + sudo exportfs -a + sudo systemctl restart nfs-kernel-server + ``` + +5. **Disable the firewall** (if enabled) to ensure that NFS traffic is not blocked between nodes: + + Check the status of `ufw` (Uncomplicated Firewall): + + ```bash + sudo ufw status + ``` + + If the firewall is enabled, disable it with: + + ```bash + sudo ufw disable + ``` + + Alternatively, you can allow NFS traffic through the firewall, but disabling it is the easiest way to ensure communication between the nodes. + +#### On the Follower Node (Secondary Machine) + +1. **Create a directory to mount the NFS share**: + + ```bash + mkdir -p /home/user1/mnt/data/shared/ + ``` + + This is where the shared dataset will be mounted on the follower node. + +2. **Install NFS client**: + + On the follower node, install the NFS client software to mount the NFS share: + + ```bash + sudo apt install nfs-common + ``` + +3. **Mount the shared NFS directory**: + + Mount the directory shared by the leader node to the local directory on the follower node: + + ```bash + sudo mount -t nfs 192.168.18.13:/mnt/data/shared/ /home/user1/mnt/data/shared/ + ``` + + - Replace `192.168.18.13` with the IP address of the leader node. + - This command mounts the NFS share to `/home/user1/mnt/data/shared/` on the follower node. + +4. **Verify the mount**: + + You can verify that the directory is mounted correctly using the following command: + + ```bash + mount -l | grep /mnt/data/shared + ``` + + This should show an entry like: + + ```bash + 192.168.18.13:/mnt/data/shared on /home/user1/mnt/data/shared type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.18.8,local_lock=none,addr=192.168.18.13) + ``` + + This confirms that the NFS share is mounted and ready to use. + +--- + +### Step 2: Docker Compose Setup + +Now that the NFS share is set up, we can configure **Docker Compose** for **Graphistry**. Both the leader and follower nodes will utilize their own `cluster.env` file to define environment variables and configure the deployment. + +1. **Ensure the correct configuration** for the `cluster.env` file. This file should contain the appropriate settings for **multinode mode**, **node type** (leader or follower), the **shared dataset directory**, and the **PostgreSQL connection**. + + Example of `cluster.env` for the leader node: + + ```bash + ENABLE_CLUSTER_MODE=true + NODE_TYPE=leader + LOCAL_DATASET_CACHE_DIR=/mnt/data/shared/ + POSTGRES_HOST=postgres + ``` + + Example of `cluster.env` for a follower node: + + ```bash + ENABLE_CLUSTER_MODE=true + NODE_TYPE=follower + LOCAL_DATASET_CACHE_DIR=/home/user1/mnt/data/shared/ + POSTGRES_HOST=192.168.18.13 + ``` + +2. **Start Docker Compose**: + + On the leader node, run the following command to start the Docker Compose instance: + + ```bash + ./release up -d + ``` + + On each follower node, run the same command: + + ```bash + ./release up -d + ``` + + This will start the **Graphistry** containers across all nodes, enabling them to connect to the PostgreSQL instance on the leader node. If the leader is not ready, the followers will wait for the PostgreSQL service to become available. Once the leader is online, the followers will resume their operations, ensuring a smooth startup. For example, follower nodes will log messages like: + + + ```javascript + compose-forge-etl-python-1 | 2025-01-08T00:37:51.432950416Z Waiting for PostgreSQL to become available... + compose-streamgl-viz-1 | 2025-01-08T00:37:51.433181258Z Waiting for PostgreSQL to become available... + compose-pivot-1 | 2025-01-08T00:37:51.820240166Z Waiting for PostgreSQL to become available... + compose-forge-etl-1 | 2025-01-08T00:37:51.820134913Z Waiting for PostgreSQL to become available... + ``` + +### Step 3: Verifying the Setup + +After the deployment, ensure that the following checks are in place: + +1. **Leader Node**: + - The leader node should be running the PostgreSQL instance. + - The dataset ingestion feature should be available, and you can upload datasets to the shared NFS directory. + - The leader should be exposed to the host for PostgreSQL connections. + +2. **Follower Nodes**: + - Followers should be able to access the dataset via the NFS mount and create graphs. + - Followers should connect to the PostgreSQL instance on the leader node but will not be able to perform write operations (ingestion). + +To verify the operation, you can check the logs of each node using: + +```bash +./release logs +``` + +### Some utilities for NFS Management + +- **Unmounting NFS** on the follower node: + + ```bash + sudo umount /home/user1/mnt/data/shared/ + ``` + +- **Changing NFS permissions** (e.g., making the share read-only for the follower): + + 1. Edit the `/etc/exports` file on the leader node: + + ```bash + sudo nano /etc/exports + ``` + + 2. Change the permissions from read-write to read-only: + + ```bash + /mnt/data/shared/ 192.168.18.8(ro,sync,no_subtree_check) + ``` + + 3. Apply the changes: + + ```bash + sudo exportfs -ra + sudo systemctl restart nfs-kernel-server + ``` + + 4. On the follower node, remount the NFS share to apply the changes. + +--- + +### Troubleshooting + +- **Mounting Issues**: If the NFS mount does not appear or fails, verify the IP addresses and paths in the `/etc/exports` file on the leader node. Ensure that the follower node has access to the shared directory. + +- **Firewall Issues**: Ensure that firewalls are disabled on both the leader and follower nodes or that the NFS ports are allowed. If the firewall is enabled, use `ufw` or `iptables` to allow NFS traffic. + +- **Permission Issues**: If permission errors occur when accessing the shared directory, check the directory permissions on the leader node and ensure they are accessible by the user running Docker on the follower node. diff --git a/docs/install/on-prem/index.rst b/docs/install/on-prem/index.rst index 9b926d0..6fbff43 100644 --- a/docs/install/on-prem/index.rst +++ b/docs/install/on-prem/index.rst @@ -8,6 +8,7 @@ On-Prem Installation rhel_7_6_setup ubuntu_18_04_lts_setup vGPU + cluster From 6c9d67b2fa86fb0225657c8e28a61b7c06c04199 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Thu, 9 Jan 2025 00:39:42 -0500 Subject: [PATCH 2/7] improve cluster docs --- docs/install/on-prem/cluster.md | 68 ++++++++++++--------------------- 1 file changed, 25 insertions(+), 43 deletions(-) diff --git a/docs/install/on-prem/cluster.md b/docs/install/on-prem/cluster.md index 75a582e..bbdfd81 100644 --- a/docs/install/on-prem/cluster.md +++ b/docs/install/on-prem/cluster.md @@ -51,7 +51,7 @@ POSTGRES_HOST=postgres ## Setup Instructions -For this setup example, we will use the **Network File System (NFS)**, but any **Distributed File System** can be used to achieve the same goal. The file system must ensure that all nodes in the cluster can access the shared dataset directory. We will use **Ubuntu 22.04** on both the follower and leader nodes, with the follower having the IP address **192.168.18.8** and the leader **192.168.18.13**. +For this setup example, we will use the **Network File System (NFS)**, but any **Distributed File System** can be used to achieve the same goal. The file system must ensure that all nodes in the cluster can access the shared dataset directory. We will use **Ubuntu 22.04** on both the follower and leader nodes, with the follower having the IP address **192.168.0.20** and the leader **192.168.0.10**. ### Step 1: Configure the NFS Shared Directory @@ -80,7 +80,7 @@ NFS will be used to share the dataset directory between nodes. Follow the steps 3. **Configure NFS exports**: - Edit the `/etc/exports` file to specify which directories should be shared and with what permissions. The following configuration allows the follower node (with IP `192.168.18.8`) to mount the shared directory with read/write permissions. + Edit the `/etc/exports` file to specify which directories should be shared and with what permissions. The following configuration allows the follower node (with IP `192.168.0.20`) to mount the shared directory with read/write permissions. ```bash sudo nano /etc/exports @@ -89,7 +89,7 @@ NFS will be used to share the dataset directory between nodes. Follow the steps Add the following line to export the shared dataset directory: ```bash - /mnt/data/shared/ 192.168.18.8(rw,sync,no_subtree_check) + /mnt/data/shared/ 192.168.0.20(rw,sync,no_subtree_check) ``` - `rw`: Allows read and write access. @@ -142,10 +142,10 @@ NFS will be used to share the dataset directory between nodes. Follow the steps Mount the directory shared by the leader node to the local directory on the follower node: ```bash - sudo mount -t nfs 192.168.18.13:/mnt/data/shared/ /home/user1/mnt/data/shared/ + sudo mount -t nfs 192.168.0.10:/mnt/data/shared/ /home/user1/mnt/data/shared/ ``` - - Replace `192.168.18.13` with the IP address of the leader node. + - Replace `192.168.0.10` with the IP address of the leader node. - This command mounts the NFS share to `/home/user1/mnt/data/shared/` on the follower node. 4. **Verify the mount**: @@ -159,7 +159,7 @@ NFS will be used to share the dataset directory between nodes. Follow the steps This should show an entry like: ```bash - 192.168.18.13:/mnt/data/shared on /home/user1/mnt/data/shared type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.18.8,local_lock=none,addr=192.168.18.13) + 192.168.0.10:/mnt/data/shared on /home/user1/mnt/data/shared type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.20,local_lock=none,addr=192.168.0.10) ``` This confirms that the NFS share is mounted and ready to use. @@ -172,7 +172,7 @@ Now that the NFS share is set up, we can configure **Docker Compose** for **Grap 1. **Ensure the correct configuration** for the `cluster.env` file. This file should contain the appropriate settings for **multinode mode**, **node type** (leader or follower), the **shared dataset directory**, and the **PostgreSQL connection**. - Example of `cluster.env` for the leader node: + Example of `cluster.env` for the **leader** node: ```bash ENABLE_CLUSTER_MODE=true @@ -181,24 +181,18 @@ Now that the NFS share is set up, we can configure **Docker Compose** for **Grap POSTGRES_HOST=postgres ``` - Example of `cluster.env` for a follower node: + Example of `cluster.env` for a **follower** node: ```bash ENABLE_CLUSTER_MODE=true NODE_TYPE=follower LOCAL_DATASET_CACHE_DIR=/home/user1/mnt/data/shared/ - POSTGRES_HOST=192.168.18.13 + POSTGRES_HOST=192.168.0.10 ``` 2. **Start Docker Compose**: - On the leader node, run the following command to start the Docker Compose instance: - - ```bash - ./release up -d - ``` - - On each follower node, run the same command: + On the leader and on each follower node, run the following command to start the Docker Compose instance: ```bash ./release up -d @@ -233,40 +227,28 @@ To verify the operation, you can check the logs of each node using: ./release logs ``` -### Some utilities for NFS Management - -- **Unmounting NFS** on the follower node: - - ```bash - sudo umount /home/user1/mnt/data/shared/ - ``` - -- **Changing NFS permissions** (e.g., making the share read-only for the follower): - - 1. Edit the `/etc/exports` file on the leader node: +## Usage - ```bash - sudo nano /etc/exports - ``` +Once the deployment is complete, you can use the leader node to upload datasets, files and perform other data ingestion tasks. The `VISUALIZE FILES (BETA)` feature in Graphistry can be used to upload graph datasets and files. Additionally, you can use the Graphistry Clients (such as `pygraphistry`, `graphistry-js`) or the `REST API` to interact with the data (all of them pointing to the IP/address of the leader): - 2. Change the permissions from read-write to read-only: +* PyGraphistry: https://github.com/graphistry/pygraphistry +* Graphistry JS: https://github.com/graphistry/graphistry-js +* REST API: API Docs: https://hub.graphistry.com/docs/api - ```bash - /mnt/data/shared/ 192.168.18.8(ro,sync,no_subtree_check) - ``` +For example, you can interact with the leader node from **PyGraphistry** like this: - 3. Apply the changes: - - ```bash - sudo exportfs -ra - sudo systemctl restart nfs-kernel-server - ``` +```python +import graphistry +leader_address = "192.168.0.10" +graphistry.register(api=3, protocol="http", server=leader_address, username="user1", password="password1") +... +``` - 4. On the follower node, remount the NFS share to apply the changes. +Once the upload is finished, these datasets and files will be available on all follower nodes and the leader for visualization. Each graph session on any instance is independent by default. This means that visualizations on the leader and follower nodes are isolated from one another. However, collaborative features will be enabled if users are pointed to the same instance (leader or follower). In this case, multiple users can interact with the same visualization, sharing insights and collaborating in real-time. ---- +This setup provides flexibility for both individual exploration and team collaboration, while ensuring that the data and visualizations remain synchronized across the deployment. It also provides high availability and better scalability for Graphistry deployments. -### Troubleshooting +## Troubleshooting - **Mounting Issues**: If the NFS mount does not appear or fails, verify the IP addresses and paths in the `/etc/exports` file on the leader node. Ensure that the follower node has access to the shared directory. From 2b8cf91efc2af73cb600ba0a51bb4715f521725b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Thu, 9 Jan 2025 20:31:46 -0500 Subject: [PATCH 3/7] minor improvement --- docs/install/on-prem/cluster.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/install/on-prem/cluster.md b/docs/install/on-prem/cluster.md index bbdfd81..1f17660 100644 --- a/docs/install/on-prem/cluster.md +++ b/docs/install/on-prem/cluster.md @@ -242,9 +242,8 @@ import graphistry leader_address = "192.168.0.10" graphistry.register(api=3, protocol="http", server=leader_address, username="user1", password="password1") ... -``` -Once the upload is finished, these datasets and files will be available on all follower nodes and the leader for visualization. Each graph session on any instance is independent by default. This means that visualizations on the leader and follower nodes are isolated from one another. However, collaborative features will be enabled if users are pointed to the same instance (leader or follower). In this case, multiple users can interact with the same visualization, sharing insights and collaborating in real-time. +Once the upload is finished, these datasets and files will be available on all follower nodes and the leader for visualization. Each graph session on any instance is independent by default. This means that visualizations on the leader and follower nodes are isolated from one another. However, collaborative features will be enabled if users are pointed to the same instance (leader or follower). In this case, multiple users can interact with the same visualization, sharing insights and collaborating in real-time. Additionally, both the leader and follower nodes will have the ability to delete shared datasets and files using the Nexus dashboard, ensuring that data management can be handled across the entire deployment. This setup provides flexibility for both individual exploration and team collaboration, while ensuring that the data and visualizations remain synchronized across the deployment. It also provides high availability and better scalability for Graphistry deployments. From 396cbc3ea1cdc18f561cf3b7b873b929c23734b8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Thu, 9 Jan 2025 20:34:36 -0500 Subject: [PATCH 4/7] fix Sphinx warnings for doc generation --- docs/install/on-prem/cluster.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/install/on-prem/cluster.md b/docs/install/on-prem/cluster.md index 1f17660..456bd18 100644 --- a/docs/install/on-prem/cluster.md +++ b/docs/install/on-prem/cluster.md @@ -5,14 +5,14 @@ This document offers step-by-step instructions for deploying **Graphistry** in a The leader and followers will share datasets using a **Distributed File System**, for example, using the Network File System (NFS) protocol. This setup allows all nodes to access the same dataset directory. This configuration ensures that **Graphistry** can be deployed across multiple machines, each with different GPU configuration profiles (some with more powerful GPUs, enabling multi-GPU on multinode setups), while keeping the dataset storage centralized and synchronized. -### Cluster Configuration Overview +## Cluster Configuration Overview 1. **Leader Node**: Handles the ingestion of datasets, PostgreSQL write operations, and exposes the required PostgreSQL ports. 2. **Follower Nodes**: Connect to the PostgreSQL instance on the leader and can visualize graphs using the shared datasets. However, they do not have their own attached PostgreSQL instance. 3. **Shared Dataset**: All nodes will access the dataset directory using a **Distributed File System**. This ensures that the leader and followers use the same dataset, maintaining consistency across all nodes. 4. **PostgreSQL**: The PostgreSQL instance on the leader node is used by all nodes in the cluster for querying. The **Nexus** service, which provides the main dashboard for Graphistry, on the **Leader** node is responsible for managing access to the PostgreSQL database. The **Nexus** services on the **follower** nodes will use the PostgreSQL instance of the **Leader**. -### Configuration File: `cluster.env` +## Configuration File: `cluster.env` The configuration for multinode deployment is managed through the environment file `cluster.env`. This file will vary depending on whether the node is a **leader** or a **follower** in the deployment. It defines key settings such as the node type (`leader` or `follower`), the shared dataset directory, and the single shared PostgreSQL connection. @@ -242,6 +242,7 @@ import graphistry leader_address = "192.168.0.10" graphistry.register(api=3, protocol="http", server=leader_address, username="user1", password="password1") ... +``` Once the upload is finished, these datasets and files will be available on all follower nodes and the leader for visualization. Each graph session on any instance is independent by default. This means that visualizations on the leader and follower nodes are isolated from one another. However, collaborative features will be enabled if users are pointed to the same instance (leader or follower). In this case, multiple users can interact with the same visualization, sharing insights and collaborating in real-time. Additionally, both the leader and follower nodes will have the ability to delete shared datasets and files using the Nexus dashboard, ensuring that data management can be handled across the entire deployment. From 987d3fcd17a3957601a8b561e824e4b52cede50b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Thu, 9 Jan 2025 22:07:38 -0500 Subject: [PATCH 5/7] update footer info and move cluster to a more general space to include cloud use cases --- docs/admin.rst | 1 + docs/conf.py | 2 +- docs/install/{on-prem => }/cluster.md | 0 docs/install/on-prem/index.rst | 1 - 4 files changed, 2 insertions(+), 2 deletions(-) rename docs/install/{on-prem => }/cluster.md (100%) diff --git a/docs/admin.rst b/docs/admin.rst index 3136b39..7a37657 100644 --- a/docs/admin.rst +++ b/docs/admin.rst @@ -13,6 +13,7 @@ install/cloud/index install/on-prem/index install/testing-an-install + install/cluster app-config/index debugging/index security/index diff --git a/docs/conf.py b/docs/conf.py index 6e94394..e1f3e1c 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -29,7 +29,7 @@ # -- Project information ----------------------------------------------------- project = "Graphistry CLI" -copyright = "2024, Graphistry, Inc." +copyright = "2025, Graphistry, Inc." author = "Graphistry, Inc." html_title = "Graphistry Administration Documentation" diff --git a/docs/install/on-prem/cluster.md b/docs/install/cluster.md similarity index 100% rename from docs/install/on-prem/cluster.md rename to docs/install/cluster.md diff --git a/docs/install/on-prem/index.rst b/docs/install/on-prem/index.rst index 6fbff43..9b926d0 100644 --- a/docs/install/on-prem/index.rst +++ b/docs/install/on-prem/index.rst @@ -8,7 +8,6 @@ On-Prem Installation rhel_7_6_setup ubuntu_18_04_lts_setup vGPU - cluster From 9785239abdd4ad4e39de5db43e7d069b6e7f41aa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Thu, 9 Jan 2025 22:15:37 -0500 Subject: [PATCH 6/7] add cloud info --- docs/install/cluster.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/install/cluster.md b/docs/install/cluster.md index 456bd18..8ee5cef 100644 --- a/docs/install/cluster.md +++ b/docs/install/cluster.md @@ -4,6 +4,7 @@ This document offers step-by-step instructions for deploying **Graphistry** in a The leader and followers will share datasets using a **Distributed File System**, for example, using the Network File System (NFS) protocol. This setup allows all nodes to access the same dataset directory. This configuration ensures that **Graphistry** can be deployed across multiple machines, each with different GPU configuration profiles (some with more powerful GPUs, enabling multi-GPU on multinode setups), while keeping the dataset storage centralized and synchronized. +This deployment mode is flexible and can be used both in **on-premises** clusters or in the **cloud**. For example, it should be possible to use **Amazon Machine Images (AMIs)** from the [Graphistry AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-ppbjy2nny7xzk?sr=0-1&ref_=beagle&applicationId=AWSMPContessa), assigning Amazon VMs created from those images to the **leader** and **follower** roles. This allows for scalable and customizable cloud-based deployments with the same multinode architecture. ## Cluster Configuration Overview From ed29797213c5ea0e71ac6937fb4f90d3a66507b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Percy=20Camilo=20Trive=C3=B1o=20Aucahuasi?= Date: Fri, 10 Jan 2025 13:36:14 -0500 Subject: [PATCH 7/7] improve firewall instructions (PROD system should not disable it) and mark the feature as experimental --- docs/install/cluster.md | 24 +++++------------------- 1 file changed, 5 insertions(+), 19 deletions(-) diff --git a/docs/install/cluster.md b/docs/install/cluster.md index 8ee5cef..2421fe0 100644 --- a/docs/install/cluster.md +++ b/docs/install/cluster.md @@ -1,5 +1,7 @@ # Multinode Deployment with Docker Compose +**Note**: *This deployment configuration is currently **experimental** and subject to future updates.* + This document offers step-by-step instructions for deploying **Graphistry** in a multinode environment using Docker Compose. In this architecture, a **leader** node handles dataset ingestion and manages the single PostgreSQL instance, while **follower** nodes can visualize graphs too using the shared datasets. Currently, only the leader node has permission to upload datasets and files (data ingestion), but future updates will allow follower nodes to also perform dataset and file uploads (data ingestion). The leader and followers will share datasets using a **Distributed File System**, for example, using the Network File System (NFS) protocol. This setup allows all nodes to access the same dataset directory. This configuration ensures that **Graphistry** can be deployed across multiple machines, each with different GPU configuration profiles (some with more powerful GPUs, enabling multi-GPU on multinode setups), while keeping the dataset storage centralized and synchronized. @@ -54,6 +56,8 @@ POSTGRES_HOST=postgres For this setup example, we will use the **Network File System (NFS)**, but any **Distributed File System** can be used to achieve the same goal. The file system must ensure that all nodes in the cluster can access the shared dataset directory. We will use **Ubuntu 22.04** on both the follower and leader nodes, with the follower having the IP address **192.168.0.20** and the leader **192.168.0.10**. +Additionally, ensure that the **firewall** on both the leader and follower nodes is configured to allow **NFS** traffic on the necessary ports (e.g., 2049), enabling seamless communication between the nodes. + ### Step 1: Configure the NFS Shared Directory NFS will be used to share the dataset directory between nodes. Follow the steps below to set up NFS on both the leader and follower machines. @@ -104,22 +108,6 @@ NFS will be used to share the dataset directory between nodes. Follow the steps sudo systemctl restart nfs-kernel-server ``` -5. **Disable the firewall** (if enabled) to ensure that NFS traffic is not blocked between nodes: - - Check the status of `ufw` (Uncomplicated Firewall): - - ```bash - sudo ufw status - ``` - - If the firewall is enabled, disable it with: - - ```bash - sudo ufw disable - ``` - - Alternatively, you can allow NFS traffic through the firewall, but disabling it is the easiest way to ensure communication between the nodes. - #### On the Follower Node (Secondary Machine) 1. **Create a directory to mount the NFS share**: @@ -165,8 +153,6 @@ NFS will be used to share the dataset directory between nodes. Follow the steps This confirms that the NFS share is mounted and ready to use. ---- - ### Step 2: Docker Compose Setup Now that the NFS share is set up, we can configure **Docker Compose** for **Graphistry**. Both the leader and follower nodes will utilize their own `cluster.env` file to define environment variables and configure the deployment. @@ -253,6 +239,6 @@ This setup provides flexibility for both individual exploration and team collabo - **Mounting Issues**: If the NFS mount does not appear or fails, verify the IP addresses and paths in the `/etc/exports` file on the leader node. Ensure that the follower node has access to the shared directory. -- **Firewall Issues**: Ensure that firewalls are disabled on both the leader and follower nodes or that the NFS ports are allowed. If the firewall is enabled, use `ufw` or `iptables` to allow NFS traffic. +- **Firewall Issues**: Ensure that the firewall on both the leader and follower nodes is properly configured to allow NFS traffic. Use tools like `ufw` or `iptables` to open the necessary NFS ports (e.g., 2049) based on your installation. - **Permission Issues**: If permission errors occur when accessing the shared directory, check the directory permissions on the leader node and ensure they are accessible by the user running Docker on the follower node.