Split cache config from queue config. (#902)

* Split out queue and cache config

* Update usages of cache config,

* Update default

* Cleanup

* Make queue optional.

* config updates.

* changelog

* update spec config

* Update tests

* tweak import

* Update default config.

* fixup test

* Update config.sample.yml

Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
Signed-off-by: Will Hunt <will@half-shot.uk>

* Update encryption.md

Signed-off-by: Will Hunt <will@half-shot.uk>

* Clear up worker config

Signed-off-by: Will Hunt <will@half-shot.uk>

* Update src/config/Config.ts

Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
Signed-off-by: Will Hunt <will@half-shot.uk>

* update helm config

* lint

* fix meta

* tidy tidy

* revert logging change

* lint rust

---------

Signed-off-by: Will Hunt <will@half-shot.uk>
Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>

changelog.d/902.removal

@@ -0,0 +1,3 @@
+The cache/queue configuration has been changed in this release. The `queue.monolithic` option has been deprecated, in place of a dedicated `cache`
+config section. Check the ["Cache configuration" section](https://matrix-org.github.io/matrix-hookshot/latest/setup.html#cache-configuration) for
+more information on how to configure Hookshot caches.

config.sample.yml

@@ -134,12 +134,15 @@ listeners:
 #  # (Optional) Prometheus metrics support
 #  enabled: true
 
+#cache:
+#  # (Optional) Cache options for large scale deployments. 
+#  #    For encryption to work, this must be configured.
+#  redisUri: redis://localhost:6379
+
 #queue:
-#  # (Optional) Message queue / cache configuration options for large scale deployments.
-#  # For encryption to work, must be set to monolithic mode and have a host & port specified.
-#  monolithic: true
-#  port: 6379
-#  host: localhost
+#  # (Optional) Message queue configuration options for large scale deployments.
+#  # For encryption to work, this must not be configured.
+#  redisUri: redis://localhost:6379
 
 #widgets:
 #  # (Optional) EXPERIMENTAL support for complimentary widgets

docs/advanced/encryption.md

@@ -13,7 +13,7 @@ Hookshot supports end-to-bridge encryption via [MSC3202](https://github.com/matr
 In order for Hookshot to use encryption, it must be configured as follows:
 - The `experimentalEncryption.storagePath` setting must point to a directory that Hookshot has permissions to write files into. If running with Docker, this path should be within a volume (for persistency). Hookshot uses this directory for its crypto store (i.e. long-lived state relating to its encryption keys).
     - Once a crypto store has been initialized, its files must not be modified, and Hookshot cannot be configured to use another crypto store of the same type as one it has used before. If a crypto store's files get lost or corrupted, Hookshot may fail to start up, or may be unable to decrypt command messages. To fix such issues, stop Hookshot, then reset its crypto store by running `yarn start:resetcrypto`.
-- [Redis](./workers.md) must be enabled. Note that worker mode is not yet supported with encryption, so `queue.monolithic` must be set to `true`.
+- [Redis](./workers.md) must be enabled. Note that worker mode is not yet supported with encryption, so `queue` MUST **NOT be configured**.
 
 If you ever reset your homeserver's state, ensure you also reset Hookshot's encryption state. This includes clearing the `experimentalEncryption.storagePath` directory and all worker state stored in your redis instance. Otherwise, Hookshot may fail on start up with registration errors.
 

docs/advanced/workers.md

@@ -11,18 +11,19 @@ This feature is <b>experimental</b> and should only be used when you are reachin
 
 You must first have a working Redis instance somewhere which can talk between processes. For example, in Docker you can run:
 
-`docker run --name github-bridge-redis -p 6379:6379 -d redis`.
+`docker run --name redis-host -p 6379:6379 -d redis`.
 
 The processes should all share the same config, which should contain the correct config to enable Redis:
 
 ```yaml
 queue:
-  monolithic: false
-  port: 6379
-  host: github-bridge-redis
+  redisUri: "redis://redis-host:6379"
+cache:
+  redisUri: "redis://redis-host:6379"
 ```
 
-Note that if [encryption](./encryption.md) is enabled, `queue.monolithic` must be set to `true`, as worker mode is not yet supported with encryption.
+Note that if [encryption](./encryption.md) is enabled, you MUST enable the `cache` config but NOT the `queue` config. Workers require persistent
+storage in Redis, but cannot make use of worker-mode queues.
 
 Once that is done, you can simply start the processes by name using yarn:
 ```

docs/setup.md

@@ -227,6 +227,20 @@ Please note that the appservice HTTP listener is configured <strong>separately</
 in the upstream library. See <a href="https://github.com/turt2live/matrix-bot-sdk/issues/191">this issue</a> for details.
 </section>
 
+### Cache configuration
+
+You can optionally enable a Redis-backed cache for Hookshot. This is generally a good thing to enable if you can
+afford to, as it will generally improve startup times. Some features such as resuming RSS/Atom feeds between restarts
+is also only possible with a external cache.
+
+To enable, simply set:
+
+```yaml
+cache:
+  redisUri: "redis://redis-host:3679"
+```
+
+
 ### Services configuration
 
 You will need to configure some services. Each service has its own documentation file inside the setup subdirectory.