Spago docs, now with indexing and search

[CharlesTaylor7]

Part of #1020

How to demo:

# checkout my branch
git checkout spago-docs-search

# install new punycode dep, transitive dep of markdown-it
npm i

# bootstrap spago
./bin/bundle.js build

# bundle spago
./bin/index.dev.js bundle -p spago-bin

# bundle docs-search
./bin/bundle.js bundle -p docs-search-client-halogen

# run docs command
./bin/bundle.js docs --open

You should see output like:

Generating documentation for the project. This might take a while...
[1 of 1] Compiling documentation for Docs.Search.App
Documentation written to: generated-docs/html
Building the search index...
"bower.json" decoding failed for .spago/packages/avar-5.0.0/bower.json: An error occurred while decoding a JSON value:
  Expected value of type 'ROOT."main" - Expected a value of type Array but got String'.
Indexing 1004 modules from 123 packages...
Added 8062 definitions and 9634 type definitions from 123 packages to the search index.
Link: file:///Users/chuck/repos/spago/generated-docs/html/index.html

For the issue decoding the bower.json avar-5.0.0, I opened an upstream PR to fix. However it is just a log message and the indexing continues in the presence of the error.

Click me for a Gif Demo

Checklist:

• Added the change to the "Unreleased" section of the changelog
• Added some example of the new feature to the README
• Added a test for the contribution (if applicable)

[CharlesTaylor7]

@f-f We're getting really close to resolving #1020 .

Unfortunately while demoing this I realized type directed search is not working, the index is only responding to searches for concrete types and values.

I'm planning to get type directed search working in a follow up PR.

[f-f]

This is excellent - let's add that comment about punycode to the Spago config then this is good to merge 👏

[CharlesTaylor7]

@f-f Heads up, the docs-search-app.js halogen bundle will only get properly embedded when you use bin/bundle.js or an official spago release.

But running ./bin/index.dev.js docs will fail. Because the app bundle is assumed to be a sibling of the spago output, and that's not true for the unbundled dev builds.

[CharlesTaylor7]

@f-f The spago offline tests are failing. Do you know what's happening there?

[f-f]

@CharlesTaylor7 could you set this bit to false so we can have a cleaner output in CI? Right now it's all crowded with logs.

[f-f]

But running ./bin/index.dev.js docs will fail. Because the app bundle is assumed to be a sibling of the spago output, and that's not true for the unbundled dev builds.

Can we symlink it in the right place so that the bundle picks up the real one and the index.dev.js picks up the link?

[CharlesTaylor7]

But running ./bin/index.dev.js docs will fail. Because the app bundle is assumed to be a sibling of the spago output, and that's not true for the unbundled dev builds.

Can we symlink it in the right place so that the bundle picks up the real one and the index.dev.js picks up the link?

Ok just realized, the test failure is related to this.
I was able to symlink and get this working with both bin/index.dev.js and bin/bundle.js when used locally.
But the tests are failing to find the bundled halogen app. So I'm troubleshooting that now.

[f-f]

Ah, one of the publishing tests is failing 🤔 think running a "spago install" before the final "spago publish" should fix it

[CharlesTaylor7]

@f-f The issue was twofold. The doc tests were failing. I have fixed those. I suspected some of the tests weren't sufficiently future proofed, so I reran the workflow for the latest commit to master.

The test suite is failing for that commit too:
https://github.com/purescript/spago/actions/runs/6443301225/job/17533396875#step:14:11115

[f-f]

Ah it looks like Spago can't find the client bundle on Windows

[CharlesTaylor7]

@f-f I don't think windows likes the Unix style symlink. I replaced it with a stub file with the right name, but does nothing except log an error message.

I have a better solution in mind for a development workflow, which is better than a symlink and rebundling the app constantly. It'll be in a followup PR though. Basically, we can run a small dev server which enables us to serve unbundled ES modules.