TLDR:

• Design is the same except when accessibility mattered most
• HTML was heavily tweaked in some spaces for accessibility and semantic reasons, but the design staus the same
• CSS was remade from scratch but old files are in the "css_old" folder
• Loading performances were increased by using woff2 instead of ttf

Notable changes:

• A reset was added to limit the number of basic CSS fixes or addons to add. It's placed inside the
• For accessibility reasons, an input MUST have a label. A label was added to the search input. Not very esthetic I know, but well...
• For accessibility reasons, each page MUST start with at least an h1. It was added in some pages. In other places, h2 were converted to simple text.
• The CSS is loosely following the CUBE CSS methodology, could maybe use some cleaning.
• Entries are now lists for accessibility reasons. Screen readers announce the number of elements inside a list when they enter them, and shortcuts help jumping from one to the next

The commits can globally tell my though process when I coded this. If you need tweaks or corrections please ask.

This PR adds a configuration option which allows a user to configure an http:// or socks:// proxy in the lieu.toml, and an additional option for extracting links from web sites that don't have a them in the form of <li><a></a></li>. My intent with this is to build a plugin to the I2P network which enhances the user's ability to search for sites by sharing the task of crawling sites and making it easy to run a small search engine. Simply setting the http_proxy environment variable does not appear to be sufficient due to DNS leakage, it must be done by replacing the default transport.

Hello, light theme is needed. I tried to fork and modify the project but had some problems (on my side, probably) related to sqlite. I don't want to deal with them currently, so I'm leaving the the task to you :-)

You probably need to add this to base.css:

@media screen and (prefers-color-scheme: light) {
    :root {
        --primary: #000;
        --secondary: #fefefe;
    }
}

It will basically swap colors everywhere, except for the search button. You have hard-coded colors in the logo.svg, there are two ways to swap the colors:

1. Provide two versions of the file and serve them depending on current theme.
2. Change svg color with svg. Look here or somewhere else maybe.

Thanks

Added some comments that explain a bit better what is happening and added checks for the other fields to prevent the foreground without background issue.

• This added a favicon that was slightly modified to make it easier to recognize at small scales as svg
• Fixed the problem that some browsers don't like non-square favicons (by making the favicon a square)
• regenerated an optimized version in the ico and png formats

This version of the favicon is already used by my test instance of lieu over at: https://fediring-lieu.slatecave.net/

Adds support some selectors needed for more advanced searches like

• -site: to exclude sites
• lang: to only search for pages that have a given language tag prefix
• rank:count and rank:score to choose a ranking algorithm (also exposed as url parameter)

Note: #13 is the same PR without the rank selector

Adds support some selectors needed for more advanced searches like

• -site: to exclude sites
• lang: to only search for pages that have a given language tag prefix

Note: #14 is a version of this PR that includes a rank: tag that allows one to choose the ranking algorithm

• Made the paragraph scraping a lot more flexible
• Scraper now collects opengraph descriptions
• Ingest now considers opengraph descriptions as preview text
• Fixed: Fixed the ingest not being able to handle lots of words per batch
• Fixed: URIs are case sensitive, don't lowercase them

When running Lieu over all the sites in the fediring, we've found that it's only bound by domain rather than domain+path. This causes quirks with static site hosts like cronut.cafe; the only cronut.cafe user who's also a member of the ring is ~sfr, but multiple other users who aren't members have been indexed as well: https://search.fediring.net/?q=cronut

I think a good solution might be keeping track of not only the domain that's being crawled but also the original URL and ignoring links to parent directories.

This allows Lieu to be discovered as a search engine by browsers, which then can be set as the default search engine for example.

Looks like this in Firefox:

Webring sites passed to Lieu which end with a path, e.g. https://example.com/site/lupin will now only have their children pages crawled (as opposed to allowing all pages of example.com being crawled).

This falls more in line with expectations for webring sites which might exist on shared hosting, or just sites which have separate areas that should not be crawled.

Thanks @amolith for the issue!

Includes the following patches:

• .View variable in templates (to make some deduplication possible)
• Deduplicated the navigation and search-form
• The tagline is now a paragraph instead of an h2

NOT included are:

• a 404 page, the patch inserts code in places that were changed by #13 and #14, I'll submit it once one of those has either been merged or rejected
• the patch introducing error messages for the case when no results are available, it also depends on one of either #13 or #14

Currently Lieu treats every page that is not a special route like its home page, which is not ideal since all those other pages should throw a proper 404 code and have their own template (which doesn't stop them from having a search form though …)

Lieu crawls fast; it seems to crawl at a rate of 5req/sec by default.

Normally, this wouldn't bee too worrying; however, the types of sites Lieu crawls are small hobbyist sites. Some peoples' sites may be self-hosted on limited hardware.

Respecting a crawl-delay robots.txt directive could help avoid overwhelming smaller sites.

Hi there administrators! My blog in the webring uses a link format of index.php?post=20220612001238, like this, however it looks to me that the crawler doesn't like query arguments in the url: https://github.com/cblgh/lieu/blob/b0ad7dce102d35123bb0092527b7ceea6df8ad86/crawler/crawler.go#L51

Can there be a way for sites to hint that they may want to use ? or # separated URLs? From what I know, MDWiki is quite popular and it uses #! to specify page links, so that way we could index more pages for these sites as well.

I can see where # could post some problems with title links... I'd suggest that allow a <meta> or some sort of tag in the page head to hint the crawler that some formats of the link can be allowed, and if href regex matches the "allowed link format", the link will be preserved?

Thanks!

:wave: r a d project :partying_face:

As an admin, it feels a bit uncomfortable putting domains into that config with such a name. The functionality is great & relevant but in a pubnix / shared server environment, other users might get the wrong idea seeing the naming? If I'm trying to instead focus my search space on relevant links and not saying I think their stuff is "boring".

Proposal: skipDomains.

I noticed that a search for pizza just shows stuff that is on a site with a .pizza domain. I don't know if this is preventing actual pizza content from turning up or if there isn't any, but either way, it could be more useful if it was aware of it just being part of the domain.