Skip to content

Commit

Permalink
chg: Use valkey by default
Browse files Browse the repository at this point in the history
  • Loading branch information
Rafiot committed Aug 8, 2024
1 parent 5af64f1 commit 4805bf5
Show file tree
Hide file tree
Showing 5 changed files with 31 additions and 24 deletions.
10 changes: 5 additions & 5 deletions modules/ROOT/pages/dev-docs.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,19 @@ and gave it a try.
== General guidelines

* After editing a file, you can run the `stop` and `start` scripts to restart the platform and see the changes.
It will restart everything, including the redis databases, and can take a bot of time. See below if you're
It will restart everything, including the valkey databases, and can take a bot of time. See below if you're
working on a specific part of the tool and want to stop/start a specific module.
* You can disable individual services by commenting them out in `bin/start.py`
* All the services can be started/killed manually (`CTRL+c` to kill)

== Scripts

* `start`: Starts all the services
* `stop`: Triggers `shutdown`, stop redis databases.
* `shutdown`: Requests the services to stop by putting a `shutdown` key in redis, wait for all of them to stop.
* `run_backend`: Starts/stops redis databases
* `stop`: Triggers `shutdown`, stop valkey databases.
* `shutdown`: Requests the services to stop by putting a `shutdown` key in valkey, wait for all of them to stop.
* `run_backend`: Starts/stops valkey databases
* `async_capture`: Triggers a capture from the queue. Multiple instances of this process can be run in parallel
* `background_indexer`: Caches the captures in redis, creates the tree pickles, adds the urls, cookies,
* `background_indexer`: Caches the captures in valkey, creates the tree pickles, adds the urls, cookies,
and body hashes in indexes for fast lookup
* `archiver`: Archives the old captures (default: 180+ days), maintain the indexes files in the capture directories
* `processing`: Triggers actions happening once a day. Currently: creates the user-agent file of the users of the platform (if enabled)
Expand Down
4 changes: 2 additions & 2 deletions modules/ROOT/pages/implementation-details.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ developed by others.

== Backend

=== Redis
=== Valkey

link:https://redis.io/[Redis] is the database used to store the cached information visualized on the web interface.
link:https://valkey.io/[Valkey] is the database used to store the cached information visualized on the web interface.

=== Playwright

Expand Down
35 changes: 21 additions & 14 deletions modules/ROOT/pages/install-lookyloo.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,42 +3,40 @@

[IMPORTANT]
====
Requires Python 3.8 or higher (3.10 prefered), expects Ubuntu 22.04 or more recent.
Requires Python 3.8 or higher (3.12 prefered), expects Ubuntu 22.04 or more recent (24.04 prefered).
====

== Install Lookyloo Dependencies

=== Install Poetry

link:https://python-poetry.org/[Poetry] is tool to handle dependency installation
link:https://python-poetry.org/[Poetry] handles dependency installation
as well as building and packaging of Python packages.

```bash
curl -sSL https://install.python-poetry.org | python3 -
```
See the link:https://python-poetry.org/docs/#installation[installation guide] for more details.

More details in the the link:https://github.com/python-poetry/poetry/#installation[installation guide].
=== Install Valkey

=== Install Redis
NOTE: Lacus supports valkey or redis, but valkey is prefered now due to the change of license.

link:https://redis.io/[Redis]: An open source (BSD licensed), in-memory data structure
link:https://valkey.io/[Valkey]: An open source (BSD licensed), in-memory data structure
store, used as a database, cache and message broker.

NOTE: Redis should be installed from the source, and the repository must be in
NOTE: Valkey should be installed from the source, and the repository must be in
the same directory as the one you will be cloning Lookyloo in to.

NOTE: Since Lookyloo 1.15, redis 7.0 is required.
NOTE: Since Lookyloo 1.25, valkey 7.2 is required.

In order to compile and test redis, you will need a few packages:
In order to compile and test valkey, you will need a few packages:

```bash
sudo apt-get update
sudo apt install build-essential tcl
```

```bash
git clone https://github.com/redis/redis.git
cd redis
git clone https://github.com/valkey-io/valkey
cd valkey
git checkout 7.2
make
# Optionally, you can run the tests:
Expand All @@ -51,9 +49,10 @@ cd ..
=== Prerequisites

* Poetry is installed.
* Redis is installed from the source. The repository must be in the same directory
* Valkey is installed from the source. The repository must be in the same directory
as the one you will be cloning Lookyloo in to.


* Python3 development package

```bash
Expand All @@ -68,6 +67,14 @@ sudo apt install python3-dev
git clone https://github.com/Lookyloo/lookyloo.git
```

The directory tree must look like that:

```
.
├── valkey => cloned valkey
└── lookyloo => cloned lookyloo
```

. Change directory
+
```
Expand Down
2 changes: 1 addition & 1 deletion modules/ROOT/pages/lookyloo-auth.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ To authenticate your session, open `http://your.lookyloo.instance/login`.
When logged in, you have access to the following URLs:

* `/hidden`: show the tree hidden on the index
* `/rebuild_cache`: rebuild only the redis cache. Can take a while, but not as much as `/rebuild_all`
* `/rebuild_cache`: rebuild only the valkey cache. Can take a while, but not as much as `/rebuild_all`
* `/rebuild_all`: rebuild all the tree (drop all the pickles of all the trees). **warning**: it can take a very, very long time.
* `/logout`: drop the session
* `/tree/<string:tree_uuid>/rebuild`: rebuild the tree
Expand Down
4 changes: 2 additions & 2 deletions modules/ROOT/pages/update-lookyloo.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ The sections below are only required if the tools have been updated, or if upgra
poetry self update
```

=== Update Redis
=== Update Valkey

It requires to shutdown lookyloo completely.

Expand All @@ -50,7 +50,7 @@ From the lookyloo repository:
poetry run stop
```

From the directory you cloned redis into:
From the directory you cloned valkey into:

```bash
# Optional: git checkout <version>
Expand Down

0 comments on commit 4805bf5

Please sign in to comment.