Shovel is a web application that offers a graphical user interface to explore Suricata EVE outputs. Its primary focus is to help Capture-the-Flag players analyse network traffic dumps during stressful and time-limited attack-defense games such as FAUSTCTF or ECSC. Shovel is developed in the context of ECSC Team France training.
You might also want to have a look at these other awesome traffic analyser tools:
- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
Compared to these traffic analyser tools, Shovel relies on Suricata while making some opinionated choices for the frontend. This has a few nice implications:
- dissection of all application protocols already supported by Suricata (TCP and UDP),
- use a single SQLite database,
- on disk TCP/UDP/HTTP payload deduplication,
- filters based on libmagic, e.g. quickly filter flows containing PDF documents or PNG images,
- no heavy build tools needed, Shovel is easy to tweak.
Moreover, Shovel is batteries-included with Grafana visualizations and some Suricata alert rules.
Copy example.env to .env and tweak the configuration parameters.
Also add the flag format in suricata/rules/suricata.rules if needed.
If you are playing a CTF using an IPv6 network, you might want to enable IPv6 support in Docker deamon before the CTF starts.
You should place network captures in input_pcaps/ folder.
Capture files should be splitted into chunks to be progressively imported.
If the CTF event does not already provide PCAP files, then you can adapt one
of the following commands for a GNU/Linux system:
ssh [email protected] tcpdump -i game -n -w - 'tcp port not 22' | tcpdump -n -r - -G 30 -w input_pcaps/trace-%Y-%m-%d_%H-%M-%S.pcapFor a Microsoft Windows system, you may adapt the following command (3389 is RDP):
.\tshark.exe -b duration:60 -w \\share\captures\trace -f "tcp port not 3389"Start Suricata, the web application and Grafana using docker compose up -d --build.
By default, all services are only accessible from localhost.
You should edit docker-compose.yml if you want to expose some services to your local network.
Please note that restarting Suricata will cause all network capture files to be loaded again from zero.
You may launch Suricata then the web application using the following:
# Start Suricata
export $(grep -vE "^(#.*|\s*)$" .env)
./suricata/entrypoint.sh# Start web app
export $(grep -vE "^(#.*|\s*)$" .env)
(cd webapp && uvicorn --host 127.0.0.1 main:app)Please note that restarting Suricata will cause all network capture files to be loaded again from zero.
flow_id is derived from timestamp (ms scale) and current flow parameters (such
as source and destination ports and addresses). See source code:
https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680.
You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules
using:
kill -USR2 $(pidof suricata)A SQLite database is generated in webapp/database/database.db on the first run
of the uvicorn webapp.
If you want to host a read-only Shovel instance (e.g. after the end of a CTF
event for further analysis), you may run the webapp in immutable mode using the
following environment variable:
DATABASE_URL=file:database/database.db?immutable=1
Copyright (C) 2023 ANSSI
Shovel is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.
Shovel is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with Shovel. If not, see https://www.gnu.org/licenses/.