AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed

AIL Framework version 2.8 released with a domain screenshot browser and many bugs fixed.

There is also a "Practical Darkweb and criminal Blockchain monitoring using AIL (Framework for Analysis of Information Leaks) - free Training/Workshop" hosted by CIRCL (Luxembourg) which will take place the 20th Feb 2020. Registration link.

Next release

In the next release of AIL, only Python 3.6 and upper will be supported.

Changes

• [domain core + UI] add domain explorer v1. [Terrtia]
• [test api] increase import timeout. [Terrtia]
• [UI + core] tag decoded items, fix: #455. [Terrtia]
• [UI item tags search] refractor: search item tag by object, use new
  functions. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]
• [UI tags] add + delete image (screenshot) tags. [Terrtia]

Fix

• [MISP export] force pymisp version. [Terrtia]
• [tag core] typo. [Terrtia]
• [tag core] item date type. [Terrtia]
• [Tag core] add tag, update tag last seen. [Terrtia]
• [Flask server + cookie session] chg default cookie name (also use port
  number) + add Flask port number to config. [Terrtia]
• [expand btc adress] filter empty addr fields. [Terrtia]
• [UI navbar] chg icon decoded tags. [Terrtia]
• [UI show item min] fix empty modal. [Terrtia]
• Change name popper.js-1.14.3 to popper-core-1.14.3. [mangelft]
• [UI] screenshot url. [Terrtia]

Other

• Merge pull request #453 from mangelft/master. [Thirion Aurélien]

  fix: change name popper.js-1.14.3 to popper-core-1.14.3

• Merge pull request #450 from CIRCL/tags_v2. [Thirion Aurélien]

  Tags v2

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released with an improved tagging system

AIL Framework version 2.7 released includes a major refactoring of the tagging system. Tagging speed has been significantly improved. Tags can now be used with items, domains and images and added in the objects. A new feature to search by tags has been also added. Multiple bugs were fixed.

Detailed change logs (v2.7 (2020-01-13))

Changes

• [UI domain] add input: show domain by name. [Terrtia]
• [tags blueprint] clean code. [Terrtia]
• [update v2.7] sort domain full_onion_up and full_regular_up. [Terrtia]
• [UI tags] search domains by tags. [Terrtia]
• [core + UI] search domain by tags. [Terrtia]
• [Update v2.7] add update v2.7 scripts. [Terrtia]
• [tags UI] edit object tags (delete tags) [Terrtia]
• [Tag core] objects tagging, Part 2/2 TODO: UI tags domain + screenshot
  • object rename paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2 TODO UI (tags) + rename
  paste=>item. [Terrtia]
• [Tag core] objects tagging, Part 1/2. [Terrtia]
• [README] remove top terms. [Terrtia]
• [correlation graph UI] add json error handler + add loading status.
  [Terrtia]
• [UI correlation graph + UI domain] correlation screenshot: show img in
  toolip + show hash in ShowDomain TODO: pixelate images. [Terrtia]

Fix

• [UI tags] fix domain links. [Terrtia]
• [UI term] remove deprecated trending charts, fix #446 #447. [Terrtia]
• [Crawler] typo. [Terrtia]
• [UI decoded item] sort mimetype. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain typo. [Terrtia]
• [Crawler] fix screenshot-domain map. [Terrtia]
• [UI showDomain] fix screenshot accordeon. [Terrtia]

Other

• Merge pull request #449 from CIRCL/tags_v2. [Alexandre Dulaunoy]

  Tags v2 - Tagging system refractoring

• Merge branch 'master' into tags_v2. [Terrtia]

• Update README.md. [Thirion Aurélien]

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes

AIL Framework version 2.6 released with improved correlations (hover information, screenshot hash correlation), API improvements and various fixes. Thanks to the enforce project for the feedback during the training. New features were based on constructive remarks from the users.

New and Improvements

• [slides] source code added. [Alexandre Dulaunoy]
• [screenshot correlation + v2.6] add screenshot-domain correlation + v2.6 update. [Terrtia]
• [API] get domain min metadata (first up, last up) + get crawled domain by daterange and status. [Terrtia]
• [Domain + Date] get domain up range + get date days and months by daterange. [Terrtia]
• [Domain] get all/by month domains up. [Terrtia]
• [API] get domain metadata (minimal) [Terrtia]
• [UI correlation graph] tooltip: show domain tags. [Terrtia]
• [UI correlation graph] popover: add loading status + chg css. [Terrtia]
• [correlation UI] add basic popover. [Terrtia]
• [slide] update slide. [Terrtia]
• [pgpdump] add debug. [Terrtia]
• Linked TOR installation instruction in the README. [Sami Mokaddem]
• [Onion] add discovery queue. [Terrtia]
• [Showpaste] check if tags are safe (img) + fix domain link. [Terrtia]
• [crawler dashboard UI] add UP/Down domains url. [Terrtia]

Bugs fixed

• [UI correlation graph] typo. [Terrtia]
• [UI showDomain] fix down domain history. [Terrtia]
• [Domain] domain was up. [Terrtia]
• [Domain] is_domain_up. [Terrtia]
• Typo. [Terrtia]
• [Update] force manual update, fix #443. [Terrtia]
• [UI] fix show paste modal. [Terrtia]
• [screenshot canevas + domain link] fix item domain link + screenshot
  canevas: chg colors and icons for unsafe tags. [Terrtia]
• [PgpDump] catch bs4 error. [Terrtia]
• Placed Tor installation instruction in the installation section. [Sami Mokaddem]
• [Update v2.4] fix empty set. [Terrtia]
• [Update v2.4] fix empty set. [Terrtia]
• [Item lib] fix import. [Terrtia]
• [Paste submit] fix tags unpack. [Terrtia]
• [Show Domain UI] fix screenshot link, fix #431. [Terrtia]
• [Update] filter invalid tags. [Terrtia]

AIL Framework version 2.5 released with improved correlation and experimental support for MISP modules

AIL Framework version 2.5 released.

AIL Framework version 2.5 released with correlation and experimental support for MISP modules. The correlation is now improved to add correlation between PGP, cryptocurrencies, pastes, decoded values against any items in AIL framework. The correlation interface has been redesigned to allow filtering per type (pastes, crawled) and also limit the number of correlations. The MISP modules is still very early but the objective is gain from all MISP modules expansion within AIL. We introduced an experimental module to automatically expand BTC transactions from addresses seen in AIL and pivot to new correlations. New roles were added (read_only and users without) to improve the profile of the various AIL users.

The improvements were designed with some requirements from the ENFORCE project to better support law-enforcement usage.

AIL Framework version 2.4 released with improved crawled domain correlation (cryptocurrency, pgp keys, decoded...)

AIL version 2.4 released

AIL version 2.4 has been released including the following new features:

• Improved crawled domain correlation to correlate such domain via cryptocurrency addresses, PGP keys UID, decoded hash content
• Screenshot of crawled item can be selected from the UI
• Crawled domain and port are now properly supported
• Tagging functionality added to domain crawled
• Configuration files have been moved to a coherent directory
• Documentation of the code improved
• Various bugs fixed and small improvements

AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi, reconnaissance tools

AIL Framework version 2.3 released with improved cryptocurrencies detection, SQLi and detection of network reconnaissance tools output. Many bugs fixed and small improvements were performed.

Changes

• [Cryptocurrency + Tools] launch by default + remove old Bitcoin module. [Terrtia]
• [Keys module] detect public key. [Terrtia]
• [Tools detection] add tool detection module. [Terrtia]
• [Cryptocurrency, RegexTracker] update cryptocurrency list + fix: RegexTracker typo. [Terrtia]
• [Cryptocurrency] add private_key entry + fix dash regex. [Terrtia]
• [Cryptocurrency] add new Cryptocurrency module. [Terrtia]
• [Tracker] add optional description field. [Terrtia]

Fix

- [Tool] fix searchsploit regex. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] typo. [Terrtia]
- [Tools] fix loop. [Terrtia]
- [url_prefix] add root blueprint, fix:#403. [Terrtia]
- [TermTracker] fix performance: disable token stats. [Terrtia]
- [SQL module] fix typo. [Terrtia]

# Other

- Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia]
- Merge pull request #417 from andurin/master. [Alexandre Dulaunoy]

  Fix pybgpranking dependency in requirements
- Fix pybgpranking dependency in requirements. [Hendrik]

  Relates #334
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #404 from WimpyMan/master. [Thirion Aurélien]
- LAUNCH.py: Added execution of script IPAddress.py. [Bastien Schils]
- IPAddress.py: use ipaddress module. [Bastien Schils]

  Improved readability, maintainability and use of standard module
- Modules.cfg: Minor: Added \n to seperate sections. [WimpyMan]
- Config.cfg.sample: Improved example for IP module. [WimpyMan]

  By default, the list of networks to monitor is now empty.
  The previous value is now given as example.
- Added: IP matching module. [Bastien Schils]
- Merge pull request #411 from krial057/patch-1. [Alexandre Dulaunoy]

  Fixed some typos
- Fixed some typos. [krial057]

  Fixed some typos in the readme
- Merge pull request #408 from stamparm/master. [Thirion Aurélien]

  Adding more tools
- Adding more tools. [Miroslav Stampar]
- Merge pull request #407 from stamparm/patch-1. [Thirion Aurélien]

  Covering special cases (on pastebin)
- Covering special cases (on pastebin) [Miroslav Stampar]

  There is no need for checking `()` in case of (e.g.) Litecoin and Dash as those are also additionally checked with Bitcoin address verifier
- Merge pull request #406 from stamparm/master. [Thirion Aurélien]

  Adding tool regexes
- Adding tool regexes. [Miroslav Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #405 from stamparm/master. [Thirion Aurélien]

  Enforcing Base58 check on Litecoin and Dash addresses
- Enforcing Base58 check on Litecoin and Dash addresses. [Miroslav
  Stampar]
- Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]
- Merge pull request #401 from stamparm/master. [Thirion Aurélien]

  Improvement of crypto-address regexes (lesser FPs)
- Improvement of crypto-address regexes (lesser FPs) [Miroslav Stampar]
- Merge pull request #398 from stamparm/master. [Thirion Aurélien]

  Implementation for different cryptocurrencies
- Implementation for different cryptocurrencies. [Miroslav Stampar]
- Merge pull request #396 from stamparm/master. [Thirion Aurélien]

  Improvement of SQLi detection
- Improvement of SQLi detection. [Miroslav Stampar]

AIL Framework version 2.2 released with refactoring of term tracking

AIL Framework version 2.2 released with refactoring of term tracking

Changes

• [API] add Tracker documentation. [Terrtia]
• [Tracker] add more info. [Terrtia]
• [update] add v2.2 update + fix default update + fix Empty
  Item.get_tags() + add new LAUNCHER options. [Terrtia]
• [tracker] add missing btn. [Terrtia]
• [trackers] filter trackers list by type + minor fix. [Terrtia]
• [UI sparklines] sparklines: fix + factory. [Terrtia]
• [merge] master. [Terrtia]
• [UI term tracker] refractor term management: trackers list + show
  trackers + add new trackers. [Terrtia]
• [api] add endpoint: get tracked item_id by uuid and daterange.
  [Terrtia]
• [api] add endpoint: delete tracker term (regex/set/word) [Terrtia]
• [Term Tracker] refractor term tracker word/set/regex modules + remove
  old modules. [Terrtia]
• [Term tracker] add term tracker module (word + set) + API: add new
  term to track (word + set + regex) [Terrtia]
• [term] refractor + add new tracked word/set. [Terrtia]
• [README] add link to API documentation. [Thirion Aurélien]
• [tests API] use argv api key. [Terrtia]
• [api] add new endpoints: get bitcoin/pgp name/pgp keys/pgp mail
  metadata + items list. [Terrtia]

Fixes

• [sparkline] datatable drawing. [Terrtia]
• [d3 graph] fix script path. [Terrtia]
• Add missing file. [Terrtia]
• [d3 js plugin] [Terrtia]
• [template] add trackers. [Terrtia]
• [Term Tracker module] chg module flow. [Terrtia]
• [BankAccount] fix #385. [Terrtia]
• [API doc] get bitcoin metadata + list of items. [Terrtia]
• [UI caching] avoid domain archive caching. [Terrtia]
• Typo. [Terrtia]
• [UI caching] fix: #373 avoid screenshot caching. [Terrtia]

(free) Trainings

• Prague, Thursday, 19 Sep 2019 09:00 : https://en.xing-events.com/CLTDKUU.html
• Luxembourg, Monday, 23 Sep 2019 10:00 : https://en.xing-events.com/JDVIRXW.html

AIL Framework version 2.1 released with an API

AIL Framework now includes an advanced API to query the items and information within an AIL instance. You can query items (e.g. such as pastes, crawled website content), meta-data (tag). The API allows to submit item to be processed by the AIL instance. More API endpoints will be added in the next releases.

AIL Framework version 2.0 released including a user management functionality, many bugs fixed and improvements

v2.0 (2019-07-05)

Changes

• [helper] dump crawler history by daterange. [Terrtia]
• [UI submit items] bootstrap 4 migration. [Terrtia]
• [Flask login] add brute force protection + log login errors. [Terrtia]
• [helper] generate self signed certificates. [Terrtia]
• [Flask server] https support + create self signed certificate.
  [Terrtia]
• [user_management 2.0] add update scripts + fix create_default_user.
  [Terrtia]
• [UI user_management] user_role acl: hide admin panel. [Terrtia]
• [UI user_management] incorrect passwords: display errors. [Terrtia]
• [user_management endpoint] check user roles + add 503 template.
  [Terrtia]
• [UI dashboard + search] bootstrap 4 migration: dashbaord + fix search
  input. [Terrtia]
• [UI crawler, show_domain] domain history: remove target blank.
  [Terrtia]
• [UI crawler, show_domain] add domain history list + navigation.
  [Terrtia]
• [user_managemant] clean code + check password and email length.
  [Terrtia]
• [user_management UI] add admin section: edit + create users. [Terrtia]
• [user_management UI] edit my_profile + renew api tokens. [Terrtia]
• [user_management] create default admin user (temp passwd save in
  AIL_HOME) + change password UI + logout UI + create random password.
  [Terrtia]
• [restapi] add rest api authentification + create default user.
  [Terrtia]
• [user_management] add user role_management. [Terrtia]
• [user_management] create + check user password. [Terrtia]
• [UI] add basic user management. [Terrtia]

Fix

• [items submit UI] fix tags dropdown. [Terrtia]
• [helper dump_crawler] fix files not found. [Terrtia]
• [helper dump_crawler] fix empty dict. [Terrtia]
• [MISP export] fix event creation. [Terrtia]
• [UI items_submit] add active tag + fix template name. [Terrtia]
• [UI login/change_password] add missing ail-logo. [Terrtia]
• [Update] clean output. [Terrtia]
• [Update] add default update script. [Terrtia]
• [install_dep] create update current_version. [Terrtia]
• [UI settings] fix toggle_sidebar. [Terrtia]
• [install_dep] create default user. [Terrtia]
• [user_management] fix tokens duplicate + check user_acl_integrity +
  add login errors messages. [Terrtia]
• [server endpoint] unknow users: avoid endpoint enumeration. [Terrtia]
• [TheHive feeder] create_alert: push all items tags TODO check items
  status + add more item metadata. [Terrtia]
• [domain history] fix domain status. [Terrtia]

Other

• Merge pull request #359 from CIRCL/user_management. [Alexandre
  Dulaunoy]

  User/role management

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework.
  [Terrtia]

• Update README.md. [Thirion Aurélien]

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'user_management' of https://github.com/CIRCL/AIL-
  framework into user_management. [Terrtia]

• Update README.md. [Thirion Aurélien]

  Update install instructions

• Merge branch 'master' into user_management. [Terrtia]

• Merge branch 'master' into user_management. [Terrtia]

AIL Framework version 1.8 released including many bug fixes, improved PGP handling, ability to export crawled resource.

v1.8 (2019-06-12)

Changes

• [UI crawled domains] Download all domain content (HTML + HAR +
  screenshot) [Terrtia]
• [backend crawler] domains: download 1 archive by crawled (most recent)
  [Terrtia]
• [paste_submit UI] add debug. [Terrtia]
• [template] the "item" project. [Alexandre Dulaunoy]

Fix

• [PgpDump] catch UnicodeDecodeError error. [Terrtia]
• [backend crawler] rename downloaded archive. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [PgpDump] process large pgp blocks. [Terrtia]
• [paste_submit UI] filter empty file field. [Terrtia]
• [UI crawler endpoints] display crawler status + fix #353. [Terrtia]
• [update v1.7] add bs4 requirement. [Terrtia]

Example

Finding relationships between two hidden services via the PGP key published on the website.