AIL Framework version 4.0 released with new synchronisation feature, tracker webhook improvement and many bugs fixed.

AIL Framework version 4.0 has been released including a major new feature to allow synchronisation to other AIL instance(s).

The new synchronisation mechanism allow the sync from one AIL instance to another AIL using a standard WebSocket using AIL JSON protocol. The synchronisation allows to filter and sync specific collected items including crawled items or specific tagged items matching defined rules. This feature can be very useful to limit the scope of analysis in specific fields or resource intensive activity. This sync can be also used to share filtered streams with other partners.

A new functionality has been added to trigger a webhook when a tracker is matched in AIL. This is in addition to email notification. The webhook can be used to trigger additional pipelines in AIL.

Additional API endpoints were added such get_item_sources get_check_item_source and get_default_yara_rule_content.

Thanks to the numerous external contributors such as Olivier Sagit and Tony Jabbour. A special thank for the webhook and API developed by Tony Jabbour from CSIRT POST Cyberforce in Luxembourg.

The first version of the synchronisation protocol has been developed in the scope of the JTAN (Joint Threat Analysis Network), a CEF co-funded project (2020-EU-IA-0260).

Many bugs were fixed in this release and many small improvements were added.

Detailed changelog is available on https://www.ail-project.org/ChangeLog.

Changelog

v4.0 (2021-12-01)

Changes

• [sync UI] disable pull. [Terrtia]

• [sync UI] dashboard, show nb of imported items + launch/kill ail servers when a queue is subscribed/unsubscribed. [Terrtia]

• [ail sync UI] restarr/launch/kill sync connections + show sync mode api/pull/push. [Terrtia]

• [ail sync server] add server controller + list connected clients ail_uuid->sync_modes. [Terrtia]

• [ail sync ui] copy to clipboard ail_uuid, ail server key. [Terrtia]

• [ail sync] edit ail_servers/sync_queues + fix logs. [Terrtia]

• [api] rename endpoints. [Terrtia]

• [ail sync] add sync api (ping, version) + UI/client error handler. [Terrtia]

• [doc] GI Badge. [Steve Clement]

• [v4.0 AIL SYNC / AIL 2 AIL] SYNC Manager + fixs + views. [Terrtia]

• [crawler] add auto crawler functions. [Terrtia]

Fix

• [sync client] don't launch client if ail server not linked with a sync queue. [Terrtia]

• [sync server] remove hardcoded host. [Terrtia]

• [sync server] host. [Terrtia]

• [sync client] fix websockets client connect for python >= 3.8. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] fix refresh_ail_instance_connection. [Terrtia]

• [ail sync] server + client: resend object in queue on ConnectionClosedError. [Terrtia]

• [crawler] add comment. [Terrtia]

• [UI ail sync] fix nav. [Terrtia]

• [UI ail sync] add missing ail icon. [Terrtia]

• [doc] Remove Travis. [Steve Clement]

• [py] Minor python dependency change. [Steve Clement]

• Inherit AbstractModule to prevent stuck queues. [osagit]

  regex compiled only at start, not in the loop
  no duplicate warning string
  comments

• Error message contains http protocol twice. [osagit]

  Error Can't connect to AIL Splash Manager, http://https://localhost:7001/

Other

• Merge pull request #130 from TonyJabbour/master. [Thirion Aurélien]

  New restAPIs

• Merge branch 'dev' into master. [Thirion Aurélien]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Alexandre Dulaunoy]

• Merge pull request #569 from SteveClement/master. [Steve Clement]

• Chg_ [AIL 2 AIL] add backend. [Terrtia]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Add tracker fixed api function replaced it with internal function. [TonyJabbour]

• Added get_tracker_metadata_api Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Fixed get_item_content_encoded_text Added get_item_sources Added get_check_item_source Added get_default_yara_rule_content. [TonyJabbour]

• Removed unnecessarily parentheses. [TonyJabbour]

• New API Endpoint: Return Item Content in base64 in non JSON format. [TonyJabbour]

• Merge remote-tracking branch 'origin/master' [TonyJabbour]

• Merge branch 'ail-project:master' into master. [Tony]

• Merge pull request #129 from TonyJabbour/master. [Thirion Aurélien]

  Webhook implementation

• Unnecessarily parenthesis removed. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Base64 Problem. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Error Fixed. [TonyJabbour]

• Add new API endpoint that return only content encoded in base64. [TonyJabbour]

• Webhook unnecessarily line removed Removed unnecessarily parentheses. [TonyJabbour]

• Type fixed. [TonyJabbour]

• Added a try/catch to handle exceptions Replaced the raise to send message to redis_logger. [TonyJabbour]

• Fixed tracker_metadata. [TonyJabbour]

• Typo Fixed. [TonyJabbour]

• Fixed UI Datatable in showTracker Page Removed "Advanced Search" text from menu misleading button. [TonyJabbour]

• Fixed Webhook integration with Trackers. [TonyJabbour]

• -Fixed "get_tracker_metedata" typo -Typo Fixed. [TonyJabbour]

• -Fixed "description" arg -Typo Fixed. [TonyJabbour]

• -Fixed the 500 error issue when installing new instance of ail when adding new trackers -Fixed missing arguments -Typo Fixed. [TonyJabbour]

• Fix webhook. [TonyJabbour]

• Remove dict from Trackers. [TonyJabbour]

• Add webhook post support in yara and regex trackers. [TonyJabbour]

• Fix get_term_webhook. [TonyJabbour]

• Add some changes for webhook. [TonyJabbour]

• Add initial support for Webhook in Term Trackers. [TonyJabbour]

• Fix spelling issue in Webhook. [TonyJabbour]

• Add more support for Webhook URL. [TonyJabbour]

• Add initial UI support for Webhook in tracker. [TonyJabbour]

• Merge pull request #127 from osagit/patch-3. [Thirion Aurélien]

  fix: inherit AbstractModule to prevent stuck queues

• Merge pull request #126 from osagit/patch-2. [Thirion Aurélien]

  fix: error message contains http protocol twice

AIL Framework version 3.7 released with many bugs fixed, improvement and new feeders.

AIL Framework version 3.7 released with many bugs fixed and improvement. The term tracker has been improved including the first_seen and last_seen. Various bugs were fixed and documentation improved. Thanks to all the contributors and especially Tony Jabbour for the new CentOS installation documentation. Thanks to Relega for the improved documentation about the pystemon integration. And a huge thank to @FBroy for the new feeders: Discord, ActivityPub feeder and RSS/Atom feeder. There is an ongoing feeder to include AIL2AIL synchronisation and a first draft of the message format has been proposed. Feedback is more than welcome.

v3.7 (2021-08-27)

Changes

• [tracker + update] add update v3.7 + add map item_id:tracker_uuid (data retention) + fix tracker first_seen/last_seen. [Terrtia]

• [tracker] typo fixed. [Alexandre Dulaunoy]

• [Credential + tags] add misp-taxonomies submodule + fix typo. [Terrtia]

• [gitchangelog.rc] updated to output Markdown. [Alexandre Dulaunoy]

Fix

• [tracker] global tracker list: fix first/last seen. [Terrtia]

• [v3.7] add missing file. [Terrtia]

• [trackers] items by daterange. [Terrtia]

• [correlation graph] fix legend, remove icon text. [Terrtia]

• [Credential] fix moduleStats. [Terrtia]

• [Credemtial module] fix stats. [Terrtia]

• [Yara + regex trackers] remove tests. [Terrtia]

• [Decoder] log level. [Terrtia]

• [abstract_modules + Global] log message on error + fix Global exception on invalid gzip. [Terrtia]

• [Credential] fix old funct call. [Terrtia]

• [UI Item submit] tags input: avoid browser and password managers autocomplete. [Terrtia]

• [term tracker] typo. [Terrtia]

Other

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge pull request #123 from TonyJabbour/master. [Alexandre Dulaunoy]

  CentOS 8 installation script Fixed a problem

• CentOS 8 installation script Fixed a problem Added centos installation guide in README.md. [Tony Jabbour]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge pull request #122 from TonyJabbour/master. [Alexandre Dulaunoy]

  Added CentOS 8 installation script

• Added CentOS 8 installation script. [Tony Jabbour]

• Merge branch 'master' of github.com:ail-project/ail-framework. [Terrtia]

• Merge pull request #119 from Relega/patch-1. [Alexandre Dulaunoy]

  Update HOWTO.md

• Update HOWTO.md. [Relega]

  • replaced pystemon url (from circl repository to the original repository)
  • refined pystemon instructions

AIL Framework version 3.6 released with new features (such as YARA retrohunt) and many bugs fixed

AIL Framework released version 3.6

AIL Framework version 3.6 released with new features (such as YARA retrohunt), significant performance improvements, refactoring of the modules and many bugs were fixed. This version includes a new advanced to perform YARA retrohunt on the whole AIL dataset. YARA retrohunts can be started and stopped live, sources can be defined and also the period where to search retroactively for. Tags can be also applied to reclassify information collected in AIL. The speed of YARA retrohunt depends of the hardware used, SSD devices are strongly recommended. New tool such as export of crypto currencies correlation has been added to allow the use of data from AIL in other tools for analysing cryptocurrency transactions.

Many bugs were also fixed in this release. Don't forget to update.

A huge thanks to all the contributors and especially @osagit for the numerous contributions.

Changelog

v3.6 (2021-07-14)

Changes

• Chg: [README] add Olivier Sagit @osagit to contributors list. [Terrtia]

• [requirements] minimal version + remove old packages. [Terrtia]

• Add Retro Hunt. [Terrtia]

• [UI trackers] add/edit/remove tracker source/target #43 #102. [Terrtia]

• [Trackers regex + yara] filter by item source. [Terrtia]

• [Tracker term] track terms by sources. [Terrtia]

• [test] add new workflow. [Thirion Aurélien]

• [requirements] minimal version + remove old packages. [Terrtia]

• [modules + tests] fix modules + test modules on samples. [Terrtia]

• [update v3.6 + installer] fix faup install + add v3.6 update. [Terrtia]

• [Launcher] update modules directory. [Terrtia]

• [SQLInjectionDetection LibInjection modules] add module class. [Terrtia]

• [Urls (Web) module] fix regex + rename. [Terrtia]

• [Tracker_Regex module] create module class + mv module. [Terrtia]

• [Tracker_Yara module] create module class. [Terrtia]

• [modules] create new modules repository + small fixs. [Terrtia]

• [Keys] add test. [Terrtia]

• [modules] add tests: CreditCard, Global, DomClassifier. [Terrtia]

• [Categ] tests + docs. [Terrtia]

• [ApiKey] refactor module + tests. [Terrtia]

• [launcher + modules] add module tests (Onion module) [Terrtia]

• [AIL items + Onion] create AIL item objects + Onion module refactor. [Terrtia]

• [tools extract_cryptocurrency] filter by correlation objects + errors messages. [Terrtia]

• [tools extract_cryptocurrency] new input: list of addresses to extract. [Terrtia]

• [tools] add 2 new tools: extract cryptocurrency correlation by type + trigger manual update. [Terrtia]

• [modules] set log level to critcal on unexpected errors. [Terrtia]

Fix

• [Term tracker] fix fct args. [Terrtia]

• [crawler] fix new crawled item id. [Terrtia]

• [Retro Humt UI] clarify tags input. [Terrtia]

• URLs www word to match. [osagit]

  There is 4 'w' chars instead of 3

• [crawlers] get_all_splash return type. [Terrtia]

• [Splash_Manager errors] catch invalid response. [Terrtia]

• [Trackers UI] fix sources logo + tracker metadata. [Terrtia]

• Change module name Web to Urls. [lpwm9803]

• [Splash_Manager errors] catch invalid response. [Terrtia]

• [github workflows] fix test. [Terrtia]

• [github workflows] clone depth. [Thirion Aurélien]

• [installer] pyfaup install. [Terrtia]

• [installer] pyfaup install. [Terrtia]

• [UI crawler dashboard] block_languages_search: fix domains_types #110. [Terrtia]

• [UI crawler dashboard] block_domains_name_search: fix domains_types #110. [Terrtia]

• [modules] print + save traceback in logs. [Terrtia]

• [paste_submit] restrict source characters. [Terrtia]

• [submit_paste] restrict source name. [Terrtia]

• [tools extract_cryptocurrency] argparse flags. [Terrtia]

• [tags] invalid tags. [Terrtia]

• Replace redis init with generic ConfigLoader. [osagit]

  StrictRedis() replaced by ConfigLoader.get_redis_conn()

• Debug() string takes 1 positional argument. [osagit]

• FILE_ALLOWED_EXTENSIONS without quotes. [osagit]

• Stuck queues and submit paste. [Olivier SAGIT]

• Name pystemon feeder in feeder monitor dashboard. [Olivier SAGIT]

• [tools extract cryptocurrency] correlation type. [Terrtia]

• [Updater] don't check if modified config files (redis and ardb config) [Terrtia]

• [Web module] resolver, change log level. [Terrtia]

• [WebStats] typo. [Terrtia]

• [Indexer] debug messages. [Terrtia]

• [WebStats] typo. [Terrtia]

• [terms tracker] refresh Tracked terms. [Terrtia]

• [redis cache] remove old paste_name db. [Terrtia]

• [crawler] typo: splash restart. [Terrtia]

Other

• Merge branch 'master' of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #115 from My-WAF/master. [Thirion Aurélien]

  Remove Block Copy Git Directory

• Update .dockerignore. [VNC Company]

• Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #116 from osagit/patch-1. [Thirion Aurélien]

  fix: URLs www word to match

• Merge. [Terrtia]

• Merge branch 'dev' of https://github.com/ail-project/ail-framework into dev. [Terrtia]

• Merge pull request #114 from osagit/dev. [Thirion Aurélien]

  fix: Change module name Web to Urls

• Merge branch 'master' into dev. [Terrtia]

• Merge branch 'master' of https://github.com/CIRCL/AIL-framework. [Terrtia]

• Merge pull request #559 from lesleyxyz/patch-1. [Thirion Aurélien]

  Submit paste contents to TheHive

• Submit file contents to TheHive. [Lesley De Keyser]

• Merge pull request #103 from osagit/scriptsenhance. [Thirion Aurélien]

  fix: stuck queues and submit paste

• Merge branch 'dev' into scriptsenhance. [Thirion Aurélien]

• Merge branch 'master' of https://github.com/ail-project/ail-framework. [Terrtia]

• Perf: use defined compiled regex. [osagit]

  re.compile(regex) definition was not used
  use compile_regex.findAll() directely instead of re.findall(regex)

• Log message split error + perf. [osagit]

  fix: log message split errors
  perf: string affected at start
  doc: comments

• Merge pull request #101 from osagit/namedfeeder. [Alexandre Dulaunoy]

  fix: name pystemon feeder name in feeder monitor dashboard

• Merge pull request #97 from osagit/abstract-module. [Thirion Aurélien]

  feat: module factorization

• Feat: module factorization. [lpwm9803]

AIL Framework version 3.5 released with a new flexible crawler manager, built-in Passive DNS sensor and many fixes

AIL Framework version 3.5 released with a new flexible crawler manager, built-in Passive DNS sensor and many fixes.

Changes

• [passivedns] D4server port. [Terrtia]
• [v3.5] install pyAIL. [Terrtia]
• [passivedns] D4server port. [Terrtia]
• [passiveDns] add dns records. [Terrtia]
• [passiveDns D4 Client] add passiveDns D4 Client. [Terrtia]
• [crawler] add test + relaunch crawlers + major fixs. [Terrtia]
• [update] add v3.5 update. [Terrtia]
• [doc] Splash Manager Configuration. [Terrtia]
• [UI] add update note. [Terrtia]
• [IPAddress] Add Debug + Check if option is empty. [Terrtia]
• [splash manager] update enpoints + use Splash name to restart docker.
  [Terrtia]
• [merge master] [Terrtia]
• [domains search] search domains by name. [Terrtia]
• [screenshot + har directory] add option to change screenshots
  directory. [Terrtia]
• [crawler_manager] UI edit config + fix crawler queues. [Terrtia]
• [config] save config on DB + default values. [Terrtia]
• [crawlers manager] show setings. [Terrtia]
• [crawler] show all crawlers type on dashboard. [Terrtia]
• [crawler] manage crawlers. [Terrtia]
• [crawler] crawler queue + restart docker on error. [Terrtia]
• [core module] disable phone module by default. [Terrtia]
• Merge master -> crawler_manager. [Terrtia]
• [crawler manager] get all splash dockers, proxies and launch all
  crawlers. [Terrtia]

Fixes

• [crawler] crawler test: remove print. [Terrtia]
• [crawler UI] remove unused crawlers configuration. [Terrtia]
• [crawler] user agent + splash restart. [Terrtia]
• [crawler UI] crawler by domain type, remove old dashboard. [Terrtia]
• [MISP Export UI] object type selector. [Terrtia]
• [UI] fix navbar: invalid url. [Terrtia]
• [crawler] typo. [Terrtia]
• [crawler] typo. [Terrtia]
• [crawler] typo. [Terrtia]
• [Crawler] faup. [Terrtia]
• [Launcher] launch core module: Crawler_manager. [Terrtia]
• [cralers] remove debug. [Terrtia]
• [Domain search] fix regex. [Terrtia]
• [API ACL] avoid user_no_api users to access the API. [Terrtia]
• [Module Queue] module without subscriber. [Terrtia]
• [term Tracker] TimeoutException. [Terrtia]
• [API ACL] read_only user role. [Terrtia]
• [OVERVIEW] [Terrtia]
• [gitignore] [Terrtia]
• [showDomain] empty father field. [Terrtia]
• [showDomain] empty father field. [Terrtia]
• [redis cache] remove old Redis_Data_Merging db. [Terrtia]
• [redis cache] remove old paste_name db. [Terrtia]
• [background updater] Don't launch updates on fresh install. [Terrtia]
• [v3.4 update] fix update progress. [Terrtia]
• [background updater] remove completed updates. [Terrtia]
• [install] force virtual environmemt activation. [Terrtia]

Other

• Merge branch 'master' into crawler_manager. [Terrtia]

• Merge pull request #89 from ail-project/crawler_manager. [Alexandre
  Dulaunoy]

  Crawler manager

• Merge branch 'master' into crawler_manager. [Terrtia]

• Merge branch 'master' into crawler_manager. [Terrtia]

• Merge pull request #536 from simonsigre/patch-1. [Alexandre Dulaunoy]

  Typo in placeholder "Optionnal" --> "Optional"

• Typo in placeholder "Optionnal" --> "Optional" [Simon Sigré]

  Updated placeholder value to correct typo;
  "Optionnal" --> "Optional"

• Merge branch 'master' of github.com:CIRCL/AIL-framework. [Alexandre
  Dulaunoy]

• Merge pull request #534 from simonsigre/patch-1. [Alexandre Dulaunoy]

  Added 'wget' as a dependency

• Added 'wget' as a dependency. [Simon Sigré]

  'wget' is a dependency as without this additional components won't download as part of initial install, one such example is;

  wget -O /ail-framework/ardb/src/../deps/jemalloc-5.1.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/5.1.0/jemalloc-5.1.0.tar.bz2 && \
  

• Merge branch 'master' into crawler_manager. [Terrtia]

• Merge branch 'master' into crawler_manager. [Terrtia]

• Merge branch 'master' into crawler_manager. [Terrtia]

AIL Framework version 3.4 released with a new language detection module and many improvements

AIL Framework version 3.4 released with a new language detection module and many small improvements

AIL Framework version 3.4 has been released with a new language detection which can classifies items analysed by
the language detected. The module relies on Compact Language Detector v3 (CLD3).
Older items will be updated (there is a background job doing the update, so it might take some time depending of your dataset).
You can then browse crawled items per language detected.

The trackers can now be edited and various bugs were fixed.

Thanks to all the contributors.

Changes

• [Languages]detect + search domains languages. [Terrtia]
• [Tracker] edit tracker. [Terrtia]
• [web] we process items nowadays not only pastes ;-) [Alexandre
  Dulaunoy]

Fix

• [install] pycld3 dependency. [Jean-Louis Huynen]

  Without this Flask fails starting

• [update v3.4] updater. [Terrtia]

• [Languages] import + update message. [Terrtia]

• [Languages] update. [Terrtia]

• [UI trackers] None trackers values. [Terrtia]

• [Tracker] edit tracker ACL. [Terrtia]

• [Yara Tracker] catch yara timeout. [Terrtia]

• [Terms Trackcers] fix item link. [Terrtia]

Other

• Merge pull request #70 from ail-project/gallypette-patch-1. [Jean-
  Louis Huynen]

  fix: [install] pycld3 dependency

• Merge branch 'master' of https://github.com/ail-project/ail-framework.
  [Terrtia]

AIL Framework version 3.3 released with an improved item view and many improvements

AIL Framework version 3.3 released with improved item view and many improvements

The AIL Framework version 3.3 released included a refactoring of the item view including a convenient html2text preview of item fetched.
Last origin table is now accessible for all domain crawled to show where the information is coming. Improved yara view in AIL to display the yara rule directly. Various bugs were fixed and bitcoin bech32 addresses are now supported.

Changes

• [UI show Item] refactoring + bootstrap 4 migration. [Terrtia]
• [UI show domain] add last origin table: domain + item. [Terrtia]
• [travis] -> bionic. [Alexandre Dulaunoy]
• [yara trackers UI] add yara trackers, show default yara rule.
  [Terrtia]
• [bitcoin tags + correlation] add bech32 adresses. [Terrtia]

Fix

• [UI correlation graph] fix item links. [Terrtia]
• [UI correlation graph] fix item links. [Terrtia]
• [users management] hide API keys by default + fix delete/edit users.
  [Terrtia]
• [crawler] fix ResponseNeverReceived retry time. [Terrtia]
• [crawler] fix ResponseNeverReceived hanlder, check if splash
  restarted. [Terrtia]
• [Yara tracker] fix mail notification. [Terrtia]
• [Yara tracker] fix mail notification. [Terrtia]
• [RegexTracker] fix missing hostname in mail notification. [Terrtia]
• [Flask server] change default host. [Terrtia]
• [ardb] switch to ail-project ardb fork #38. [Jean-Louis Huynen]
• [Launcher] fix virtualenv loader. [Terrtia]

AIL Framework version 3.2 released with new YARA detection engine with a default set of search patterns, many new correlations type and many bugs fixed.

AIL Framework version 3.2 released with new YARA detection engine with a default set of search patterns, many new correlations type and many bugs fixed.

Changes

• [update] add v3.2. [Terrtia]
• [yara trackers] add debug. [Terrtia]
• [tracker yara] show rule content. [Terrtia]
• [correlation] basic correlation: get_correlation_all_object function.
  [Terrtia]
• [submodule + YARA] add submodule auto update + update v3.1.1.
  [Terrtia]
• [trackers] add yara trackers. [Terrtia]
• [Trackers] email notifications: add tracker description in email
  subject. [Terrtia]
• [username correlation + login redirection] add twitter username
  correlation + redirect to the requested page on login. [Terrtia]
• [Item delete] delete father/child link + remove from domain tree +
  delete all child from the same domain. [Terrtia]
• [feeder pystemon] add debug. [Terrtia]
• [whosh index] add data retention fct. [Terrtia]
• [core] disable Release module. [Terrtia]
• [tor crawler] nyt added. [Alexandre Dulaunoy]
• [API + item library] add a way to check if a user can access the API +
  item import (use item_basic) [Terrtia]
• [black-list onion] keybase added. [Alexandre Dulaunoy]
• [core crawler] add screen script create screen + windows + kill them +
  get list. [Terrtia]
• [crawler proxy] interact with the splash manager API (get list of
  proxies + splash containers + launch them) [Terrtia]
• [launcher] add option to reset UI admin password. [Terrtia]
• [UI dashboard] sort module by name. [Terrtia]
• [LAUNCH] launch telegram module by default. [Terrtia]

Fix

• [install] canevasjs. [Terrtia]
• [UI tags] fix add tags template. [Terrtia]
• [install] travis git unshallow. [Terrtia]
• [trackers yara] remove sleep time. [Terrtia]
• [trackers] typo. [Terrtia]
• [updater] tag subversion. [Terrtia]
• [update v3.1.1] init submodule. [Terrtia]
• [travis] [Terrtia]
• [crawler] fix auto crawler creation. [Terrtia]
• [travis] fetch git last tag. [Terrtia]
• [update thirparty] fix canevasjs install. [Terrtia]
• [install crawler] remove old python requirement. [Terrtia]
• [TheHive feeder] create alert, get item full path. [Terrtia]
• [TheHive feeder] create alert. [Terrtia]
• Typo. [Terrtia]
• Typo. [Terrtia]
• Typo. [Terrtia]
• [username correlation] fix domain correlation. [Terrtia]
• [RegexTracker] fix search regex helper. [Terrtia]
• [regex_helper] [Terrtia]
• [regex tracker] fix timeout. [Terrtia]
• [reset_AIL] add helper + fix soft reset. [Terrtia]
• [Exporter] ad missing file. [Terrtia]
• [update_thirparty] fix popper install. [Terrtia]
• [UI popper version] use popper v1
  (twbs/bootstrap#29842) [Terrtia]
• [MISP auto Export] fix import path. [Terrtia]
• [MISP auto export] fix MISP_TheHive_feeder. [Terrtia]
• [travis] virtualenv. [Terrtia]
• [install] unshallow clone. [Terrtia]
• [requirement] remove old crawler requirements (installed by default)
  [Terrtia]
• [crawler] option to disable screenshots and har. [Terrtia]
• [crawlers lib] missing config file. [Terrtia]
• [create au to crawler] save crawler_delta time. [Terrtia]
• [updater] return boolean. [Terrtia]
• [Credential] fix timeout error. [Terrtia]
• [Credential] fix timeout error. [Terrtia]
• [Credential] force lowercase of mail address. [Terrtia]
• [Credential] fix timeout message. [Terrtia]
• [Mail module] regex timeout. [Terrtia]
• [update v1.5] [Terrtia]
• [Mail module] remove debug. [Terrtia]
• [Mail module] fix dns caching + use redis queue. [Terrtia]
• [UI Decoded items] fix download decoded file. [Terrtia]

Other

• Merge pull request #519 from myasn1k/docker_fixes. [Thirion Aurélien]

  Docker fixes

• Update update_thirdparty.sh. [Pietro Mazzini]

• Remove useless copy. [Pietro Mazzini]

• Uncomment update and explain in docker readme. [Pietro Mazzini]

• Add tz as argument. [Pietro Mazzini]

• Update docker readme (cp) [Pietro Mazzini]

• Update docker readme. [Pietro Mazzini]

• Fix canvas folder name. [Pietro Mazzini]

• Add to docker start crawlers launch. [Pietro Mazzini]

• Update Dockerfile: use ubuntu 18, remove redundant update_thirdparty,
  remove old crawler requirements pip install. [Pietro Mazzini]

• Comment git repo tag in installing_deps, git repo not recognised.
  [Pietro Mazzini]

• LAUNCH update function fail because the folder isn't a git repo,
  comment it. [Pietro Mazzini]

• Merge pull request #22 from C00kie-/master. [Alexandre Dulaunoy]

  allow to configure binding interface

• Update Flask_server.py. [C00kie-]

• Update core.cfg.sample. [C00kie-]

• Typo fixed as ail repo is now lower-case. [Alexandre Dulaunoy]

  Typo fixed as ail repo is now lower-case

• Merge pull request #16 from ChilliSec/patch-1. [Thirion Aurélien]

  Update HOWTO.md

• Update HOWTO.md. [ChilliSec]

• Merge branch 'master' of https://github.com/ail-project/ail-framework.
  [Terrtia]

• Merge pull request #497 from ronaldtf/fix-case-thehive. [Thirion
  Aurélien]

  Fix create case with TheHive

• Fix create case with TheHive. [Ronald Teijeira Fernandez]

AIL Framework version 3.1 with new crawling with cookies, Telegram username correlation, new external feeders (e.g. Twitter) and many improvements

AIL framework v3.1 released

AIL framework v3.1 has been released including many new features such as:

• Crawling website with a set of cookies collected from a browser (allowing to crawl website password protected or alike)
• An extraction module to find Telegram users
• New correlation engine for username starting with Telegram
• Improve timeout on various modules
• New JSON importer to be used with new external feeder. A first Twitter feeder and url-feeder is available to feed specific tweets and discovered urls in AIL.

The AIL project moved into a dedicated project to allow new contributions and project within the AIL project scope. If you want to join us and contribute new modules or specific feeders, don't hesitate to contact us.

Complete changelog

• [update] add update v3.1 + install crawler python requirements by
  default. [Terrtia]
• [UI correlation] add username correlation card. [Terrtia]
• [UI correlation] add username correlation graph. [Terrtia]
• [UI correlation] add username correlation: daterange pages. [Terrtia]
• [telegram module] add new tag: telegram invite code. [Terrtia]
• [telegram backend] add username correlation + save invite hash.
  [Terrtia]
• [importer] add map twitter id - item id, add parents link between
  twitter and url extracted. [Terrtia]
• [travis install] shallow clone. fetch tags + commit id. [Terrtia]
• [travis install] shallow clone. fetch tags. [Terrtia]
• [install] debug travis. [Terrtia]
• [README] update gitter link. [Terrtia]
• [import er url_extract] add item twitter parent. [Terrtia]
• [importer url_extract] fix item id. [Terrtia]
• [importer] add url_extract importer. [Terrtia]
• [API json import] take list as imput. [Terrtia]
• [core import] add AIL JSON import format, API endpoint added (AIL
  feeders) [Terrtia]
• [update doc] update doc install + logo + fix updater. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [Updater] change default branch origin. [Terrtia]
• [update Readme logo + links] [Terrtia]
• [add new logo] [Terrtia]
• [AIL logo UI] [Terrtia]
• [doc] add cookiejar screenshot. [Terrtia]
• [Crawler] default docker memory usage. [Terrtia]
• [MISP export] export domain as domain-crawled object. [Terrtia]
• [domain explorer UI] add shortcut button: misp export + correlation.
  [Terrtia]
• [crawler] edit cookie and cookiejar + add cookie to cookiejar + fix
  screenshot duplicate. [Terrtia]
• [Crawler core + UI] crawler lua: handle retry + fix cookie loader and
  selector. [Terrtia]
• [cookiejar UI] add cookiejar + show all. [Terrtia]
• [crawler cookies] use cookiejar. [Terrtia]
• [crawler - cookies] add/show/select cookies. [Terrtia]
• [crawler] add cookies list by user/global, save cookies from file +
  dict(name, value), TODO: API + handle errors. [Terrtia]
• [crawler] bypass login: use cookie provided by user and accept cookie
  from server + refractor. [Terrtia]
• [Splash Crawler] use cookies to bypass login. [Terrtia]
• [UI] bump jquery to 3.4.1. [Terrtia]
• [import_dir] fix is_gzip test, use magic number. [Terrtia]
• [MISP Importer] files: handle missing sha1/sha256 attributes + fix
  Items and Screenshots dir. [Terrtia]
• [Updater] relauch updater on change (git pull) [Terrtia]

Fixes

• [Mail module] replace signal by multiprocessing
  (https://docs.python.org/3.4/library/signal.html#execution-of-python-
  signal-handlers) [Terrtia]
• [Mail] debug signal. [Terrtia]
• [Mail] debug. [Terrtia]
• [Mail module] remove test time.wait. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [Mail module] debug signal timeout. [Terrtia]
• [telegram module] typo. [Terrtia]
• [telegram module] fix tagging. [Terrtia]
• [telegram module] check username length. [Terrtia]
• [telegram module] remove debug. [Terrtia]
• [Mails] regex timeout. [Terrtia]
• [Mails] change module output. [Terrtia]
• [Mails] remove print + test. [Terrtia]
• [Mails] refactor Mail module. [Terrtia]
• [urlextract importer] fix parent map + replace set JSON queue by list.
  [Terrtia]
• [import urlextract parent] fix typo. [Terrtia]
• [travis install] shallow clone. fetch missing commit id. [Terrtia]
• [Mails regex timeout] reduce default timeout. [Terrtia]
• [Mails dns resolver] update timeout exception. [Terrtia]
• [Mails] typo. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] import. [Terrtia]
• [Mails] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [Credential] add regex timeout. [Terrtia]
• [urlexport importer] fix item name + redis config. [Terrtia]
• [urlexport importer] item_id, force str type. [Terrtia]
• [urlexport importer] add missing import. [Terrtia]
• [urlextract importer] class name. [Terrtia]
• [importer] fix typo. [Terrtia]
• [Global] extend ungzip error catching, catch invalid compressed file.
  [Terrtia]
• [json import API] remove list input. [Terrtia]
• [install create default user] default passwd file: add missing new
  line. [Terrtia]
• [installer] get last git version. [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme travis badge] [Terrtia]
• [Readme logo size] [Terrtia]
• [show item UI] return 404. [Terrtia]
• [Cookiejar UI] fix typo. [Terrtia]
• [Flask session cookie name] add uuid to cookie name. [Terrtia]
• [crawler] typo. [Terrtia]
• [Crawler splash ResponseNeverReceived] add retry. [Terrtia]
• [crawler] error catcher. [Terrtia]
• [MISP export UI] fix input: name + value overwrite. [Terrtia]
• [update thirdparty] update taxonomies. [Terrtia]
• [crawler] cleanup. [Terrtia]
• [PgpDump parser] remove header comment (rfc4880) + remove empty lines.
  [Terrtia]
• [Pgp Dump] remove tool version. [Terrtia]
• [ZMQ Feeder] performance: replace zmq recv NOBLOCK by Poller.
  [Terrtia]
• [pgpdump] fix subtype save. [Terrtia]
• [Updater] force updater update. [Terrtia]
• [Updater] fix current_tag parser. [Terrtia]
• [import_dir] remove special characters. [Terrtia]
• [import_dir] remove dir whitespaces #475. [Terrtia]

Other

• Merge pull request #492 from sunil3590/master. [Thirion Aurélien]

  crawler_time -> crawler_delta

• Crawler_time -> crawler_delta. [Sunil D S]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Chg [telegram + correlation] new module: telegram (username + login
  code + join_chat) + add simple_correlation backend. [Terrtia]

• Merge pull request #487 from CIRCL/crawler_v2. [Thirion Aurélien]

  fix: [crawler] error catcher

• Merge pull request #486 from CIRCL/crawler_v2. [Thirion Aurélien]

  Crawler v2 - Add cookiejar - use cookie to bypass login form

• Merge branch 'master' into crawler_v2. [Terrtia]

AIL Framework version 3.0 with full MISP format export/import and small improvements

AIL Framework version 3.0 with full MISP format export and import has been released. AIL users can now export a set of selected items as a MISP event including objects (items, decoded, screenshot, pgp...), correlations and metadata. There is an import to allow analyst to keep a specific set of analysis in AIL for moving across AIL instances. This release also includes support for authentication of the SMTP server (thanks to Mike Peters for the contribution). Multiple bugs fixed and small improvements.

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

AIL Framework version 2.9 released with a critical security fix (CVE-2020-8545) and minor bugs fixed

This release of AIL includes a major security fix for CVE-2020-8545. The security vulnerability was in the handler of the global feed which could allow malicious feed providers to overwrite and potentially, execute Python code in the environment. This release also includes various bugs fixed. We urge users to upgrade as soon as possible.

Changes

• [domain explorer] domains explorer v2, filter domains by daterange.
  [Terrtia]

Fix

• [IPAddress] catch empty config error. [Terrtia]
• [Global: already saved filename] save updated + filter duplicated
  items. [Terrtia]
• [Global: filename provided by all feeders] avoid path tranversal.
  [Terrtia]
• [Domain explorer UI] fix daterange pagination links. [Terrtia]
• [Tag core] check if item_date type is an integer. [Terrtia]