Skip to content
This repository has been archived by the owner on Apr 26, 2021. It is now read-only.

Azure Machinery for Cuckoo #3120

Open
wants to merge 64 commits into
base: master
Choose a base branch
from
Open

Azure Machinery for Cuckoo #3120

wants to merge 64 commits into from

Commits on Oct 8, 2020

  1. Initial commit for Azure machinery

    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    a7c288f View commit details
    Browse the repository at this point in the history

  2. NICs weren't being deleted when instance creation failed

    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    e3cd8ff View commit details
    Browse the repository at this point in the history

  3. Moved API calls to the beginning, since they were being made more oft… 

    …en than required
    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    5e69fa6 View commit details
    Browse the repository at this point in the history

  4. Adding ability to specify devops environment, such that multiple Cuck… 

    …oo envs can exist in same resource group
    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    534fa13 View commit details
    Browse the repository at this point in the history

  5. Added ability to use multiple guest os versions

    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    d01b16e View commit details
    Browse the repository at this point in the history

  6. Removing files deleted by AV

    @cccs-kevin
    cccs-kevin committed Oct 8, 2020
    Configuration menu
    Copy the full SHA
    b865fff View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2020

  1. Refactoring az.py so that scaling is in tune with DB, also edited som… 

    …e other files to fit cloud application
    @cccs-kevin
    cccs-kevin committed Oct 14, 2020
    Configuration menu
    Copy the full SHA
    00d7e9b View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2020

  1. Added more comments, updated test conf files

    @cccs-kevin
    cccs-kevin committed Oct 15, 2020
    Configuration menu
    Copy the full SHA
    f7376d7 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2020

  1. Some renaming of methods, added handling of snapshot id for non-reque… 

    …sted type
    @cccs-kevin
    cccs-kevin committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    07ded61 View commit details
    Browse the repository at this point in the history

  2. Correcting method name

    @cccs-kevin
    cccs-kevin committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    4ee39d0 View commit details
    Browse the repository at this point in the history

  3. Mistake on naming methods addressed

    @cccs-kevin
    cccs-kevin committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    1874a29 View commit details
    Browse the repository at this point in the history

  4. Added collision avoidance on instances

    @cccs-kevin
    cccs-kevin committed Oct 16, 2020
    Configuration menu
    Copy the full SHA
    783f41d View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2020

  1. Adding platform for Ubuntu

    @cccs-kevin
    cccs-kevin committed Oct 19, 2020
    Configuration menu
    Copy the full SHA
    2f63291 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2020

  1. Adding better logging regarding acquiring machines

    @cccs-kevin
    cccs-kevin committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    d830b73 View commit details
    Browse the repository at this point in the history

  2. Moving acquire logging to machinery

    @cccs-kevin
    cccs-kevin committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    889c031 View commit details
    Browse the repository at this point in the history

  3. Tweaking what tags contains when being sent to acquire in abstracts

    @cccs-kevin
    cccs-kevin committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    8393407 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2020

  1. Adding case if no machine exists in DB

    @cccs-kevin
    cccs-kevin committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    2147975 View commit details
    Browse the repository at this point in the history

  2. Fine-tuning the acquire method

    @cccs-kevin
    cccs-kevin committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    c5dc312 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2020

  1. Adding method to tag disk, adding class constants

    @cccs-kevin
    cccs-kevin committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    84b4698 View commit details
    Browse the repository at this point in the history

  2. Small logic tweak

    @cccs-kevin
    cccs-kevin committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    52408ed View commit details
    Browse the repository at this point in the history

  3. Correcting platform value for Ubuntu

    @cccs-kevin
    cccs-kevin committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    f0b565b View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2020

  1. Revised marking disks for deletion

    @cccs-kevin
    cccs-kevin committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    7e2966d View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2020

  1. Adding logging for api limits

    @cccs-kevin
    cccs-kevin committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    d8631d7 View commit details
    Browse the repository at this point in the history

  2. Reworking Azure API errors, resizing machines being created when vm f… 

    …ails
    @cccs-kevin
    cccs-kevin committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    0b2d21d View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2020

  1. Fixing logic bug in peak throuput limiting

    @cccs-kevin
    cccs-kevin committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    0779544 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2020

  1. Replaced API calls that would mark nic/disk for deletion with set

    @cccs-kevin
    cccs-kevin committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    abc39a8 View commit details
    Browse the repository at this point in the history

  2. Adding polling thread that deletes leftover resources

    @cccs-kevin
    cccs-kevin committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    3af045e View commit details
    Browse the repository at this point in the history

  3. Giving the people what they want!

    @cccs-kevin
    cccs-kevin committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    24475f3 View commit details
    Browse the repository at this point in the history

  4. Have patience!

    @cccs-kevin
    cccs-kevin committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    bb79317 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2020

  1. Added clause that wil delete hanging VMs that were created by Cuckoo

    @cccs-kevin
    cccs-kevin committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    e2afa6e View commit details
    Browse the repository at this point in the history

  2. Fixed logical bug where in some cases the availables weren't relevant… 

    … and therefore not acquired and batch creation was not happening
    @cccs-kevin
    cccs-kevin committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    f22ffd5 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2020

  1. Adding ability to use Azure Spot Instances by checking for VM status … 

    …and creating a couple pollers that monitor the pools
    @cccs-kevin
    cccs-kevin committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    4e770c1 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2020

  1. Tweaked logic in scheduler such that a pending task will only be fetc… 

    …hed if there is an available relevant machine for it. Also made machine_queue global
    @cccs-kevin
    cccs-kevin committed Dec 10, 2020
    Configuration menu
    Copy the full SHA
    3893201 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2021

  1. Additional machinery for Azure using virtual machine scale sets

    @cccs-kevin
    cccs-kevin committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    c610585 View commit details
    Browse the repository at this point in the history

  2. Updating VMSS machinery

    @cccs-kevin
    cccs-kevin committed Jan 4, 2021
    Configuration menu
    Copy the full SHA
    bb43937 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2021

  1. Small tweaks

    @cccs-kevin
    cccs-kevin committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    6e45832 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2021

  1. Adding additional checks to ensure up-time

    @cccs-kevin
    cccs-kevin committed Jan 15, 2021
    Configuration menu
    Copy the full SHA
    ba84673 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2021

  1. Improved error-handling and logging

    @cccs-kevin
    cccs-kevin committed Jan 19, 2021
    Configuration menu
    Copy the full SHA
    f3a1418 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2021

  1. More thorough error-handling

    @cccs-kevin
    cccs-kevin committed Jan 25, 2021
    Configuration menu
    Copy the full SHA
    960f315 View commit details
    Browse the repository at this point in the history

Commits on Jan 27, 2021

  1. Adding logic to handle Azure error states, improved error-handling an… 

    …d logging
    @cccs-kevin
    cccs-kevin committed Jan 27, 2021
    Configuration menu
    Copy the full SHA
    a7c28ce View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2021

  1. Wrapping _thr_scale_machine_pool in try-catch to find errors, improvi… 

    …ng logging, adding ability to acquire by machine_id
    @cccs-kevin
    cccs-kevin committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    2a4c142 View commit details
    Browse the repository at this point in the history

Commits on Feb 2, 2021

  1. Adding more logging, additional logic in scaling machine pools, and f… 

    …ixing ClientRequestError bug
    @cccs-kevin
    cccs-kevin committed Feb 2, 2021
    Configuration menu
    Copy the full SHA
    0d5d672 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2021

  1. Removed bottleneck when dealing with multiple tasks directed for diff… 

    …erent machine pools
    @cccs-kevin
    cccs-kevin committed Feb 3, 2021
    Configuration menu
    Copy the full SHA
    462b726 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2021

  1. Improving exception handling, adding method to recreate failed VMSS

    @cccs-kevin
    cccs-kevin committed Mar 15, 2021
    Configuration menu
    Copy the full SHA
    85c7b68 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2021

  1. Adding default tag value

    @cccs-kevin
    cccs-kevin committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    6e3df73 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2021

  1. Optimizing speed at which a task comes off the queue and is assigned … 

    …to a machine
    @cccs-kevin
    cccs-kevin committed May 5, 2021
    Configuration menu
    Copy the full SHA
    ec0dd19 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2021

  1. Ordering tasks by FIFO

    @cccs-kevin
    cccs-kevin committed May 13, 2021
    Configuration menu
    Copy the full SHA
    13af923 View commit details
    Browse the repository at this point in the history

  2. Adding catch for psycopg UniqueViolation when adding sample

    @cccs-kevin
    cccs-kevin committed May 13, 2021
    Configuration menu
    Copy the full SHA
    3e60bc0 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2021

  1. Adding try catch for better logging

    @cccs-kevin
    cccs-kevin committed May 14, 2021
    Configuration menu
    Copy the full SHA
    25c7ecc View commit details
    Browse the repository at this point in the history

Commits on May 17, 2021

  1. Adding platform support

    @cccs-kevin
    cccs-kevin committed May 17, 2021
    Configuration menu
    Copy the full SHA
    0923f0c View commit details
    Browse the repository at this point in the history

Commits on May 25, 2021

  1. Scaling based on platform

    @cccs-kevin
    cccs-kevin committed May 25, 2021
    Configuration menu
    Copy the full SHA
    564a166 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2021

  1. Preventing IP collision

    @cccs-kevin
    cccs-kevin committed May 28, 2021
    Configuration menu
    Copy the full SHA
    ff31944 View commit details
    Browse the repository at this point in the history

  2. Adding logging if machine doesn't exist upon delete

    @cccs-kevin
    cccs-kevin committed May 28, 2021
    Configuration menu
    Copy the full SHA
    a5cb11b View commit details
    Browse the repository at this point in the history

Commits on May 31, 2021

  1. Cannot use Manual

    @cccs-kevin
    cccs-kevin committed May 31, 2021
    Configuration menu
    Copy the full SHA
    59edd91 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2021

  1. Revamp stop() and delete_vm_from_vmss() to avoid errors in Azure

    @cccs-kevin
    cccs-kevin committed Jun 4, 2021
    Configuration menu
    Copy the full SHA
    a3d7afc View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2021

  1. This polling is required if the Azure VM agent is not installed in VMs

    @cccs-kevin
    cccs-kevin committed Jun 10, 2021
    Configuration menu
    Copy the full SHA
    b3f174a View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2021

  1. New information about how Azure VMSSs work caused more reliance on ba… 

    …tching and concurrent operations
    @cccs-kevin
    cccs-kevin committed Jun 18, 2021
    Configuration menu
    Copy the full SHA
    0165f37 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2021

  1. Adding missing global variable

    @cccs-kevin
    cccs-kevin committed Jun 29, 2021
    Configuration menu
    Copy the full SHA
    060e850 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2021

  1. Removing VMs in delete phase if error occurs

    @cccs-kevin
    cccs-kevin authored Jul 6, 2021
    Configuration menu
    Copy the full SHA
    03f50f2 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2021

  1. Making program threadsafe, adding handling to spot instance deletion … 

    …in reimaging
    @cccs-kevin
    cccs-kevin committed Jul 14, 2021
    Configuration menu
    Copy the full SHA
    b4a41f1 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2021

  1. Adding handling for 'badrequest' Azure error

    @cccs-kevin
    cccs-kevin committed Jul 16, 2021
    Configuration menu
    Copy the full SHA
    95eacd2 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2021

  1. Bug fix if tag doesn't exist

    @cccs-kevin
    cccs-kevin committed Aug 3, 2021
    Configuration menu
    Copy the full SHA
    43455aa View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2022

  1. Adding a scaling ceiling based on quota availability

    @cccs-kevin
    cccs-kevin committed Apr 25, 2022
    Configuration menu
    Copy the full SHA
    00f318e View commit details
    Browse the repository at this point in the history

Commits on May 3, 2022

  1. Fixing bug in usage quota logic

    @cccs-kevin
    cccs-kevin committed May 3, 2022
    Configuration menu
    Copy the full SHA
    74ad92c View commit details
    Browse the repository at this point in the history