diff --git a/docs/about-me/projects.md b/docs/about-me/projects.md index 5a59c115b3a..c1534973a6f 100755 --- a/docs/about-me/projects.md +++ b/docs/about-me/projects.md @@ -2,7 +2,7 @@ ### Stashfin Business (Feb'20 - Feb'23) -[Stashfin Business](about-me/projects/89-stashfin.md) +[Stashfin Business](about-me/projects/80-stashfin.md) ### Stashfin Infra / DevOps (Feb'20 - Jan'21) diff --git a/docs/about-me/projects/70-interview-datainsight-alerting-engine.md b/docs/about-me/projects/70-interview-datainsight-alerting-engine.md new file mode 100644 index 00000000000..3a9a26003be --- /dev/null +++ b/docs/about-me/projects/70-interview-datainsight-alerting-engine.md @@ -0,0 +1,173 @@ +# Interview - Datainsight / Alerting Engine + +## Data Insight + +Your company is developing a new cloud-based analytics platform, "DataInsight," designed to provide advanced data processing and visualization capabilities for enterprise clients. The platform integrates with various data sources (telemetry metrics) utilizes machine learning for predictive analytics, and offers a highly interactive and customizable dashboard for end-users. + +### Technical architecture +- Ingestion to database +- Database to services (visualization and analytics including ML) + +### Integration Setbacks (Technical Problem) + +The platform struggles to integrate smoothly with a +- wide range of customer databases especially legacy systems +- causing delays +- data inconsistency issues. +- Common set of communication protocols from client sensors too + +#### Assumption +- We have standardized data at our end in Influxdb + - Clients (50-5000) - + - Branches / Locations - (less - 500-2,50,000) + - Sensors - measurement (500 sensors, millions) + - Table schema + - (row oriented, column oriented, non-normalized) + - Clients + - client_id pk + - Locations + - client fk + - location_id pk + - Raw Sensor Data + - location_id fk + - sensor_type_fk - fk + - values + - timestamp + - Sensor Type + - id + - sensor_id - 5 + - parameter - 1 (temp) + - metadata - temp (5, 1), humidity (5, 2) + - pk - (sensor_id + parameter) + - Query + - Aggregate - last week average + - 15-16 values + - Device (not a table) - AC + - What temp i am set at + - What current and voltage i am consuming +- Multi-tenant / single-tenant + - multi-tenant system + +#### Problems +- wide range of customer databases + +#### Solutions +- Standard set of Ingestion APIs, so integration should be simpler +- Producers can be standardized +- Serialization / deserialization + +### Technological Hurdles (Technical Problem) + +The machine learning models are not performing as expected, leading to inaccurate predictions and inefficient data processing. The cause is unclear, whether it's due to poor model selection, inadequate training data, or something else. + +#### Visualization (end to end from databases to visualization service) + +##### Assumption +1. Sensor + +##### Tech architecture + +- Database +- Backend service + - APIs for highcharts that will give the sensor data + - Batch reporting apis +- Frontend service - show users the end data + - IAM + - React / Angular - Client side rendered + - Client side app + - Don't need to send containers all the time + - Responsive - Desktop / Mobile + - Visualization library - highcharts + +##### Questions +1. How much data (10 years, 1 years) + 1. Roll ups +2. Accuracy of old data +3. Data tiering +4. Query speed + 1. Batch Reporting - for downloading large datasets, export + 1. Add queueing and do asynchronous processing + 2. Wait for data, either in mail, or downloads page + 2. **Real time analytics** + +## Low Latency Alerting System + +Time series data is being published from loT devices(MQTT) across the building. There are various data types like efficiency, energy consumption, equipment on/off, etc. + +The user wants different categories of alerts like critical, high, medium, and low. + +- Critical - need to be initiated instantly. (within milliseconds) +- High - need to be initiated within 10 seconds to 1 minute. +- Medium - need to be initiated within 5 minutes - 6 hours. +- Low- can be sent up to once a day or week. + +Whenever an alerting condition is met, it should trigger an alert and send it to all subscribed users via their preferred mode in the given below template- + +- Alert Summary + - Timestamp + - Priority +- Asset Name + - Suggested Action For e.g Most likely Automation commands are not working on the chiller, Please switch the chiller to JouleTrack mode, turn it on, and then return it to recipe mode. + +What all different system components can be brought together to send user alerts on different mediums (push notification, whatsapp, email, message)? + +Instruction + +1. Please create a component level diagram to discussion the solution design +2. Solution should be low latency, fault-tolerance and distributed in nature. It can be easily scaled to serve millions of monthly alerts to users. +3. Data modeling diagram to represent how things are connected to each other and how data is flowing. +4. Trade Off decisions should be highlighted. + +### Questions + +1. size of data? +2. Rate of stream? - 10K per second + +### Solution + +- Alert mapping table (in memory or redis) +- Subscribe to mqtt topics +- Log to alerts too - rdms +- Async processing +- Better processing + - Kafka streams + - Apache Pulsar + - Druid +- Consumer Lag + +## Architecture + +![mqtt-alerting-engine](../../media/MQTT%20Alerting%20Engine.drawio.png) + +- Alert mapping table - Main copy in RDBMS + - Pushed copy in redis + - Updates pushed to redis from RDMS whenever changed +- Backend service - mqtt broker to redis streams conversion for creating consumer group +- Redis streams (single topic) +- Backend service - alert processing module + - Process the packet + - match it with alert mapping table + - push to redis streams communication channel +- Backend service - alerting engine + - Log the incoming alert + - Call the communications api (based on alert mapping, or alert processing module can **send the channel too**) + - Whatsapp + - Email + - SMS + - Push + - Telegram, etc + - Can send multiple customers + - Webhook for delivery status +- Scaling ways + - Multithreading + - Asyncio + - Horizontal Scaling + +##### Followups +- Duplicate +- Automatic resolutions +- Fire forget + +### Alerting Exceptions Handling + +![alerting-exceptions-handling-flow](../../media/Communication%20exception%20flow.png) diff --git a/docs/about-me/projects/89-stashfin.md b/docs/about-me/projects/80-stashfin.md similarity index 90% rename from docs/about-me/projects/89-stashfin.md rename to docs/about-me/projects/80-stashfin.md index c78b5664381..dd2ab42d2d5 100644 --- a/docs/about-me/projects/89-stashfin.md +++ b/docs/about-me/projects/80-stashfin.md @@ -4,6 +4,7 @@ [NBFC / Banking Terms](economics/nbfc-banking-terms.md) +![stashfin-product-architecture](../../media/Pasted%20image%2020231201172554.png) ## Processes ### Dev Onboarding diff --git a/docs/about-me/projects/86-stashfin-customer-support.md b/docs/about-me/projects/86-stashfin-customer-support.md new file mode 100644 index 00000000000..506ce255e74 --- /dev/null +++ b/docs/about-me/projects/86-stashfin-customer-support.md @@ -0,0 +1,49 @@ +# Stashfin Customer Support + +### Improvements + +#### Proactive Communication + +- Implement timely notifications for users about upcoming payments, EMIs, and system maintenance. +- Provide clear and transparent information on payment breakdowns and costs. + +#### Enhanced User Interface + +- Redesign the online platform for a user-friendly experience. +- Include easy-to-use payment options and troubleshoot common issues. + +#### Real-time Issue Resolution + +- Introduce a real-time customer support chat for immediate issue resolution. (YM) +- Establish a dedicated support team to address payment-related concerns promptly. + +#### Education and Support + +- Develop user guides and FAQs to educate customers about the lending platform. +- Offer online tutorials and resources to guide users through the payment process. + +#### Others + +- Added Hindi language support +- CSAT of agents  +- Moved from fixed pricing for CS agents to dynamic pricing where based on CSAT, payments will be done +- Canned responses for CS agents for increased productivity + +### KPIs + +- Total tickets +- Number of unique customers +- **Tickets > 24 hours** +- Ticket Resolution TAT +- Ticket Closed TAT +- Ticket First Contact TAT +- Chat First Contact TAT +- **Ticket Reopen Count** +- **Ticket Reopen Percent** +- Play Store Rating +- Quality Audit Score +- RBI Escalation + +### Creating and maintaining product roadmaps + +![product-roadmap-example](../../media/Pasted%20image%2020231201183958.png) diff --git a/docs/about-me/projects/87-stashfin-team-management-culture.md b/docs/about-me/projects/87-stashfin-team-management-culture.md new file mode 100644 index 00000000000..efdeeb668d5 --- /dev/null +++ b/docs/about-me/projects/87-stashfin-team-management-culture.md @@ -0,0 +1,58 @@ +# Stashfin Team Management / Culture + +### Hiring + +- Automated the hiring process +- Hired around 12 freshers and 5 senior developers +- Processes around onboarding and automated team pulses +- Onboarding tasks + manager and team introductions + onboarding buddy + +### Culture + +- Processes around reviews, appraisals and feedbacks +- Project management tool - first gitlab issue boards, then moved to JIRA +- Daily standups + scrums +- Documentation + +### Mandatory Code Reviews + +- Atleast 2 approvals, one from senior dev and one from junior dev is mandatory for merging the code +- Using Git and a proper PR process. Every feature or bug fix is a separate branch and submitted as a PR + +![stashfin-git-review-process](../../media/Pasted%20image%2020231201181214.png) + +### Scrum / Kanban / Project Management + +- Implemented Agile project management methodology across teams + +![example-scrum-board](../../media/Pasted%20image%2020231201181414.png) + +## Documentation + +- Used a combination of google docs with team folders, etc +- Introduced confluence for documentation + +![example-confluence-documentation](../../media/Pasted%20image%2020231201181347.png) + +### Process process for documentation + +#### ADRs (Architecture Design Records) and HLD (High Level Diagrams) + +- ADRs are documents that capture the important decisions regarding the architecture of our software. They serve as a record of the context, options considered, and the rationale behind the chosen solution. +- HLD provides an overview of the system architecture, major components, and their interactions. It helps in aligning the team and stakeholders on the overall structure of the application. +- We should update the HLD whenever there are significant changes to the system architecture. It serves as a reference for new team members and ensures everyone has a shared understanding of the system. + +![high-level-diagram-example](../../media/Pasted%20image%2020231201183011.png) + +#### LLD (Low Level Diagrams) and ER (Entity Relationship Diagrams) + +- LLD dives into the details of individual components or modules. It includes class diagrams, data flow diagrams, and other specifics that guide the implementation. +- LLD documents are often created in collaboration with the development team. They serve as a valuable resource during the implementation phase and aid in code reviews. +- ERDs visually represent the relationships between entities in our database. They are crucial for understanding the data model and ensuring that it aligns with the requirements. +- Each table and its relationships are clearly defined in the ERD, making it easier for developers, database administrators, and stakeholders to comprehend the data structure. +- We include ERDs as part of our documentation to maintain a clear understanding of the database schema. This is especially helpful during database migrations or when onboarding new team members. +- During code reviews or discussions about database changes, referring to the ERD ensures that everyone is on the same page regarding the data model. + +![low-level-diagram-example](../../media/Pasted%20image%2020231201183115.png) + +![entity-relationship-diagram](../../media/Pasted%20image%2020231201183143.png) diff --git a/docs/about-me/projects/88-stashfin-security-iam-apis.md b/docs/about-me/projects/88-stashfin-security-iam-apis.md new file mode 100644 index 00000000000..31fadfef5af --- /dev/null +++ b/docs/about-me/projects/88-stashfin-security-iam-apis.md @@ -0,0 +1,59 @@ +# Stashfin Security / IAM / APIs + +## Security + +### External + processes + +Followed - AAA - Authentication, Authorization and Audit with best practices + +- Cloudflare WAF + fixing all our own APIs that were getting blocked because of using unsafe practices like non-escaped inputs, unsanitized inputs, etc +- Regular VAPT tests and fixing each vulnerability, scheduled the reports monthly, so in monthly hackathons, whole team can address multiple vulnerabilities in one sitting +- Regularly and automated rotating all keys +- Service mesh + Cloudflare for monitoring all APIs and status codes + +### Internal + +- Added private VPN and blocked all internet traffic coming from outside to internal applications, added openvpn for developers to access internal resources and jump server to access internal compute instances +- Eventually moved to zero trust access, between applications too, where each application will have it’s own api keys to access another application. +- Centralized internal IAM for internal users’ applications access using keycloak for devops resources and django-admin for other applications +- Using groups for permissions and adding users to groups instead of individual permissions +- Immutable audit logs for all changes +- No access to production databases +- Rate limits for ddos protection in open public client APIs + +### Immutable Logs for Audit + +![stashfin-immutable-audit-logs](../../media/Pasted%20image%2020231201175020.png) + +### Authorization + +![stashfin-authorization](../../media/Pasted%20image%2020231201175035.png) + +### Postman implementation and documentation of all APIs + +[Stashfin Partners API](https://documenter.getpostman.com/view/16927648/TzzGGtg9) + +![stashfin-screenshot](../../media/Pasted%20image%2020231201175731.png) + +![stashfin-screenshot](../../media/Pasted%20image%2020231201175751.png) + +#### API Testing + +![stashfin-screenshot](../../media/Pasted%20image%2020231201175759.png) + +## WebView Implementations + +### WebView inside Apps + +- Customer support +- Payments +- New products + - referral + - brand ambassador program + - stashearn + +![stasfin-screenshot](../../media/Pasted%20image%2020231201180310.png) + +![stashfin-screenshot](../../media/Pasted%20image%2020231201180349.png) + +![stashfin-screenshot](../../media/Pasted%20image%2020231201180442.png) diff --git a/docs/about-me/projects/89-stashfin-devops-overhaul.md b/docs/about-me/projects/89-stashfin-devops-overhaul.md new file mode 100644 index 00000000000..bf5f82a7792 --- /dev/null +++ b/docs/about-me/projects/89-stashfin-devops-overhaul.md @@ -0,0 +1,42 @@ +# Stashfin DevOps Overhaul + +- Scaled the stack and team to increase loan disbursals from 100K USD to 60M USD per month +- Increased the resiliency and stability of the system many fold. Achieved 99.99% of infrastructure uptime +- Moved all workloads to Kubernetes along with full monitoring, alerting and logging solution around applications +- Added Jenkins CICD and fully automated integration, testing and deployment pipelines for all applications + +## Screenshots + +![stashfin-jenkins-screenshot](../../media/Pasted%20image%2020231201181252.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173646.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173733.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173742.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173759.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173816.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173823.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173831.png) + +![stashfin-devops-screenshot](../../media/Pasted%20image%2020231201173837.png) + +## Keeping all repositories consistent and follow same standards across + +- Added Auto formatters and static code analysis +- [Pre-commit](https://deepaksood619.github.io/python/documentation/27-development-tools/static-code-analysis) for running all hooks before commit locally +- Black (opinionated) for Python - auto formatters +- Keeping code complexity low for slowly reducing the function and file sizes +- [Code smells](https://deepaksood619.github.io/computer-science/software-engineering/code-smell) + +![autoformatter-implementation](../../media/Pasted%20image%2020231201182421.png) + +![autoformatter-implementation](../../media/Pasted%20image%2020231201182427.png) + +### SonarQube / snyk - Continuous Code Quality Inspector + +![sonarqube-implementation](../../media/Pasted%20image%2020231201182608.png) diff --git a/docs/about-me/projects/95-zenatix-solutions.md b/docs/about-me/projects/95-zenatix-solutions.md index a980cb8f1b2..8df84e51ab5 100644 --- a/docs/about-me/projects/95-zenatix-solutions.md +++ b/docs/about-me/projects/95-zenatix-solutions.md @@ -120,6 +120,8 @@ Demo, presentation and meetings with Director General (Shri Abhay Bakre) and Dir - Nginx setup as web server - Automation using Ansible +![kiali-screenshot](../../media/Pasted%20image%2020231201175111.png) + ### Data Engineering - Kafka production deployment diff --git a/docs/about-me/projects/readme.md b/docs/about-me/projects/readme.md new file mode 100644 index 00000000000..3e7cfe3a926 --- /dev/null +++ b/docs/about-me/projects/readme.md @@ -0,0 +1,12 @@ +# Readme + +- [70-interview-datainsight-alerting-engine](about-me/projects/70-interview-datainsight-alerting-engine.md) +- [80-stashfin](about-me/projects/80-stashfin.md) +- [86-stashfin-customer-support](about-me/projects/86-stashfin-customer-support.md) +- [87-stashfin-team-management-culture](about-me/projects/87-stashfin-team-management-culture.md) +- [88-stashfin-security-iam-apis](about-me/projects/88-stashfin-security-iam-apis.md) +- [89-stashfin-devops-overhaul](about-me/projects/89-stashfin-devops-overhaul.md) +- [90-stashfin-tech-stack-infra](about-me/projects/90-stashfin-tech-stack-infra.md) +- [95-zenatix-solutions](about-me/projects/95-zenatix-solutions.md) +- [96-zenatix-architecture](about-me/projects/96-zenatix-architecture.md) +- [99-ciso-cybersecurity](about-me/projects/99-ciso-cybersecurity.md) diff --git a/docs/about-me/readme.md b/docs/about-me/readme.md index c8449ccdb04..5473c2acb6d 100755 --- a/docs/about-me/readme.md +++ b/docs/about-me/readme.md @@ -3,6 +3,8 @@ - [Resume](https://bit.ly/deepak_sood_resume) - [Experience](experience) - [Projects](projects) + - [Interview - Datainsight / Alerting Engine](about-me/projects/70-interview-datainsight-alerting-engine.md) + - [All projects outline](about-me/projects/readme.md) - [Paper / Publications / References](paper-publications-references) - [Courses / Certifications](courses-certificates) - [Achievements / Activities / Awards](achievements-activities-awards) diff --git a/docs/ai/data-science/data-visualization/charts.md b/docs/ai/data-science/data-visualization/charts.md index 455794f631d..34c8149c356 100755 --- a/docs/ai/data-science/data-visualization/charts.md +++ b/docs/ai/data-science/data-visualization/charts.md @@ -157,3 +157,9 @@ 3. Filled-Area Animation 4. Visualizing MRI Volume Slices 5. Heatmap Animation + +## Libraries + +- [GitHub - highcharts/highcharts: Highcharts JS, the JavaScript charting framework](https://github.com/highcharts/highcharts) +- [Chart.js | Open source HTML5 Charts for your website](https://www.chartjs.org/) +- [10+ Free JavaScript Chart Library📊you must use in 2023. - DEV Community](https://dev.to/themeselection/10-javascript-chart-library-you-must-use-k20) diff --git a/docs/computer-science/links.md b/docs/computer-science/links.md index b4b06f278de..5759b3e3f20 100644 --- a/docs/computer-science/links.md +++ b/docs/computer-science/links.md @@ -16,6 +16,21 @@ 12. [Effective and Efficient Observability with OpenTelemetry - YouTube](https://www.youtube.com/watch?v=kh_IZlSj8DU) 13. [130 People – No Bosses: Too Good To Be True? in 4 Minutes • Helle Markmann • GOTO 2023 - YouTube](https://www.youtube.com/watch?v=fR12tWwTLHo) 14. [Quarkus Unveiled: Efficiency & Green Impact • Holly Cummins & Charles Humble • GOTO 2023 - YouTube](https://www.youtube.com/watch?v=7bvQXb3Nv0E) +15. [One Rule to Rule Them All • Pragmatic Dave Thomas • GOTO 2023](https://youtu.be/ug8XX2MpzEw?si=f_QqQSSIKoqNBYda) +16. [The One Where We Threat Model During Development • Izar Tarandach • GOTO...](https://youtu.be/_Pam1hkkzyM?si=u6MvN7WcyOJseiEV) +17. [Advanced Music Production with FL Studio – Tutorial](https://youtu.be/I_ShMaNw0Rc?si=QI4YhgrivzFnEExr) +18. [How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integra...](https://youtu.be/x-FPdvCCbL8?si=wKHl5wyv_YKOT9BC) +19. [F2023 #22 - Distributed Transaction Processing Databases (CMU Intro to D...](https://youtu.be/QRJs_57Pung?si=UvLjXPQUFdmkjP7w) +20. [Deep Q-Networks Explained!](https://youtu.be/x83WmvbRa2I?si=2JQkkSpExE26txKR) +21. [F2023 #22 - Distributed Transaction Processing Databases (CMU Intro to D...](https://youtu.be/QRJs_57Pung?si=QK9LXnr_AIG5ajY8) +22. [Chroma Vector Database: Retrieval for LLMs (Hammad Bashir + Liquan Pei)](https://youtu.be/E4ot5d79jdA?si=6apWFfCdKbi7_yFK) +23. [Pijul: Version-Control Post-Git in 4 Minutes • Pierre-Étienne Meunier • ...](https://youtu.be/dgXnVx94c9E?si=Bc1DnpV_RS7I3GQP) +24. [How Git Works: Explained in 4 Minutes](https://youtu.be/e9lnsKot_SQ?si=5hKhbdIy2r4Apope) +25. [CodeSandbox Frontend System Design | Mock Interview | 💪Chakde System Des...](https://youtu.be/o5aoJlcS8Rc?si=LzU5hRiYbISxdElA) +26. [Ace the Senior Frontend Interview & beyond at Atlassian | Chakde Intervi...](https://youtu.be/pBDpZtuGKxI?si=1KE6BekUOJMaz7cM) +27. [What Is A Backend For A Frontend (BFF) Architecture Pattern](https://youtu.be/SSo-z16wEnc?si=Mhvmiw2V-E11gm-4) +28. [Mastering how web works ? | Frontend For Everyone](https://youtu.be/s7fa2s8Ry4M?si=57a7i--t3uqTu0Ou) +29. [Resource Lists for Senior Engineers: WhitePapers, Blogs and Videos](https://www.youtube.com/live/dbc6kC8lWvk?si=hUwanVhbr4SvwHp4) ## Podcasts diff --git a/docs/computer-science/system-design/architecture-guide.md b/docs/computer-science/system-design/architecture-guide.md index 10795907adb..321a766a797 100755 --- a/docs/computer-science/system-design/architecture-guide.md +++ b/docs/computer-science/system-design/architecture-guide.md @@ -26,53 +26,61 @@ Web-queue-worker is suitable for relatively simple domains with some resource-in | Big data | Divide a huge dataset into small chunks. Parallel processing on local datasets. | Batch and real-time data analysis. Predictive analysis using ML. | | Big compute | Data allocation to thousands of cores. | Compute intensive domains such as simulation. | -- Design Principles - - Design for self-healing - - Make all things redundant - - Minimize coordination - - Design to scale out - - Partition around limits - - Design for operations - - Use managed services - - Use the best data store for the job - - Design for evolution - - Build for the needs of business -- Best Practices - - API Design - - API Implementation - - Autoscaling - - Background jobs - - Caching - - Content Delivery Network - - Data Partitioning - - Data Partitioning strategies (by service) - - Monitoring and diagnostics - - Naming Conventions - - Retry Guidance for Specific services - - Transient fault handling -- Performance Tuning - - Scenario 1 - Distributed Transactions - - Scenario 2 - Multiple backend services - - Scenario 3 - Event Streaming -- Performance Antipatterns - - Busy Database - - Busy Front End - - Chatty I/O - - Extraneous Fetching - - Improper Instantiation - - Monolithic Persistence - - No Caching - - Synchronous I/O2. Design Patterns - - [Ambassador](https://docs.microsoft.com/en-us/azure/architecture/patterns/ambassador) - - [Anti-corruption Layer](https://docs.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer) - - [Availability](https://docs.microsoft.com/en-us/azure/architecture/patterns/category/availability) - - [Backends for Frontends](https://docs.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends) - - [Bulkhead](https://docs.microsoft.com/en-us/azure/architecture/patterns/bulkhead) +### - Design Principles + +- Design for self-healing +- Make all things redundant +- Minimize coordination +- Design to scale out +- Partition around limits +- Design for operations +- Use managed services +- Use the best data store for the job +- Design for evolution +- Build for the needs of business + +### - Best Practices + +- API Design +- API Implementation +- Autoscaling +- Background jobs +- Caching +- Content Delivery Network +- Data Partitioning +- Data Partitioning strategies (by service) +- Monitoring and diagnostics +- Naming Conventions +- Retry Guidance for Specific services +- Transient fault handling + +### - Performance Tuning + +- Scenario 1 - Distributed Transactions +- Scenario 2 - Multiple backend services +- Scenario 3 - Event Streaming + +### - Performance Antipatterns + +- Busy Database +- Busy Front End +- Chatty I/O +- Extraneous Fetching +- Improper Instantiation +- Monolithic Persistence +- No Caching +- Synchronous I/O2. Design Patterns +- [Ambassador](https://docs.microsoft.com/en-us/azure/architecture/patterns/ambassador) +- [Anti-corruption Layer](https://docs.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer) +- [Availability](https://docs.microsoft.com/en-us/azure/architecture/patterns/category/availability) +- [Backends for Frontends](https://docs.microsoft.com/en-us/azure/architecture/patterns/backends-for-frontends) +- [Bulkhead](https://docs.microsoft.com/en-us/azure/architecture/patterns/bulkhead) The Bulkhead pattern is a type of application design that is tolerant of failure. In a bulkhead architecture, elements of an application are isolated into pools so that if one fails, the others will continue to function. It's named after the sectioned partitions (bulkheads) of a ship's hull. If the hull of a ship is compromised, only the damaged section fills with water, which prevents the ship from sinking. - - [Cache-Aside](https://docs.microsoft.com/en-us/azure/architecture/patterns/cache-aside) + +- [Cache-Aside](https://docs.microsoft.com/en-us/azure/architecture/patterns/cache-aside) - [Choreography](https://docs.microsoft.com/en-us/azure/architecture/patterns/choreography) - [Circuit Breaker](https://docs.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker) - [Claim Check](https://docs.microsoft.com/en-us/azure/architecture/patterns/claim-check) @@ -136,12 +144,8 @@ The Bulkhead pattern is a type of application design that is tolerant of failure ## Cloud Adoption Framework - - - - - - - - -[Cloud Architecture Guidance and Topologies  |  Cloud Architecture Center | Google Cloud](https://cloud.google.com/architecture) +- +- +- +- +- [Cloud Architecture Guidance and Topologies  |  Cloud Architecture Center | Google Cloud](https://cloud.google.com/architecture) diff --git a/docs/computer-science/system-design/cloud-native.md b/docs/computer-science/system-design/cloud-native.md index 82d32312fab..09446a9275e 100755 --- a/docs/computer-science/system-design/cloud-native.md +++ b/docs/computer-science/system-design/cloud-native.md @@ -6,7 +6,7 @@ Where operations teams would manage the infrastructure resource allocations to t Cloud-native platforms, like Kubernetes, expose a flat network that is overlaid on existing networking topologies and primitives of cloud providers. Similarly, the native storage layer is often abstracted to expose logical volumes that are integrated with containers. Operators can allocate storage quotas and network policies that are accessed by developers and resource administrators. The infrastructure abstraction not only addresses the need for portability across cloud environments, but also lets developers take advantage of emerging patterns to build and deploy applications. Orchestration managers become the deployment target, irrespective of the underlying infrastructure that may be based on physical servers or virtual machines, private clouds or public clouds. -Kubernetes is an ideal platform for running contemporary workloads designed as cloud-native applications. It's become the de facto operating system for the cloud, in meuch the same way Linux is the operating system for the underlying machines. +Kubernetes is an ideal platform for running contemporary workloads designed as cloud-native applications. It's become the de facto operating system for the cloud, in much the same way Linux is the operating system for the underlying machines. ## 10 Key Attributes of Cloud-Native Applications diff --git a/docs/computer-science/system-design/event-driven-architecture.md b/docs/computer-science/system-design/event-driven-architecture.md index 557d1af619b..58a68e57307 100755 --- a/docs/computer-science/system-design/event-driven-architecture.md +++ b/docs/computer-science/system-design/event-driven-architecture.md @@ -54,6 +54,7 @@ Replaying all the events from the logs can give any state of the system at any t 2. CQRS - Command and Query Responsibility Segregation CQRS is a fancy name for an architecture that uses different data models to represent read and write operations. + At its heart is the notion that you can use a different model to update information than the model you use to read information. ![image](../../media/Event-driven-architecture-image1.jpg) @@ -85,8 +86,8 @@ Event-driven architecture (EDA) means constructing your system as a series of co Not withstanding their differences, we can observe that there is a close relationship between a stream and a table. We call this the [stream-table duality](https://www.confluent.io/blog/streams-tables-two-sides-same-coin/). What this means is: -- We can turn a stream into a tableby aggregating the stream with operations such asCOUNT()orSUM(), for example. In our chess analogy, we could reconstruct the board's latest state (table) by replaying all recorded moves (stream). -- We can turn a table into a streamby capturing the changes made to the table - inserts, updates, and deletes - into a "change stream." This process is often called [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) or CDC for short. In the chess analogy, we could achieve this by observing the last played move and recording it (into the stream) or, alternatively, by comparing the board's state (table) before and after the last move and then recording the difference of what changed (into the stream), though this is likely slower than the first option. +- We can turn a stream into a table by aggregating the stream with operations such asCOUNT()orSUM(), for example. In our chess analogy, we could reconstruct the board's latest state (table) by replaying all recorded moves (stream). +- We can turn a table into a stream by capturing the changes made to the table - inserts, updates, and deletes - into a "change stream." This process is often called [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) or CDC for short. In the chess analogy, we could achieve this by observing the last played move and recording it (into the stream) or, alternatively, by comparing the board's state (table) before and after the last move and then recording the difference of what changed (into the stream), though this is likely slower than the first option. In fact, a table is fully defined by its underlying change stream. If you have ever worked with a relational database such as Oracle or MySQL, these change streams exist there, too! Here, however, they are a hidden implementation detail - albeit an absolutely critical one - and have names like [redo log](https://docs.oracle.com/cd/B28359_01/server.111/b28310/onlineredo001.htm#ADMIN11302) or [binary log](https://dev.mysql.com/doc/internals/en/binary-log-overview.html). In event streaming, the redo log is much more than an implementation detail. It's a first-class entity: a stream. We can turn streams into tables and tables into streams, which is one reason why we say that event streaming and Kafka are [turning the database inside out](https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/). @@ -102,31 +103,31 @@ In fact, a table is fully defined by its underlying change stream. If you have e Consider these patterns when implementing asynchronous messaging: -- [Competing Consumers Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/competing-consumers) +### - [Competing Consumers Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/competing-consumers) Multiple consumers may need to compete to read messages from a queue. This pattern explains how to process multiple messages concurrently to optimize throughput, to improve scalability and availability, and to balance the workload. -- [Priority Queue Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/priority-queue) +### - [Priority Queue Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/priority-queue) For cases where the business logic requires that some messages are processed before others, this pattern describes how messages posted by a producer that have a higher priority can be received and processed more quickly by a consumer than messages of a lower priority. -- [Queue-based Load Leveling Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling) +### - [Queue-based Load Leveling Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling) This pattern uses a message broker to act as a buffer between a producer and a consumer to help to minimize the impact on availability and responsiveness of intermittent heavy loads for both those entities. -- [Retry Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/retry) +### - [Retry Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/retry) A producer or consumer might be unable connect to a queue, but the reasons for this failure may be temporary and quickly pass. This pattern describes how to handle this situation to add resiliency to an application. -- [Scheduler Agent Supervisor Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/scheduler-agent-supervisor) +### - [Scheduler Agent Supervisor Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/scheduler-agent-supervisor) Messaging is often used as part of a workflow implementation. This pattern demonstrates how messaging can coordinate a set of actions across a distributed set of services and other remote resources, and enable a system to recover and retry actions that fail. -- [Choreography pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/choreography) +### - [Choreography pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/choreography) This pattern shows how services can use messaging to control the workflow of a business transaction. -- [Claim-Check Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/claim-check) +### - [Claim-Check Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/claim-check) This pattern shows how to split a large message into a claim check and a payload. @@ -138,9 +139,9 @@ This pattern shows how to split a large message into a claim check and a payload ![image](../../media/Event-driven-architecture-image6.jpg) -## Orchestration - command-driven architecture +Orchestration - command-driven architecture -## Choreography - event-driven communication +Choreography - event-driven communication ![image](../../media/Event-driven-architecture-image7.jpg) @@ -167,6 +168,7 @@ This pattern shows how to split a large message into a claim check and a payload [GOTO 2017 - The Many Meanings of Event-Driven Architecture - Martin Fowler](https://www.youtube.com/watch?v=STKCRSUsyP0) + LMAX Architecture - @@ -180,7 +182,7 @@ LMAX Architecture Task queues manage background work that must be executed outside the usual HTTP request-response cycle. -## Why are task queues necessary? +### Why are task queues necessary? Tasks are handled asynchronously either because they are not initiated by an HTTP request or because they are long-running jobs that would dramatically reduce the performance of an HTTP response. diff --git a/docs/computer-science/system-design/intro.md b/docs/computer-science/system-design/intro.md index fab00cd2942..4890215f037 100755 --- a/docs/computer-science/system-design/intro.md +++ b/docs/computer-science/system-design/intro.md @@ -20,9 +20,11 @@ Systems design is the process of defining the [architecture](https://en.wikipedi 1. Features 2. Define APIs 3. Availability (CAP Theorem) - - Consistency - - Availability, and - - Partition Tolerance + +- Consistency +- Availability, and +- Partition Tolerance + 4. Latency Performance (If customer facing application, then latency matters) 5. Scalability (Add more users and requests) 6. Durability (Data is not lost or compromised) @@ -30,46 +32,55 @@ Systems design is the process of defining the [architecture](https://en.wikipedi 8. Security & Privacy 9. Cost Effective -## How to approach a system design problem - -- Gather requirements (Functional and non-functional requirements) - - Functional - - sendMessage(messageBody) - - receiveMessage() - - Non-functional - - Scalable (handles load increases, more queues and messages) - - Highly available (services hardware/network failures) - - Highly performent (single digit latency for main operations) - - Durable (once submitted, data is not lost) +## Approaching a system design problem + +### Gather requirements (Functional and non-functional requirements) + +- Functional + - sendMessage(messageBody) + - receiveMessage() +- Non-functional + - Scalable (handles load increases, more queues and messages) + - Highly available (services hardware/network failures) + - Highly performent (single digit latency for main operations) + - Durable (once submitted, data is not lost) - What is the expected read-to-write ratio? - - How many **concurrent requests** should we expect? - - What's the average expected response time? - - What's the limit of the data we allow users to provide? - - How many customers - - RPS / Traffic - - Who are the users -- Establish the Scope - - Do we want to discuss the end-to-end experience or just the API? - - What clients do we want to support (mobile, web, etc)? - - Do we require authentication? Analytics? Integrating with existing systems? -- High level design - - Cloud - CDN / LoadBalancers / Servers / Queues - - Database layer - - Scalability -- Low level design - - DS + Algo - - API - - Data model - - Database schema - - Optimizations - - Edge cases - - Viral Videos - - High number of followers -- Others - - Security - - Reliability - - Replication - - Tradeoffs +- How many **concurrent requests** should we expect? +- What's the average expected response time? +- What's the limit of the data we allow users to provide? +- How many customers +- RPS / Traffic +- Who are the users + +### Establish the Scope + +- Do we want to discuss the end-to-end experience or just the API? +- What clients do we want to support (mobile, web, etc)? +- Do we require authentication? Analytics? Integrating with existing systems? + +### High level design + +- Cloud - CDN / LoadBalancers / Servers / Queues +- Database layer +- Scalability + +### Low level design + +- DS + Algo +- API +- Data model +- Database schema +- Optimizations +- Edge cases + - Viral Videos + - High number of followers + +### Others + +- Security +- Reliability +- Replication +- Tradeoffs @@ -90,7 +101,7 @@ Vertical Scaling is expensive and have a limit whereas Horizontal Scaling is che Consider latency between cross racks, cross hosts -### CPU/ Memory/ HardDrive / Nework Bandwidth +### CPU / Memory / HardDrive / Network Bandwidth All of the above are limited resources. How to improve limitations while scaling @@ -111,8 +122,8 @@ Containers - is a way of running your applications and its dependencies in an is ### Publisher - Subscriber or Queue - - Queuing is a *point-to-point communication model* - a pool of consumers may read from a server and each message is delivered to one of them - it allows you to divide up the processing of data over multiple consumer instances and scale your processing. - - Publish-subscribe is a *broadcast communication model* - a message is broadcast to all the consumers. +- Queuing is a *point-to-point communication model* - a pool of consumers may read from a server and each message is delivered to one of them - it allows you to divide up the processing of data over multiple consumer instances and scale your processing. +- Publish-subscribe is a *broadcast communication model* - a message is broadcast to all the consumers. ### Thin clients, Thick clients, Dumb clients, Smart client @@ -172,4 +183,14 @@ This helps as you no longer have to make assumptions about [System Design Blueprint: The Ultimate Guide | by Love Sharma | Apr, 2023 | Dev Genius](https://blog.devgenius.io/system-design-blueprint-the-ultimate-guide-e27b914bf8f1) ![system-design-checklist](../../media/Screenshot%202023-05-26%20at%201.21.56%20PM.png) -[System Designs - YouTube](https://www.youtube.com/playlist?list=PLQnljOFTspQXSevtRqvMNycWfHM7cXc3d) +- [System Designs - YouTube](https://www.youtube.com/playlist?list=PLQnljOFTspQXSevtRqvMNycWfHM7cXc3d) +- [System Design Master Template: How to Answer Any System Design Interview Question. | by Arslan Ahmad | Level Up Coding](https://levelup.gitconnected.com/system-design-master-template-how-to-answer-any-system-design-interview-question-ee5dc332acd5) +- [Mastering the Art of Caching for System Design Interviews: A Complete Guide | by Arslan Ahmad | Level Up Coding](https://levelup.gitconnected.com/master-the-art-of-caching-for-system-design-interviews-a-complete-guide-676bb49d194) +- [How to Use Consistent Hashing in a System Design Interview? | by Arslan Ahmad | InterviewNoodle](https://interviewnoodle.com/how-to-use-consistent-hashing-in-a-system-design-interview-b738be3a1ae3) +- [System Design Interview Basics: CAP vs. PACELC | by Arslan Ahmad | InterviewNoodle](https://interviewnoodle.com/system-design-interview-basics-cap-vs-pacelc-cf7c5eebc313) +- [System Design Interviews: A Step-By-Step Guide | by Arslan Ahmad | InterviewNoodle](https://interviewnoodle.com/system-design-interviews-a-step-by-step-guide-cb30496f31a5) +- [System Design Interview Question: Designing a URL Shortening Service | by Arslan Ahmad | InterviewNoodle](https://interviewnoodle.com/system-design-interview-question-designing-a-url-shortening-service-eac7b147295) +- [System Design Interviews: What distinguishes you from others? | by Arslan Ahmad | InterviewNoodle](https://interviewnoodle.com/system-design-interviews-what-distinguishes-you-from-others-7095405ec48) +- [The Complete Guide to Ace the System Design Interview in 2023 | by Arslan Ahmad | CodeX | Medium](https://medium.com/codex/the-complete-guide-to-ace-the-system-design-interview-in-2022-2d21c9534905) +- [Top 7 Facebook System Design Interview Questions and Answers (2023) | by Arslan Ahmad | CodeX | Medium](https://medium.com/codex/top-7-facebook-system-design-interview-questions-and-answers-2022-8c2dd329038) +- [System Design Tutorial: 3 Must-Know Distributed Systems Concepts | by Arslan Ahmad | Geek Culture | Medium](https://medium.com/geekculture/system-design-tutorial-3-must-know-distributed-systems-concepts-279d4e9718e8) diff --git a/docs/computer-science/system-design/trade-offs.md b/docs/computer-science/system-design/trade-offs.md index 2b92d715894..138055a2f41 100755 --- a/docs/computer-science/system-design/trade-offs.md +++ b/docs/computer-science/system-design/trade-offs.md @@ -11,14 +11,14 @@ Compute is cheap. Storage is cheap. Engineering time is expensive. ## Questions -Cost +### Cost - Operational complexity - How easy is it to scale if the TSDB storage is low? - How easy is it to scale the performance with increased data? - Are there any operations tasks that regularly need to be carried out? -Capabilities +### Capabilities - Does the TSDB support metrics and events? - Can metadata be associated with an event? @@ -26,49 +26,49 @@ Capabilities - What is the consistency model? - Can the data have a Time To Live? -Performance +### Performance - How many events per second can be written in, for a given scale of the system? - How many events per second can be read out, for a given scale of the system? - Bytes per point after any compression that occurs? (What volume of space is required?) -Query capabilities +### Query capabilities - How is the data queried? DSL? API? - How easy is the query mechanism to use? - Can the query mechanism aggregate data? Can it do it by date? - Can we de-dupe data? e.g., if we ingested the same event twice? A uniqueness constraint. -Ingestion +### Ingestion - How is the data sent to the TSDB? API? Log scraping? Multi language clients? -Export +### Export - How could the entire TSDB be exported for use elsewhere or in another TSDB? -Maturity +### Maturity - How mature is the TSDB and its ecosystem? Is it likely to die soon? Does it have good support? -Community +### Community - How large and active is the community using the TSDB? If I have a problem will I be able to find someone to help me solve it? -Support +### Support - Is there any paid support for the TSDB and if so how much? -Security +### Security - What are the mechanisms for authentication and authorisation? - What other security implications are there? -Integration +### Integration - How easy is it to integrate with other services? Is there anything specific that helps? -Visualisation +### Visualisation - How can the data from the TSDB be visualised? - Are there any dashboards / high level visualisations? diff --git a/docs/decentralized-applications/ethereum/ethereum-staking.md b/docs/decentralized-applications/ethereum/ethereum-staking.md index a75cca395b0..3379b448cf9 100755 --- a/docs/decentralized-applications/ethereum/ethereum-staking.md +++ b/docs/decentralized-applications/ethereum/ethereum-staking.md @@ -127,3 +127,4 @@ wstETH is a wrapped version of stETH. Due to the nature of Lido, the amount of s - [How Do Ethereum Withdrawals Work? All You Need To Know - YouTube](https://www.youtube.com/watch?v=RwwU3P9n3uo) - [How To Stake ETH And Earn Staking Rewards: Step By Step](https://milkroad.com/invest/guide-to-staking/) - [Ethereum staking deposit contract address | ethereum.org](https://ethereum.org/en/staking/deposit-contract/) +- [Validating Ethereum | Ep 0 | Lido Community Staking Series - YouTube](https://www.youtube.com/watch?v=U-uxZRmB6_0) diff --git a/docs/knowledge/law/laws.md b/docs/knowledge/law/laws.md index 9538895073e..8e5c5c32b2c 100755 --- a/docs/knowledge/law/laws.md +++ b/docs/knowledge/law/laws.md @@ -28,6 +28,8 @@ Layering - Tumblr / Mixers - [6 BEST Bitcoin Mixers & Tumblers (2023 List)](http The **Digital Millennium Copyright Act** (**DMCA**) is a 1998 United States [copyright](https://en.wikipedia.org/wiki/Copyright "Copyright") [law](https://en.wikipedia.org/wiki/Law "Law") that implements two 1996 treaties of the [World Intellectual Property Organization](https://en.wikipedia.org/wiki/World_Intellectual_Property_Organization "World Intellectual Property Organization") (WIPO). It criminalizes production and dissemination of technology, devices, or services intended to circumvent measures that control access to [copyrighted](https://en.wikipedia.org/wiki/Copyright "Copyright") works (commonly known as [digital rights management](https://en.wikipedia.org/wiki/Digital_rights_management "Digital rights management") or DRM). It also criminalizes the act of circumventing an [access control](https://en.wikipedia.org/wiki/Access_control "Access control"), whether or not there is actual [infringement of copyright](https://en.wikipedia.org/wiki/Copyright_infringement "Copyright infringement") itself. In addition, the DMCA heightens the penalties for copyright infringement on the [Internet](https://en.wikipedia.org/wiki/Internet "Internet"). Passed on October 12, 1998, by a unanimous vote in the [United States Senate](https://en.wikipedia.org/wiki/United_States_Senate "United States Senate") and signed into law by President [Bill Clinton](https://en.wikipedia.org/wiki/Bill_Clinton "Bill Clinton") on October 28, 1998, the DMCA amended [Title 17 of the United States Code](https://en.wikipedia.org/wiki/Title_17_of_the_United_States_Code "Title 17 of the United States Code") to extend the reach of copyright, while limiting the [liability](https://en.wikipedia.org/wiki/Legal_liability "Legal liability") of the [providers of online services](https://en.wikipedia.org/wiki/Online_service_provider "Online service provider") for copyright infringement by their users. +[DMCA.com - Protect Your Online Content and Brand with DMCA Takedown Services, Compliance Solutions, and Content Protection](https://www.dmca.com/) + [What is DMCA?](https://www.dmca.com/FAQ/What-is-DMCA) [Digital Millennium Copyright Act - Wikipedia](https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_Act) diff --git a/docs/knowledge/links.md b/docs/knowledge/links.md index c907d57c0a2..ad1aff83239 100644 --- a/docs/knowledge/links.md +++ b/docs/knowledge/links.md @@ -1136,5 +1136,33 @@ 514. [Is the S&P 500 Just a Giant Bubble? - YouTube](https://www.youtube.com/watch?v=VsY8rwX8MTc) 515. [What's the deal with the food in Japan? - YouTube](https://www.youtube.com/watch?v=V2lcQfQHrzU) 516. [Why This Ultra Dense Battery Breakthrough Matters - YouTube](https://www.youtube.com/watch?v=YJ4pg_exdvs) +517. [Best Mutual Funds 2024 - Top Multi Cap and Flexi Cap Mutual Funds](https://youtu.be/80x65RRX614?si=s0ZAvRbY7lzy7N2q) +518. [Unlocking the Web: Exploring WebAuthn & Beyond • Eli Holderness & Mark R...](https://youtu.be/Gs1exPFXnQ8?si=whuR2i6zyZUVwPxk) +519. [Supercharging a Net Zero Home - Ultimate Smart Home Tour](https://youtu.be/0QH7-v2I7kU?si=i-YjswK7GFBPJJN_) +520. [Is Eating Ass Dangerous?](https://youtu.be/nFhCAedFeCc?si=BFB2i1SUjK-nTd3L) +521. [How Modern Treasure Hunting Works](https://youtu.be/8cyuOcb5O_M?si=5owCFbbQwcm8H52i) +522. [How Did OpenAI Go So Badly Wrong?](https://youtu.be/csfAd_kDh5A?si=VRzJ0ripnccehwwo) +523. [Can a Billion Oysters Save New York City?](https://youtu.be/O_mk-YJxwLw?si=J9odW0kf4j4P6E0D) +524. [Best Tax Saver/ELSS Mutual Funds in India | Best Mutual Fund 2024](https://youtu.be/RSxeWf-NFEQ?si=fYE-mjHhHchqBFrM) +525. [Is India Winning or Losing the Electric War?: Business Case Study](https://youtu.be/0QWC_N6Hi5s?si=-eNjwwTHiA47pOWp) +526. [Something Terrible Is Happening in Italy](https://youtu.be/mcsAO9AYc0c?si=BffdvFKOB3v9StKc) +527. [What are these strange dice? (Part 2) - Numberphile](https://youtu.be/8rYLBXd_kzI?si=I-jBfStQXnGfy4iK) +528. [Underwater Constructions | How do Engineers Make Them?](https://youtu.be/t5cJB6JL0_k?si=6rWSAroa-2eFWTp2) +529. [Why Ethiopia is Preparing to Invade Eritrea Next](https://youtu.be/J-hABbIseGk?si=qc2U8ynwBReo7jNT) +530. [Why is United Nations Failing? | Dark Truth about Weapons Industry | Dhr...](https://youtu.be/_J5UOvt1aLM?si=2h760t04IkJDrnbF) +531. [Why & How Capitalism Needs to Be Reformed](https://youtu.be/MNZ1TtN_KGw?si=NmXMIFff67rhJCKc) +532. [The U.S. Economy Just Hit a Major “Inflection Point” (Ray Dalio Interview)](https://youtu.be/bj2s-Y5MMag?si=BMGPEXoAeAKBNnpm) +533. [Functional Programming in Spring with Arrow, Lenses & Effects • Jacob Ba...](https://youtu.be/7XzVScKvs5Y?si=NtJhg5pD6C8OkW_k) +534. [Scaling Your Architecture With Services & Events • Randy Shoup • YOW! 2020](https://youtu.be/Idkz_rOS-ug?si=Vir8ECbMH_MO_Tj3) +535. [Finology Best Mutual Funds in India 2024 | Large Cap Mutual Fund](https://youtu.be/O23vSwiF5Ic?si=hH5hoXOAIJBkLI1J) +536. [The Insane Mathematics of Nuclear Deterrence](https://youtu.be/lPD-WlP06tM?si=nlzI2FsjZ44mil6X) +537. [Octopus vs Underwater Maze](https://youtu.be/7__r4FVj-EI?si=VJd1KNEQIAT_Hny1) +538. [How to BEAT THE INDEX by 10% Annually over 45 Years | Trending Value Str...](https://youtu.be/RjjASYLd_Os?si=Rj_xCYCE2PmoVLu-) +539. [Solar cars: Why aren’t they everywhere?](https://youtu.be/FXDWRhEOX7k?si=FAZgceLd8dmA4jZ3) +540. [Warren Buffett's 'Insider Trading' Explained (+ BIG ANNOUNCEMENT)](https://youtu.be/srPPnkXsVwQ?si=fv_r5QE-mbSAT3E3) +541. [Throw Your Computer In The Ocean.](https://youtu.be/2bHmGvro_zQ?si=9Pjl6Fd9Z4mCKhUG) +542. [Picking Winning Assets Through Time | Psychology of Money](https://youtu.be/DyXf_xBczSM?si=71huDrpFQiHZjGqg) +543. [FUTURE OF INDIAN RAILWAYS | अगले 10 साल में क्या कुछ होने वाला हैं रेलवे...](https://youtu.be/y3R21iZzXEI?si=nJjkkziJCKWHaKOV) +544. [How megacities around the world are tackling their air pollution](https://youtu.be/3F6mq20QOcE?si=fynUgqurHZVyq_OJ) ## End diff --git a/docs/knowledge/quotes-proverbs/life.md b/docs/knowledge/quotes-proverbs/life.md index ee13accd207..5ac250d6332 100755 --- a/docs/knowledge/quotes-proverbs/life.md +++ b/docs/knowledge/quotes-proverbs/life.md @@ -1047,3 +1047,7 @@ Humans have opened gates to the hell God has given us 2 ears and 1 mouth, so you must use those in proportion i.e. 2/3rd listen 1/3 talk If you kill a killer the number of killers in the world remains the same. + +I have never said most of the things I said -- Yogi berra + +## End diff --git a/docs/management/jobs/questions-to-company.md b/docs/management/jobs/questions-to-company.md index cd7d0521de4..e14924152d2 100755 --- a/docs/management/jobs/questions-to-company.md +++ b/docs/management/jobs/questions-to-company.md @@ -44,10 +44,7 @@ Always don't say yes to any offer at that time, always sleep over it - What will be my roles and responsibilities as Solution Architect? -1. **Always find the person who you will be working under, because you will learn from that person only.** - - Why - Engineering productivity is key to building good software. It comes down to the manager to make sure the developer team is performing as expected. - +1. **Always find the person who you will be working under, because you will learn from that person only.** - Why - Engineering productivity is key to building good software. It comes down to the manager to make sure the developer team is performing as expected. 2. Find how can you grow in the company. (Both professionally and academically) 3. Teaching must be a part of the company. (Best way to learn is to teach) 4. More higher positions (Experience needed or knowledge needed) @@ -95,9 +92,7 @@ Always don't say yes to any offer at that time, always sleep over it - Decision making process - How are raises calculated and awarded? - Functional and Non functional requirements? -- **Career Path / Engineering Competency Matrix** - - +- **Career Path / Engineering Competency Matrix** - 1. What do the day-to-day responsibilities of the role look like? 2. What are the company's values? What characteristics do you look for in employees in order to represent those values? @@ -115,8 +110,6 @@ Always don't say yes to any offer at that time, always sleep over it - How you do peer reviews? (core reviews, branching strategies, methodologies they follow?) - What programs do you have for continuous education. - What's the software development process that you use? - - How is project management handled? - - **Project Management and Task Management Tools** - How many projects could I expect to work on in a month? - What's the process for managing the code? - How long are the typical sprints? @@ -126,6 +119,8 @@ Always don't say yes to any offer at that time, always sleep over it - What's the typical length of a project? - How are vacation days handled? - Are there any emergency processes in place? / Disaster Recovery +- How is project management handled? + - **Project Management and Task Management Tools** - Are there technical debts in the current architecture/code base? - How do you tackle technical debt? - What's your company culture like? @@ -157,18 +152,10 @@ Always don't say yes to any offer at that time, always sleep over it ## Startup Questions -- Background of founders before joining companyc -- Do you have product-market fit? - -If not, they don't have real money, and there's no guarantee they ever will. - +- Background of founders before joining company +- Do you have product-market fit? - If not, they don't have real money, and there's no guarantee they ever will. - What is your current growth rate? -- What is your runway? - -The longer the runway, the more financially stable the company is. - -"Cashrunway" refers to the length of time in which a company will remain solvent, assuming that they are unable to raise more money. - +- What is your runway? - The longer the runway, the more financially stable the company is. "Cashrunway" refers to the length of time in which a company will remain solvent, assuming that they are unable to raise more money. - What's the company's/department's strategy for the next 6-18 months - Keep the title & position of the interviewer in mind @@ -215,23 +202,19 @@ Your percentage of ownership means more than your number of shares Percentage of the company that the ESOP pool represents 2. What is the my **preference stack**? (What is the total preference) (Classes of preferred shares) (multiples 1.5x?) - - **dilution** - - What was our most recent valuation? - - What is our current yearly growth rate? - - How much would the company need to sell for before my equity has value? - - If your most recent valuation is close to or exceeds the needed sale price, your equity offer has value. If the needed sale price is much higher than the company's most recent valuation, though, you have something to consider: Based on its current growth rate, how many years would you need to stay before its value comes close to that needed sale price? Are you comfortable investing that much time? - -- The more owed in liquidation perference, the less your equity is likely worth -- it had investors and board members whose equity was [protected](https://www.sec.gov/Archives/edgar/data/1089196/000119312515079062/d520327ds1a.htm#toc520327_13)by highliquidation preference - a guarantee that they get paid first and at least a certain amount when the company sells. When startup investors make millions in a sale, but money runs dry before reaching employees, a bad preference stack is often the cause. + +- **dilution** +- What was our most recent valuation? +- What is our current yearly growth rate? +- How much would the company need to sell for before my equity has value? +- If your most recent valuation is close to or exceeds the needed sale price, your equity offer has value. If the needed sale price is much higher than the company's most recent valuation, though, you have something to consider: Based on its current growth rate, how many years would you need to stay before its value comes close to that needed sale price? Are you comfortable investing that much time? +- The more owed in liquidation preference, the less your equity is likely worth +- it had investors and board members whose equity was [protected](https://www.sec.gov/Archives/edgar/data/1089196/000119312515079062/d520327ds1a.htm#toc520327_13)by high liquidation preference - a guarantee that they get paid first and at least a certain amount when the company sells. When startup investors make millions in a sale, but money runs dry before reaching employees, a bad preference stack is often the cause. - Liquidation preference give investors the protection they need to make the high-risk investments that startups thrive on. - If there had been a 1x liquidation preference in place, the investor would be guaranteed to get $3 million back. - The more money a startup raises, the harder it gets to fetch a high enough acquisition price. -3. Premptive rights - -What are preemptive rights and who has them? - -Preemptive rights allow common shareholders to maintain their proportional ownership in a company by buying more shares in the event that the company issues another offering. These are sometimes issued to holders of common stocks. Holders of preferred shares typically also have other protections against dilution. +3. Preemptive rights - What are preemptive rights and who has them? - Preemptive rights allow common shareholders to maintain their proportional ownership in a company by buying more shares in the event that the company issues another offering. These are sometimes issued to holders of common stocks. Holders of preferred shares typically also have other protections against dilution. 4. While the first company offers you 1000 options out of 1,00,000, the second offers 500 out of 10,000. You have more skin in the game in the second company but you have to ask the founder why the ESOP pool is so small? Is he or she averse to sharing equity with early team members? When will the ESOP pool be expanded? These questions will help you better understand the founder's vision. diff --git a/docs/management/jobs/readme.md b/docs/management/jobs/readme.md index 05b8d377767..4d88ac50af7 100644 --- a/docs/management/jobs/readme.md +++ b/docs/management/jobs/readme.md @@ -9,6 +9,21 @@ Where you apply to a lot of jobs +### Tests + Outsourcing + +- Crossover test +- Cutshort test +- Turing test +- Andela +- Uplers + +### Vendors + +1. [HuntingCube Recruitment Solutions](https://www.linkedin.com/company/huntingcube/) +2. [Careernet](https://www.linkedin.com/company/wearecareernet/) +3. [Weekday (YC W21)](https://www.linkedin.com/company/weekdayworks/) +4. [Zyoin Group](https://www.linkedin.com/company/zyoin/) + ## Platform - Hirist diff --git a/docs/media/Communication exception flow.png b/docs/media/Communication exception flow.png new file mode 100644 index 00000000000..543e23b73e1 Binary files /dev/null and b/docs/media/Communication exception flow.png differ diff --git a/docs/media/MQTT Alerting Engine.drawio.png b/docs/media/MQTT Alerting Engine.drawio.png new file mode 100644 index 00000000000..deaf4f28d41 Binary files /dev/null and b/docs/media/MQTT Alerting Engine.drawio.png differ diff --git a/docs/media/Pasted image 20231201172554.png b/docs/media/Pasted image 20231201172554.png new file mode 100644 index 00000000000..651e72c42d9 Binary files /dev/null and b/docs/media/Pasted image 20231201172554.png differ diff --git a/docs/media/Pasted image 20231201173646.png b/docs/media/Pasted image 20231201173646.png new file mode 100644 index 00000000000..ae29779be2d Binary files /dev/null and b/docs/media/Pasted image 20231201173646.png differ diff --git a/docs/media/Pasted image 20231201173733.png b/docs/media/Pasted image 20231201173733.png new file mode 100644 index 00000000000..dcedecc24e1 Binary files /dev/null and b/docs/media/Pasted image 20231201173733.png differ diff --git a/docs/media/Pasted image 20231201173742.png b/docs/media/Pasted image 20231201173742.png new file mode 100644 index 00000000000..c70bc094291 Binary files /dev/null and b/docs/media/Pasted image 20231201173742.png differ diff --git a/docs/media/Pasted image 20231201173759.png b/docs/media/Pasted image 20231201173759.png new file mode 100644 index 00000000000..a44370d115e Binary files /dev/null and b/docs/media/Pasted image 20231201173759.png differ diff --git a/docs/media/Pasted image 20231201173807.png b/docs/media/Pasted image 20231201173807.png new file mode 100644 index 00000000000..01308bf51b2 Binary files /dev/null and b/docs/media/Pasted image 20231201173807.png differ diff --git a/docs/media/Pasted image 20231201173816.png b/docs/media/Pasted image 20231201173816.png new file mode 100644 index 00000000000..23dd81a9e3a Binary files /dev/null and b/docs/media/Pasted image 20231201173816.png differ diff --git a/docs/media/Pasted image 20231201173823.png b/docs/media/Pasted image 20231201173823.png new file mode 100644 index 00000000000..314d700a776 Binary files /dev/null and b/docs/media/Pasted image 20231201173823.png differ diff --git a/docs/media/Pasted image 20231201173831.png b/docs/media/Pasted image 20231201173831.png new file mode 100644 index 00000000000..a858efb05ad Binary files /dev/null and b/docs/media/Pasted image 20231201173831.png differ diff --git a/docs/media/Pasted image 20231201173837.png b/docs/media/Pasted image 20231201173837.png new file mode 100644 index 00000000000..496a5a7f9d4 Binary files /dev/null and b/docs/media/Pasted image 20231201173837.png differ diff --git a/docs/media/Pasted image 20231201175020.png b/docs/media/Pasted image 20231201175020.png new file mode 100644 index 00000000000..d9b4d79f439 Binary files /dev/null and b/docs/media/Pasted image 20231201175020.png differ diff --git a/docs/media/Pasted image 20231201175035.png b/docs/media/Pasted image 20231201175035.png new file mode 100644 index 00000000000..fcc45369e70 Binary files /dev/null and b/docs/media/Pasted image 20231201175035.png differ diff --git a/docs/media/Pasted image 20231201175111.png b/docs/media/Pasted image 20231201175111.png new file mode 100644 index 00000000000..8b668142eda Binary files /dev/null and b/docs/media/Pasted image 20231201175111.png differ diff --git a/docs/media/Pasted image 20231201175731.png b/docs/media/Pasted image 20231201175731.png new file mode 100644 index 00000000000..933166132f8 Binary files /dev/null and b/docs/media/Pasted image 20231201175731.png differ diff --git a/docs/media/Pasted image 20231201175751.png b/docs/media/Pasted image 20231201175751.png new file mode 100644 index 00000000000..d55dbdbaf93 Binary files /dev/null and b/docs/media/Pasted image 20231201175751.png differ diff --git a/docs/media/Pasted image 20231201175759.png b/docs/media/Pasted image 20231201175759.png new file mode 100644 index 00000000000..ef08f38030f Binary files /dev/null and b/docs/media/Pasted image 20231201175759.png differ diff --git a/docs/media/Pasted image 20231201180310.png b/docs/media/Pasted image 20231201180310.png new file mode 100644 index 00000000000..b8214e41f36 Binary files /dev/null and b/docs/media/Pasted image 20231201180310.png differ diff --git a/docs/media/Pasted image 20231201180349.png b/docs/media/Pasted image 20231201180349.png new file mode 100644 index 00000000000..9b8036d2e31 Binary files /dev/null and b/docs/media/Pasted image 20231201180349.png differ diff --git a/docs/media/Pasted image 20231201180442.png b/docs/media/Pasted image 20231201180442.png new file mode 100644 index 00000000000..a9f2fc3f533 Binary files /dev/null and b/docs/media/Pasted image 20231201180442.png differ diff --git a/docs/media/Pasted image 20231201181214.png b/docs/media/Pasted image 20231201181214.png new file mode 100644 index 00000000000..f7b77b17906 Binary files /dev/null and b/docs/media/Pasted image 20231201181214.png differ diff --git a/docs/media/Pasted image 20231201181252.png b/docs/media/Pasted image 20231201181252.png new file mode 100644 index 00000000000..c37fedfa102 Binary files /dev/null and b/docs/media/Pasted image 20231201181252.png differ diff --git a/docs/media/Pasted image 20231201181347.png b/docs/media/Pasted image 20231201181347.png new file mode 100644 index 00000000000..99570d94ed3 Binary files /dev/null and b/docs/media/Pasted image 20231201181347.png differ diff --git a/docs/media/Pasted image 20231201181414.png b/docs/media/Pasted image 20231201181414.png new file mode 100644 index 00000000000..67e80f39669 Binary files /dev/null and b/docs/media/Pasted image 20231201181414.png differ diff --git a/docs/media/Pasted image 20231201182421.png b/docs/media/Pasted image 20231201182421.png new file mode 100644 index 00000000000..534a2026503 Binary files /dev/null and b/docs/media/Pasted image 20231201182421.png differ diff --git a/docs/media/Pasted image 20231201182427.png b/docs/media/Pasted image 20231201182427.png new file mode 100644 index 00000000000..c055ffcbc27 Binary files /dev/null and b/docs/media/Pasted image 20231201182427.png differ diff --git a/docs/media/Pasted image 20231201182608.png b/docs/media/Pasted image 20231201182608.png new file mode 100644 index 00000000000..863636bbb38 Binary files /dev/null and b/docs/media/Pasted image 20231201182608.png differ diff --git a/docs/media/Pasted image 20231201183011.png b/docs/media/Pasted image 20231201183011.png new file mode 100644 index 00000000000..69d72b4a42a Binary files /dev/null and b/docs/media/Pasted image 20231201183011.png differ diff --git a/docs/media/Pasted image 20231201183115.png b/docs/media/Pasted image 20231201183115.png new file mode 100644 index 00000000000..705452d7e2f Binary files /dev/null and b/docs/media/Pasted image 20231201183115.png differ diff --git a/docs/media/Pasted image 20231201183143.png b/docs/media/Pasted image 20231201183143.png new file mode 100644 index 00000000000..28869911235 Binary files /dev/null and b/docs/media/Pasted image 20231201183143.png differ diff --git a/docs/media/Pasted image 20231201183958.png b/docs/media/Pasted image 20231201183958.png new file mode 100644 index 00000000000..50fd000159c Binary files /dev/null and b/docs/media/Pasted image 20231201183958.png differ diff --git a/docs/technologies/apache/airflow/intro.md b/docs/technologies/apache/airflow/intro.md index 083ac79193d..de52e05f33c 100755 --- a/docs/technologies/apache/airflow/intro.md +++ b/docs/technologies/apache/airflow/intro.md @@ -12,15 +12,15 @@ Airflow is a WMS (Workflow Management System) that defines tasks and and their d Workflows in Airflow are collections of tasks that have directional dependencies. Specifically, Airflow uses directed acyclic graphs - or DAG for short - to represent a workflow. Each node in the graph is a task, and edges define dependencies amongst tasks (The graph is enforced to be acyclic so that there are no circular dependencies that can cause infinite execution loops). -At a high level, a DAG can be thought of as a container that holds tasks and their dependencies, and sets the context for when and how those tasks should be executed. Each DAG has a set of properties, most important of which are itsdag_id, a unique identifier amongst all DAGs, itsstart_date, the point in time at which the DAG's tasks are to begin executing, and theschedule_interval, or how often the tasks are to be executed. In addition to thedag_id, start_date, andschedule_interval, each DAG can be initialized with a set ofdefault_arguments. These default arguments are inherited by all tasks in the DAG. +At a high level, a DAG can be thought of as a container that holds tasks and their dependencies, and sets the context for when and how those tasks should be executed. Each DAG has a set of properties, most important of which are its dag_id, a unique identifier amongst all DAGs, its start_date, the point in time at which the DAG's tasks are to begin executing, and the schedule_interval, or how often the tasks are to be executed. In addition to the dag_id, start_date, and schedule_interval, each DAG can be initialized with a set of default_arguments. These default arguments are inherited by all tasks in the DAG. ## Operators, Sensors, and Tasks -Although the DAG is used to organize tasks and set their execution context, DAGs do not perform any actual computation. Instead, **tasks** are the element of Airflow that actually "do the work" we want performed. Tasks can have two flavors: they can either execute some explicit operation, in which case they are an **Operator**, orthey can pause the execution of dependent tasks until some criterion has been met, in which case they are a **Sensor**.In principle, Operators can perform any function that can be executed in Python. Similarly, Sensors can check the state of any process or data structure. +Although the DAG is used to organize tasks and set their execution context, DAGs do not perform any actual computation. Instead, **tasks** are the element of Airflow that actually "do the work" we want performed. Tasks can have two flavors: they can either execute some explicit operation, in which case they are an **Operator**, or they can pause the execution of dependent tasks until some criterion has been met, in which case they are a **Sensor**. In principle, Operators can perform any function that can be executed in Python. Similarly, Sensors can check the state of any process or data structure. ## DagRuns and TaskInstances -Once we've defined a DAG - i.e. we've instantiated tasks and defined their dependencies - we can then execute the tasks based on the parameters of the DAG. A key concept in Airflow is that of anexecution_time. When the Airflow scheduler is running, it will define a regularly-spaced schedule of dates for which to execute a DAG's associated tasks. The execution times begin at the DAG'sstart_dateand repeat everyschedule_interval. For our example the scheduled execution times would be('2017--01--01 00:00:00', '2017--01--02 00:00:00', ...). For eachexecution_time, aDagRunis created and operates under the context of that execution time. Thus aDagRunis simply a DAG with some execution time +Once we've defined a DAG - i.e. we've instantiated tasks and defined their dependencies - we can then execute the tasks based on the parameters of the DAG. A key concept in Airflow is that of an execution_time. When the Airflow scheduler is running, it will define a regularly-spaced schedule of dates for which to execute a DAG's associated tasks. The execution times begin at the DAG's start_date and repeat every schedule_interval. For our example the scheduled execution times would be ('2017--01--01 00:00:00', '2017--01--02 00:00:00', ...). For each execution_time, a DagRun is created and operates under the context of that execution time. Thus a DagRun is simply a DAG with some execution time ## Principles diff --git a/docs/technologies/kafka/intro.md b/docs/technologies/kafka/intro.md index 5c1ce58deef..9b02ea64329 100755 --- a/docs/technologies/kafka/intro.md +++ b/docs/technologies/kafka/intro.md @@ -9,7 +9,7 @@ - Kafka connects to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library - Design is heavily influenced by [transaction logs](https://en.wikipedia.org/wiki/Transaction_log). (based on commit log) - Fault tolerant highly available queue used in publisher-subscriber or streaming application -- It can deliver message exactly once and also it keeps all the messages ordered inside a parition of a topic +- It can deliver message exactly once and also it keeps all the messages ordered inside a partition of a topic - Horizontally scalable, fault-tolerant, wicked fast - Focuses on real-time analysis, not batch jobs - Streams and streams only @@ -35,15 +35,15 @@ ### Concepts -- Kafka is run as a cluster on one or more servers that can span multiple datacenters. -- The Kafka cluster stores streams of *records*in categories called *topics*. +- Kafka is run as a cluster on one or more servers that can span multiple data-centers. +- The Kafka cluster stores streams of *records* in categories called *topics*. - Each record consists of a key, a value, and a timestamp. ### APIs - The [Producer API](https://kafka.apache.org/documentation.html#producerapi) allows an application to publish a stream of records to one or more Kafka topics. - The [Consumer API](https://kafka.apache.org/documentation.html#consumerapi) allows an application to subscribe to one or more topics and process the stream of records produced to them. -- The [Streams API](https://kafka.apache.org/documentation/streams) allows an application to act as a*stream processor*, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. +- The [Streams API](https://kafka.apache.org/documentation/streams) allows an application to act as a *stream processor*, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. - The [Connector API](https://kafka.apache.org/documentation.html#connect) allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. ### Architecture @@ -107,19 +107,9 @@ K-Stream , K-Table ## Managed Services -- Confluent Cloud - Apache Kafka as a service - - - -- AWS - Amazon Kinesis Streams - - - - - -- Google Cloud - Confluent Cloud on GCP - - +- Confluent Cloud - Apache Kafka as a service - +- AWS - Amazon Kinesis Streams - - +- Google Cloud - Confluent Cloud on GCP - ## Meet Kafka (Definitive Guide) diff --git a/static/img/screenshot2..jpg b/static/img/screenshot2.jpg similarity index 100% rename from static/img/screenshot2..jpg rename to static/img/screenshot2.jpg