Skip to content

Commit

Permalink
updated docs
Browse files Browse the repository at this point in the history
  • Loading branch information
deepaksood619 committed Nov 28, 2024
1 parent 5180952 commit 6770fd3
Show file tree
Hide file tree
Showing 33 changed files with 651 additions and 61 deletions.
4 changes: 4 additions & 0 deletions docs/about-me/achievements-activities-awards.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ I was responsible of creating a product and infrastructure around it to help sav

## Activities

### Data Structures for Interview - UnsaidTalks (30 November 2024)



### Meetup Talk - Real-Time Data Warehousing Solution with AWS and Confluent Cloud - OpsTree Solutions (23 November 2024)

- Presentation - [Real-Time Data Warehousing Solution with AWS and Confluent Cloud ​](https://docs.google.com/presentation/d/118S2du_m7mJ2WFsbwtmFuLz6NApfblv-FmbBE3XQ6PU/edit?usp=sharing)
Expand Down
1 change: 1 addition & 0 deletions docs/about-me/ideas/education.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ Highly personalized education for each person
- [Meet Khanmigo, Khan Academy's AI-powered teaching assistant & tutor](https://www.khanmigo.ai/)
- [Khan Academy | Free Online Courses, Lessons & Practice](https://www.khanacademy.org/)
- Youtube Learning (Experimental)
- [Sakshm AI](https://sakshm.com/)

## Free Education with Boarding (Coaching)

Expand Down
12 changes: 7 additions & 5 deletions docs/ai/data-visualization/charts.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@
14. Pheno plot
15. Cladogram - https://en.wikipedia.org/wiki/Cladogram
16. Flame Graphs

- Icicle graph
- Flame Charts
- [Miha Rekar - What Are Flame Graphs and How to Read Them, RubyConfBY 2017](https://www.youtube.com/watch?v=6uKZXIwd6M0)
- [USENIX ATC '17: Visualizing Performance with Flame Graphs](https://www.youtube.com/watch?v=D53T1Ejig1Q)
- Icicle graph
- Flame Charts
- [Miha Rekar - What Are Flame Graphs and How to Read Them, RubyConfBY 2017](https://www.youtube.com/watch?v=6uKZXIwd6M0)
- [USENIX ATC '17: Visualizing Performance with Flame Graphs](https://www.youtube.com/watch?v=D53T1Ejig1Q)
- Django
- [GitHub - jjnanthakumar/django-flames: This is simple site created in Django framework.. If any issues ping me.](https://github.com/jjnanthakumar/django-flames)
- [GitHub - blopker/djdt-flamegraph: Flamegraphs for Django Debug Toolbar](https://github.com/blopker/djdt-flamegraph)

17. Sparklines

Expand Down
9 changes: 8 additions & 1 deletion docs/ai/llm/prompt-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Presentation should be 30 mins long with 10-15 slides. I am aiming for a script

ChatGPT, could you aid me in crafting a compelling proposal presentation script for a project centered around integrating AI customer service solutions for Client's Name? I am aiming for a script that is persuasive, highlighting the advantages of our proposed solutions distinctively.

#### Prompt 3
#### Ultimate Presentation Prompt

Create a presentation using below details

Expand All @@ -74,6 +74,7 @@ Create a presentation using below details
1. Introduction to GenAI
2. Introduction to RAG
3. Understanding RAG Fundamentals (Core Components)
- Number of slides - 10-12
- Target audience - Tech professionals and developers in the software industry
- Target audience description - The audience consists of individuals interested in AI technology and software development.
- Target audience expertise level - Intermediate to advanced level in AI and software development
Expand All @@ -92,6 +93,12 @@ Create a presentation using below details
- Large - Detailed paragraphs - ideal for standalone presentations
- Create speaker notes that can help explain more about the slide to the audience

Only create the slides till agenda, then we will start creating slides step by step

https://chatgpt.com/share/674780e9-c504-8005-a023-b7aa945329d4

### Prompting Guide

![ChatGPT Ultimate Prompting Guide](../../media/chatgpt-promt-engineering.png)

![ChatGPT Prompts Commands](../../media/chatgpt-prompts.png)
Expand Down
1 change: 1 addition & 0 deletions docs/ai/others-resources-interview-learning-courses.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,3 +278,4 @@ https://towardsdatascience.com/how-to-build-a-real-time-fraud-detection-pipeline
- [It's All About AI • Martin Förtsch, Thomas Endres & Jonas Mayer • YOW! 2022 - YouTube](https://www.youtube.com/watch?v=krihq4iSaO0)
- [The State of Competitive Machine Learning | ML Contests](https://mlcontests.com/state-of-competitive-machine-learning-2023/)
- [Deep-ML](https://www.deep-ml.com/)
- [Big Data Engineer Mock Interview | AWS | Kafka Streaming | SQL | PySpark Optimization #interview - YouTube](https://www.youtube.com/watch?v=t3_ZzPoJx_Y&ab_channel=SumitMittal)
7 changes: 7 additions & 0 deletions docs/cloud/others/low-code-no-code-lcnc.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,20 @@

## nocode website builders

- [Builder.io: Visual Development Platform](https://www.builder.io/) - Free Open Source
- [GitHub - BuilderIO/builder: Visual Development for React, Vue, Svelte, Qwik, and more](https://github.com/builderio/builder)
- [Webflow: Create a custom website | No-code website builder](https://webflow.com/)
- [Drag and Drop Website Builder - Create Websites with WebWave](https://webwave.me/)
- [Locofy.ai - ship your products 5-10x faster - with low code](https://www.locofy.ai/)
- [Plasmic | Build powerful apps fast- without the limits](https://www.plasmic.app/) - [GitHub - plasmicapp/plasmic: Visual builder for React. Build apps, websites, and content. Integrate with your codebase.](https://github.com/plasmicapp/plasmic)
- [DhiWise: App Development Platform for High Productivity](https://www.dhiwise.com/)
- [Dora AI - Generating powerful websites, one prompt at a time](https://www.dora.run/ai)

### Others

- [Create a Free Website with a Drag-and-Drop Builder | HubSpot](https://www.hubspot.com/products/cms/drag-and-drop-website-builder)
- [The 6 best free website builders in 2024 | Zapier](https://zapier.com/blog/best-website-builders/)

## Links

- [The 8 best no-code app builders | Zapier](https://zapier.com/blog/best-no-code-app-builder/)
5 changes: 5 additions & 0 deletions docs/cloud/others/others-saas.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,12 @@ A warehouse management system is software that helps companies manage and contro
- https://www.lever.co
- [sap](cloud/others/sap.md)
- Best - **[GitHub - frappe/hrms: Open Source HR and Payroll Software](https://github.com/frappe/hrms)**

## Education

- [Superset | Automate Campus Placement and Campus Recruitment](https://joinsuperset.com/) - For Colleges
- moodle - [GitHub - moodle/moodle: Moodle - the world's open source learning platform](https://github.com/moodle/moodle) - [Home | Moodle.org](https://moodle.org/)
- [Ethos Open Software | Ellucian](https://www.ellucian.com/solutions/ellucian-ethos)

## Telegram

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ from itertools import permutations
len(list(permutations('abcde', 5)))
```

## Others
## Others - Online links

- https://www.geeksforgeeks.org/split-the-given-array-into-k-sub-arrays-such-that-maximum-sum-of-all-sub-arrays-is-minimum
- [Design LRU Cache - GeeksforGeeks](https://www.geeksforgeeks.org/design-a-data-structure-for-lru-cache/) - (good, medium difficulty, doubly linkedin list + hashmap)
Expand All @@ -95,6 +95,8 @@ len(list(permutations('abcde', 5)))
- Medium - https://www.hackerrank.com/challenges/beautiful-triplets/problem
- Medium - https://www.hackerrank.com/challenges/kaprekar-numbers/problem - https://practice.geeksforgeeks.org/problems/coin-change2448/1
- Medium - [Climbing the Leaderboard | HackerRank](https://www.hackerrank.com/challenges/climbing-the-leaderboard/problem?isFullScreen=true)
- Easy - String - All languages available - [Two Strings | HackerRank](https://www.hackerrank.com/challenges/two-strings/problem?isFullScreen=true)
- Good - Easy - String - Palindrome - [The Love-Letter Mystery | HackerRank](https://www.hackerrank.com/challenges/the-love-letter-mystery/problem?isFullScreen=true)

## Sources

Expand Down
4 changes: 1 addition & 3 deletions docs/data-structures/hashtable/hash-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,9 +88,7 @@ When a new key collides, find next empty slot, and put it there
## Optimization

1. Create a bigger hash table when older hash table gets full and rehash all old values to new hash table
2. Can use Consistent Hashing Functions -

Consistent Hashing is a special kind of hashing such that when a hash table is resized, only K/n keys need to be remapped on average, where K is the number of the keys, and n is the number of slots. In contrast, in most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped because the mapping between the keys and slots are defined by a modular operation.
2. Can use Consistent Hashing Functions

## Hash Table Context

Expand Down
144 changes: 137 additions & 7 deletions docs/data-structures/hashtable/hashing.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,145 @@ A **hash function** is any [function](https://en.wikipedia.org/wiki/Function_(ma

Consistent Hashing is a special kind of hashing such that when a hash table is resized, only K/n keys need to be remapped on average, where K is the number of the keys, and n is the number of slots. In contrast, in most traditional hash tables, a change in the number of array slots causes nearly all keys to be remapped because the mapping between the keys and slots are defined by a modular operation.

**Consistent hashing** is a key concept in distributed systems, especially for load balancing, distributed caching, and data partitioning. Unlike standard hashing techniques that can cause significant reorganization when nodes are added or removed, consistent hashing minimizes disruption by redistributing only a fraction of the data.

### How Consistent Hashing Works

#### 1. Hash Ring

- Keys and nodes are hashed into the same numeric space (often visualized as a circle or ring).
- The hash function maps both nodes (e.g., servers) and keys (e.g., data items) onto this ring.

#### 2. Data Assignment

- A key is assigned to the first node that is encountered in a clockwise direction on the ring.

#### 3. Node Addition/Removal

- When a node is added, it only affects the keys that fall between it and the previous node in the ring.
- When a node is removed, only the keys that were assigned to it need reassignment.

### Implementation

```python
import hashlib

class ConsistentHashing:
def __init__(self, nodes=None, replicas=3):
self.replicas = replicas
self.ring = {}
self.sorted_keys = []
if nodes:
for node in nodes:
self.add_node(node)

def hash(self, key):
return int(hashlib.md5(key.encode('utf-8')).hexdigest(), 16)

def add_node(self, node):
for i in range(self.replicas):
hash_val = self.hash(f"{node}:{i}")
self.ring[hash_val] = node
self.sorted_keys.append(hash_val)
self.sorted_keys.sort()

def remove_node(self, node):
for i in range(self.replicas):
hash_val = self.hash(f"{node}:{i}")
del self.ring[hash_val]
self.sorted_keys.remove(hash_val)

def get_node(self, key):
if not self.ring:
return None
hash_val = self.hash(key)
for h in self.sorted_keys:
if hash_val <= h:
return self.ring[h]
return self.ring[self.sorted_keys[0]]

# Example Usage
nodes = ['Node1', 'Node2', 'Node3']
ch = ConsistentHashing(nodes)
print(ch.get_node('my_data_key'))
```

### Advantages

1. Reduces data movement during scaling
2. Provides a fair and balanced distribution
3. Supports redundancy with ease for high availability

### Disadvantages

1. Requires a reliable hash function to minimize hotspots
2. Managing virtual nodes adds complexity
3. May require additional mechanisms for replication and failure handling

### Algorithms

#### 1. Classic Consistent Hashing

- **Overview**: Maps both keys and nodes onto a circular hash ring. Each key is assigned to the closest node in the ring (moving clockwise).
- **Use Cases**: Distributed caching (e.g., Memcached), load balancing.
- **Features**:
- Minimal data movement when nodes are added or removed.
- Balances data across nodes based on their position on the ring.
- **Example Libraries**:
- Python: `hash_ring`
- Java: `Ketama`

#### 2. Maglev Hashing

- **Overview**: Developed by Google, Maglev hashing creates a consistent, precomputed lookup table for faster key-to-node mapping.
- **Use Cases**: High-performance load balancing in network services.
- **Features**:
- Simple implementation.
- Ensures minimal disruption and balanced loads when adding or removing nodes.
- **Example**: Used in Google’s load balancers.

#### 3. Jump Consistent Hashing

- **Overview**: A non-ring-based hashing technique that directly maps keys to a fixed set of buckets.
- **Use Cases**: Scenarios with a known, fixed number of buckets.
- **Features**:
- High performance (O(1) operations).
- Minimal movement of keys.
- **Example**: Popular in cloud storage systems and databases.

#### 4. Rendezvous Hashing (Highest Random Weight Hashing)

- **Overview**: Keys are mapped to nodes by calculating a weight for each key-node pair and assigning the key to the node with the highest weight.
- **Use Cases**: Distributed databases, consistent key-to-node mapping.
- **Features**:
- Provides optimal load balancing.
- Resilient to node failures.
- **Example**: Used in distributed systems like Amazon DynamoDB.

#### 5. Consistent Hashing with Virtual Nodes

- **Overview**: Adds virtual nodes (or replicas) for each physical node on the hash ring to ensure even data distribution.
- **Use Cases**: Systems with unevenly loaded or heterogeneous nodes.
- **Features**:
- Reduces the risk of hotspots.
- Improves load distribution for systems with varying node capacities.
- **Example**: Used in Apache Cassandra and Amazon Dynamo.

#### 6. Weighted Consistent Hashing

- **Overview**: Assigns weights to nodes, ensuring that higher-capacity nodes are responsible for more keys.
- **Use Cases**: Heterogeneous systems with nodes of varying capabilities.
- **Features**:
- Balances the load according to node capacity.
- **Example**: Used in CDN services and distributed storage systems.

https://www.akamai.com/es/es/multimedia/documents/technical-publication/consistent-hashing-and-random-trees-distributed-caching-protocols-for-relieving-hot-spots-on-the-world-wide-web-technical-publication.pdf

[The Ultimate Guide to Consistent Hashing | Toptal](https://www.toptal.com/big-data/consistent-hashing)

## Modular based hashing

## Applications
### Applications

- **Associative arrays:** Hash tables are commonly used to implement many types of in-memory tables. They are used to implement associative arrays (arrays whose indices are arbitrary strings or other complicated objects).
- **Database indexing:** Hash tables may also be used as disk-based data structures and database indices (such as in dbm).
Expand All @@ -33,17 +165,15 @@ https://www.akamai.com/es/es/multimedia/documents/technical-publication/consiste
- Search algorithms (e.g. game tree exploration)
- use hash table to avoid exploring any configuration (e.g. arrangement of chess pieces) more than once

## How to choose n = # of buckets (When using modulus to hash values to buckets)
How to choose n = # of buckets (When using modulus to hash values to buckets)

1. Choose n to be prime (should be few factors, within constant factor of # of objects in table)

2. not too close to a power of 2

3. not too close to a power of 10

## The load of a hash table
The load of a hash table

## alpha = # of objects in hash table / # of buckets of hash table
alpha = # of objects in hash table / # of buckets of hash table

Question - which hash table implementation strategy is feasible for load factors larger than 1 (Answer - Only chaining, because # of objects are greater than # of buckets, so only using linked list for collision can be used)

Expand All @@ -53,7 +183,7 @@ So if load factor increases, we increase the number of buckets.

A super hash function, that can handle all types of data **does not exists** (for every hash function there exists a pathological data set)

## Solution to pathological data set
### Solution to pathological data set

- Use a cryptographic hash function (e.g. SHA-2)

Expand Down
8 changes: 6 additions & 2 deletions docs/databases/sql-databases/mysql/percona-toolkit.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@
- Analyze MongoDB query profiler
- Collect vital system information when problems occur

`brew install percona-toolkit`
```bash
sudo apt-get install percona-toolkit

brew install percona-toolkit
```

- [pt-align](https://www.percona.com/doc/percona-toolkit/LATEST/pt-align.html)
- [pt-archiver](https://www.percona.com/doc/percona-toolkit/LATEST/pt-archiver.html)
Expand Down Expand Up @@ -55,7 +59,7 @@ pt-mysql-summary --host localhost --user root --ask-pass > mysql-summary.txt

pt-mysql-summary --host sttash-main-db-instance-new-cluster.cluster-ro-c1z93jsyca9u.ap-south-1.rds.amazonaws.com --user lms-website --password Rf9zdHwB9E3GHWKq2yZM > mysql-summary.txt- [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html)

pt-online-schema-change --host dailydb.snapshot.example.com --user deepak_sood --password 72062eacf89016e8c2bb4fe9c4457b90 --alter='ENGINE=InnoDB' D=sttash_website_LIVE, t=elev8_offer_tmp --preserve-triggers --dry-run
pt-online-schema-change --host dailydb.snapshot.example.com --user user_abc --password xyzabc123 --alter='ENGINE=InnoDB' D=db_name, t=table_name --preserve-triggers --dry-run
```

Works by creating an empty copy of the table to alter, modifying it as desired, and then copying rows from the original table into the new table. When the copy is complete, it moves away the original table and replaces it with the new one. By default, it also drops the original table.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -918,6 +918,9 @@ Ctrl+a d
# To resume your screen session
screen -r
# To attach to specific session
screen -r session_name
# list the current running screen sessions
screen -ls
Expand Down
4 changes: 4 additions & 0 deletions docs/economics/fintech-nbfc-banking-terms.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,10 @@ https://fintechnews.sg/62502/fintech-india/updated-top-30-open-banking-influence
- Finbox - https://finbox.in
- https://sahamati.org.in

## Conferences

- [India Global Forum](https://indiaglobalforum.com/)

## Links

[BNPL](economics/buy-now-pay-later-bnpl.md)
Expand Down
2 changes: 2 additions & 0 deletions docs/economics/mental-models/macroeconomics.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ The **impossible trinity** (also known as **the impossible [trilemma](https://en

[As India’s red hot economy set to be world no. 3, reality check on how it stacks up with China, US - YouTube](https://www.youtube.com/watch?v=BYp1HnYUiAo)

[Seeking wisdom in the Indian Stock Markets | SOIC](https://soic.in/blog-description/top-10-learnings-for-october)

## Tools

[The Atlas of Economic Complexity](https://atlas.cid.harvard.edu/)
Expand Down
Loading

0 comments on commit 6770fd3

Please sign in to comment.