diff --git a/docs/about-me/ideas/creating-a-business-startup.md b/docs/about-me/ideas/creating-a-business-startup.md index 439ea2c314a..da5df704e8f 100755 --- a/docs/about-me/ideas/creating-a-business-startup.md +++ b/docs/about-me/ideas/creating-a-business-startup.md @@ -283,3 +283,5 @@ Discovery commerce is the equivalent of 'I'm feeling lucky' in Google searches. [13 Businesses Making +$1,000,000/Year With 0 Employees - YouTube](https://www.youtube.com/watch?v=giis6_u8n3M) [How to build a company board as a founder?](https://shorts.growthx.club/how-to-build-a-board-with-intent) + +[GitHub - sdg-1/consulting-handbook: A guide for technical professionals looking to start consulting](https://github.com/sdg-1/consulting-handbook) diff --git a/docs/about-me/ideas/donations.md b/docs/about-me/ideas/donations.md index 7621a2279c2..425b8281b45 100644 --- a/docs/about-me/ideas/donations.md +++ b/docs/about-me/ideas/donations.md @@ -26,6 +26,8 @@ Donor - finds a person and donate the required item or buy for him/her [Doing Good Better](../../book-summaries/doing-good-better) +[Nudging to donate organs: do what you like or like what we do? | Medicine, Health Care and Philosophy](https://link.springer.com/article/10.1007/s11019-021-10007-6) + ### Effective Altruism When you become rich don't increase your standard of living, increase your standard of giving. diff --git a/docs/about-me/ideas/education.md b/docs/about-me/ideas/education.md index 9301885348b..a906a8848f5 100755 --- a/docs/about-me/ideas/education.md +++ b/docs/about-me/ideas/education.md @@ -25,6 +25,14 @@ Highly personalized education for each person **DigiProctor** with a 360 degree camera with all AI +#### Competitors + +[linkedin.com/in/qudduspat/](https://www.linkedin.com/in/qudduspat/)- qp@asi.tech + +[How a High Schooler in Dubai Built ASI, One of the Most Promising AI Edtech Startups](https://www.businessinsider.com/edtech-asi-ai-dubai-chatbot-tutoring-assistant-2023-9?IR=T) + +[ASI | Home](https://www.asi.tech/) + ### Free Education with Boarding (Coaching) - Job Guarantee and skill focused diff --git a/docs/about-me/ideas/startup-ideas-physical.md b/docs/about-me/ideas/startup-ideas-physical.md index 30129b2bf34..8950487508b 100755 --- a/docs/about-me/ideas/startup-ideas-physical.md +++ b/docs/about-me/ideas/startup-ideas-physical.md @@ -161,6 +161,10 @@ Airline can be transporter of fuels if passenger occupancy is lower, i.e. based - Transport - Food +## Autonomous Vehicles / Driverless Cars + +[New data shows Waymo crashes a lot less than human drivers](https://www.understandingai.org/p/new-data-shows-waymo-crashes-a-lot) + ## Others - Beekeeping diff --git a/docs/about-me/ideas/thoughts-future-ideas-vision-prediction.md b/docs/about-me/ideas/thoughts-future-ideas-vision-prediction.md index efd156fd3b6..e45977b3dec 100755 --- a/docs/about-me/ideas/thoughts-future-ideas-vision-prediction.md +++ b/docs/about-me/ideas/thoughts-future-ideas-vision-prediction.md @@ -19,10 +19,6 @@ Blockchain reinvents trust. Humans cannot be trusted, they trust software more than other humans - Haven't you studied true history as to what humans have done to other humans. -## Arguments - -What is your sample size? - ## Technologies [Can Rice last forever?](https://finshots.in/archive/can-rice-last-forever/) @@ -61,3 +57,16 @@ What is your sample size? ## Links [Ask HN: What are the big/important problems to work on? | Hacker News](https://news.ycombinator.com/item?id=37033312) + +[Why AI Will Spark Exponential Economic Growth | Cathie Wood | TED - YouTube](https://www.youtube.com/watch?v=rQEh7d-qa38&ab_channel=TED) + +- Old + - Telephone + - Electricity + - Automobile +- New + - Artificial Intelligence + - Public Blockchains + - Energy Storage + - Robotics (Autonomous Vehicles) + - Multinomic Sequencing diff --git a/docs/ai/data-science/data-visualization/metabase.md b/docs/ai/data-science/data-visualization/metabase.md index 4a927c16d01..feac5aec1d4 100644 --- a/docs/ai/data-science/data-visualization/metabase.md +++ b/docs/ai/data-science/data-visualization/metabase.md @@ -51,6 +51,23 @@ Parameters can be signed or unsigned. [Parameters for signed embeds](https://www.metabase.com/docs/latest/embedding/signed-embedding-parameters) +## Plugins + +```bash +# clickhouse + +export METABASE_DOCKER_VERSION=v0.47.2 +export METABASE_CLICKHOUSE_DRIVER_VERSION=1.2.2 + +mkdir -p mb/plugins && cd mb + +curl -L -o plugins/ch.jar https://github.com/ClickHouse/metabase-clickhouse-driver/releases/download/$METABASE_CLICKHOUSE_DRIVER_VERSION/clickhouse.metabase-driver.jar + +docker run -d -p 3000:3000 \ + --mount type=bind,source=$PWD/plugins/ch.jar,destination=/plugins/clickhouse.jar --network="host" \ + metabase/metabase:$METABASE_DOCKER_VERSION +``` + ## Dashboards ### Filters diff --git a/docs/ai/libraries/ml-model-deployment.md b/docs/ai/libraries/mlops-model-deployment.md similarity index 59% rename from docs/ai/libraries/ml-model-deployment.md rename to docs/ai/libraries/mlops-model-deployment.md index 3ce03bbce54..1a355d99d27 100755 --- a/docs/ai/libraries/ml-model-deployment.md +++ b/docs/ai/libraries/mlops-model-deployment.md @@ -1,4 +1,4 @@ -# ML Model Deployment +# MLOps / Model Deployment ![image](../../media/ML-Model-Deployment-image1.jpg) @@ -26,8 +26,21 @@ MLOps is an engineering discipline that aims to unify ML systems development (de - Deploy ML models - Iterate - Monitor, optimize and maintain the performance of the model -https://www.freecodecamp.org/news/what-is-mlops-machine-learning-operations-explained +###  Deploying models to the production system -[MLOps guide](https://huyenchip.com/mlops/) +There are mainly two ways of deploying an ML model: + +- **Static deployment or embedded model**  -  where the model is packaged into installable application software and is then deployed. For example, an application that offers batch-scoring of requests. +- **Dynamic deployment**  -  where the model is deployed using a web framework like FastAPI or Flask and is offered as an API endpoint that responds to user requests. + +[What is MLOps? Machine Learning Operations Explained](https://www.freecodecamp.org/news/what-is-mlops-machine-learning-operations-explained) [MLOps Course – Build Machine Learning Production Grade Projects - YouTube](https://www.youtube.com/watch?v=-dJPoLm_gtE) + +[MLOps guide](https://huyenchip.com/mlops/) + +### Tools + +[ZenML - Seamless End-to-End MLOps](https://www.zenml.io/) + +[Starter guide - ZenML Documentation](https://docs.zenml.io/user-guide/starter-guide) diff --git a/docs/ai/libraries/readme.md b/docs/ai/libraries/readme.md index f34fa67d9d1..a922ff85c0e 100755 --- a/docs/ai/libraries/readme.md +++ b/docs/ai/libraries/readme.md @@ -1,7 +1,7 @@ # Libraries - [AWS Sagemaker](aws-sagemaker) -- [ML Model Deployment](ml-model-deployment) +- [ML Model Deployment](ai/libraries/mlops-model-deployment.md) - [Distributed Training](distributed-training) - [Libraries](ml-libraries.md) - [Tools](ai/libraries/tools.md) diff --git a/docs/ai/llm/interview-questions.md b/docs/ai/llm/interview-questions.md new file mode 100644 index 00000000000..eeab21ee71a --- /dev/null +++ b/docs/ai/llm/interview-questions.md @@ -0,0 +1,346 @@ +# Interview Questions + +## Can you provide a high-level overview of Transformers' architecture? + +Let’s begin by looking at the model as a _single black box_. In a machine translation application, it would take a sentence in one language, and output its translation in another, as illustrated below, + +![](https://jalammar.github.io/images/t/the_transformer_3.png) + +Getting closer into the black box, transformers have on the inside: + +- An **encoding component**: which is a stack of `N` encoders. +- A **decoding component**: which is a stack of `N` decoders, +- and **connections between them**. + +![](https://jalammar.github.io/images/t/The_transformer_encoder_decoder_stack.png) + +Now, each **encoder** is broken down into two sub-layers: the **self-attention layer** and the **feed-forward neural network layer**. + +The inputs first flow through a **self-attention layer**, and the outputs of the **self-attention layer** are fed to a **feed-forward neural network**. And this sequence is repeated till reaches the last encoder. + +Finally, the **decoder** receives the output of the **encoder component** and also has both the **self-attention layer** and **feed-forward layer**, and the flow is similar to before, but between them there is an **attention layer** that helps the decoder focus on relevant parts of the input sentence. + +![](https://jalammar.github.io/images/t/Transformer_decoder.png) + +## How _next sentence prediction (NSP)_ is used in language modeling? + +**Next sentence prediction (NSP)** is used in language modeling as _one-half_ of the training process behind the **BERT** model (the other half is _masked-language modeling (MLM)_). The objective of next-sentence prediction training is to predict whether one sentence logically follows the other sentence presented to the model. + +During training, the model is presented with pairs of sentences, some of which are consecutive in the original text, and some of which are not. The model is then trained to predict whether a given pair of sentences are adjacent or not. This allows the model to **understand longer-term dependencies across sentences**. + +Researchers have found that without **NSP**, **BERT** performs worse on every single metric — so its use it’s relevant to language modeling. + +## How can you _evaluate the performance_ of Language Models? + +There are two ways to evaluate language models in **NLP**: **_Extrinsic evaluation_** and **_Intrinsic evaluation_**. + +- **Intrinsic evaluation** captures how well the model captures what it is supposed to capture, like probabilities. +- **Extrinsic evaluation** (or task-based evaluation) captures how useful the model is in a particular task. + +A common **intrinsic evaluation** of **LM** is the **perplexity**. It's a geometric average of the inverse probability of words predicted by the model. Intuitively, perplexity means to be _surprised_. We measure _how much the model is surprised_ by seeing new data. The lower the perplexity, the better the training is. Another common measure is the **cross-entropy**, which is the Logarithm (base `2`) of _perplexity_. As a thumb rule, a reduction of `10-20%` in perplexity is noteworthy. + +The **extrinsic evaluation** will depend on the task. Example: For _speech recognition_, we can compare the performance of two language models by running the speech recognizer twice, once with each language model, and seeing which gives the more accurate transcription. + +## How do _generative language models_ work? + +The very basic idea is the following: they take `n` tokens as input, and produce `one` token as output. + +![](https://miro.medium.com/v2/resize:fit:828/format:webp/1*zaRZlVk-dl0zVUOe0g_ufg.png) + +A token is a chunk of text. In the context of OpenAI GPT models, common and short words typically correspond to a single token and long and less commonly used words are generally broken up into several tokens. + +This basic idea is applied in an _expanding-window pattern_. You give it `n` tokens in, it produces `one` token out, then it incorporates that output token as part of the input of the next iteration, produces a new token out, and so on. This pattern keeps repeating until a stopping condition is reached, indicating that it finished generating all the text you need. + +Now, behind the output is a probability distribution over all the possible tokens. What the model does is return a vector in which each entry expresses the probability of a particular token being chosen. + +![](https://miro.medium.com/v2/resize:fit:828/format:webp/1*UZ_hexQINPnJV14HWLorAg.png) + +This probability distribution comes from the training phase. During training, the model is exposed to a lot of text, and its weights are tuned to predict good probability distributions, given a sequence of input tokens. + +GPT generative models are trained with a large portion of the internet, so their predictions reflect a mix of the information they’ve seen. + +## What is a _token_ in the Large Language Models context? + +ChatGPT and other LLMs rely on input text being broken into pieces. Each piece is about a word-sized sequence of characters or smaller. We call those sub-word tokens. That process is called _tokenization_ and is done using a _tokenizer_. + +**Tokens** can be words or just chunks of characters. For example, the word “hamburger” gets broken up into the tokens “ham”, “bur” and “ger”, while a short and common word like “pear” is a single token. Many tokens start with whitespace, for example, “ hello” and “ bye”. + +The models understand the statistical relationships between these tokens and excel at producing the next token in a sequence of tokens. + +The number of tokens processed in a given API request depends on the length of both your inputs and outputs. As a rough rule of thumb, the `1` token is approximately `4` characters or `0.75` words for English text. + +Consider: + +![https://miro.medium.com/v2/resize:fit:1400/1*yM5gg3cPHDvo7AAqDZryZA.png](https://miro.medium.com/v2/resize:fit:1400/1*yM5gg3cPHDvo7AAqDZryZA.png) + +## What's the advantage of using transformer-based vs LSTM-based architectures in NLP? + +To create **sequence-to-sequence** models before the **Transformer**, we used the famous **LSTM** with its **_Encoder-Decoder architecture_**, where + +- The "**Encoder**" part that creates a _vector representation_ of a _sequence of words_. +- The "**Decoder**" returns a _sequence of words_ from the _vector representation_. + +The **LSTM** model takes into account the **interdependence of words**, so we need inputs of the previous state to make any operations on the current state. This model has a limitation: it is relatively slow to train and the input sequence can't be passed in parallel. + +Now, the idea of the **Transformer** is to maintain the **interdependence of the words** in a sequence without using a recurrent network but only the **attention mechanism** that is at the center of its architecture. The **attention** measures **how closely two elements of two sequences are related**. + +In transformer-based architectures, the attention mechanism is applied to a _single sequence_ (also known as a **_self-attention layer_**). The **self-attention layer** determines the **interdependence of different words** in the same sequence, to associate a relevant representation with it. Take for example the sentence: "_The dog didn't cross the street because it was too tired_". It is obvious to a human being that "_it_" refers to the "_dog_" and not to the "_street_". The objective of the self-attention process will therefore be to detect the link between "_dog_" and "_it_". This feature makes transformers much faster to train compared to their predecessors, and they have been proven to be more robust against noisy and missing data. + +As a plus, in **contextual embeddings**, transformers can draw information from the context to correct missing or noisy data and that is something that other neural networks couldn’t offer. + +## Can you provide some examples of _alignment problems_ in Large Language Models? + +The **alignment problem** refers to the extent to which a model's goals and behavior align with human values and expectations. + +Large Language Models, such as `GPT-3`, are trained on vast amounts of text data from the internet and are capable of generating human-like text, but they may not always produce output that is consistent with human expectations or desirable values. + +The _alignment problem_ in Large Language Models typically manifests as: + +- **Lack of helpfulness**: when the model is not following the user's explicit instructions. +- **Hallucinations**: when the model is making up unexisting or wrong facts. +- **Lack of interpretability**: when it is difficult for humans to understand how the model arrived at a particular decision or prediction. +- **Generating biased or toxic output**: when a language model that is trained on biased/toxic data may reproduce that in its output, even if it was not explicitly instructed to do so. + +## How _Adaptative Softmax_ is useful in Large Language Models? + +**Adaptive softmax** is useful in large language models because it allows for efficient training and inference when dealing with _large vocabularies_. **_Traditional softmax_** involves computing probabilities for _each word_ in the vocabulary, which can become computationally expensive as the vocabulary size grows. + +**Adaptive softmax** reduces the number of computations required by grouping words together into clusters based on **_how common the words are_**. This reduces the number of computations required to compute the probability distribution over the vocabulary. + +Therefore, by using adaptive softmax, large language models can be trained and run more efficiently, allowing for faster experimentation and development. + +## How does BERT _training_ work? + +**BERT** (_Bidirectional Encoder Representations from Transformers_) utilizes a **_transformer architecture_** that learns contextual relationships between words in a text and since BERT’s goal is to generate a language representation model, it only needs the **_encoder_** part. + +The input to the **_encoder_** for **BERT** is a sequence of _tokens_, which are first converted into _vectors_ and then processed in the neural network. Then, the **BERT** algorithm makes use of the following two training techniques: + +- **Masked LM (MLM)**: Before feeding word sequences into **BERT**, a percentage of the words in each sequence are replaced with a `[MASK]` token. The model then attempts to predict the original value of the masked words, based on the context provided by the other, non-masked, words in the sequence. +- **Next Sentence Prediction (NSP)**: the model concatenates two masked sentences as inputs during pretraining. Sometimes they correspond to sentences that were next to each other in the original text, and sometimes not. The model then has to predict if the two sentences were following each other or not. + +Now, to help the model distinguish between the two sentences in training, the input is processed with some extra metadata such as: + +- **_Token embeddings_**: A `[CLS]` token is inserted at the beginning of the first sentence and a `[SEP]` token is inserted at the end of each sentence. +- **_Segment embeddings_**: these assign markers to identify each sentence and allows the encoder to distinguish between them. +- **_Positional embeddings_**: to indicate the token position in the sentence. + +Then, to predict if the second sentence is indeed connected to the first, the following steps are performed: + +- The entire input sequence goes through the _Transformer model_. +- The output of the `[CLS]` token is transformed into a `2×1` shaped vector, using a simple _classification layer_ (learned matrices of weights and biases). +- Calculating the probability of `IsNextSequence` with _softmax_. + +When training the **BERT** model, **_Masked LM_** and **_Next Sentence Prediction_** are trained together, with the goal of minimizing the combined loss function of the two strategies. + +![https://miro.medium.com/v2/resize:fit:780/1*vUuxeQKHA5RHaaZ8-oVQ_A.png](https://miro.medium.com/v2/resize:fit:780/1*vUuxeQKHA5RHaaZ8-oVQ_A.png) + +## How is the _Transformer Network_ better than _CNNs_ and _RNNs_? + +- With **RNN**, you have to go _word by word_ to access to the cell of the last word. If the network is formed with a long reach, it may take several steps to remember, each masked state (output vector in a word) depends on the previous masked state. This becomes a major problem for GPUs. This sequentiality is an obstacle to the parallelization of the process. In addition, in cases where such sequences are too long, the model tends to forget the contents of the distant positions one after the other or to mix with the contents of the following positions. In general, whenever **_long-term dependencies_** are involved, we know that **RNN** suffers from the **_Vanishing Gradient Problem_**. +- Early efforts were trying to solve the dependency problem with **_sequential convolutions_** for a solution to the **RNN**. A long sequence is taken and the convolutions are applied. The disadvantage is that **CNN** approaches require many layers to capture long-term dependencies in the sequential data structure, without ever succeeding or making the network so large that it would eventually become impractical. +- The **Transformer** presents a new approach, it proposes to _encode_ each word and apply the **_mechanism of attention_** in order to connect two distant words, then the _decoder_ predicts the sentences according to all the words preceding the current word. This workflow can be parallelized, accelerating learning and solving the **_long-term dependencies_** problem. + +## Is there a way to train a Large Language Model (LLM) to store a specific context? + +The only way at the moment to "memorize" past conversations is to **include past conversations in the prompt**. + +Consider: + +```bash + +You are a friendly support person. The customer will ask you questions, and you will provide polite responses + +Q: My phone won't start. What do I do? <-- This is a past question + +A: Try plugging your phone into the charger for an hour and then turn it on. The most common cause for a phone not starting is that the battery is dead. + +Q: I've tried that. What else can I try? <-- This is a past question + +A: Hold the button for 15 seconds. It may need a reset. + +Q: I did that. It worked, but the screen is blank. <-- This is a current question + +A: + +``` + +You will hit a token limit at some point (if you chat long enough). Each GPT-3 model has a [maximum number of tokens](https://platform.openai.com/docs/models/gpt-3) you can pass to it. In the case of `text-davinci-003`, it is `4096` tokens. When you hit this limit, the OpenAI API will throw an error. + +## What _Transfer learning Techniques_ can you use in LLMs? + +There are several **Transfer Learning** techniques that are commonly used in **LLMs**. Here are three of the most popular: + +- **Feature-based transfer learning**: This technique involves using a pre-trained language model as a _feature extractor_, and then training a separate model on top of the extracted features for the target task. +- **Fine-tuning**: involves taking a pre-trained language model and training it on a _specific task_. Sometimes when fine-tuning, you can keep the model weights fixed and just add a new layer that you will train. Other times you can slowly unfreeze the layers one at a time. You can also use unlabelled data when pre-training, by masking words and trying to predict which word was masked. +- **Multi-task learning**: involves training a single model on multiple related tasks simultaneously. The idea is that the model will learn to share information across tasks and improve performance on each individual task as a result. + +## What is _Transfer Learning_ and why is it important? + +A pre-trained model, such as GPT-3, essentially takes care of massive amounts of hard work for the developers: It teaches the model to do a basic understanding of the problem and provides solutions in a _generic_ format. + +With **transfer learning**, given that the pre-trained models can generate basic solutions, we can _transfer the learning to another context_. As a result, we will be able to customize the model to our requirements using fine-tuning without the need to retrain the entire model. + +## What's the difference between _Encoder_ vs _Decoder_ models? + +**Encoder models**: + +- They use only the _encoder_ of a **Transformer model**. At each stage, the attention layers can access **all the words** in the **initial sentence**. +- The pretraining of these models usually revolves around somehow _corrupting_ a given sentence (for instance, by _masking_ random words in it) and tasking the model with **finding or reconstructing** the initial sentence. +- They are best suited for tasks requiring an **understanding of the full sentence**, such as sentence classification, named entity recognition (and more general word classification), and extractive question answering. + +**Decoder models**: + +- They use only the _decoder_ of a **Transformer model**. At each stage, for a given word the attention layers **can only access the words positioned before it in the sentence**. +- The pretraining of decoder models usually revolves around **predicting** the next word in the sentence. +- They are best suited for tasks involving **text generation**. + +![https://miro.medium.com/v2/resize:fit:1400/1*cfNpm7aDO4lD3e-Wkwgc1g.png](https://miro.medium.com/v2/resize:fit:1400/1*cfNpm7aDO4lD3e-Wkwgc1g.png) + +## What's the difference between _Wordpiece_ vs _BPE_? + +**WordPiece** and **BPE** are both _subword tokenization algorithms_. They work by breaking down words into smaller units, called subwords. We then define a desired **_vocabulary size_** and keep adding subwords till the limit is reached. + +- **BPE** starts with a vocabulary of all the characters in the training data. It then **_iteratively merges_** the **_most frequent pairs_** of characters until the desired _vocabulary size_ is reached. The merging is done _greedily_, meaning that the most frequent pair of characters is always merged first. +- **WordPiece** also starts with a vocabulary of all the characters in the training data. It then uses a **_statistical model_** to choose the pair of characters that is most likely to improve the likelihood of the training data until the _vocab size_ is reached. + +## What's the difference between _Global_ and _Local Attention_ in LLMs? + +Consider the example sentence “_Where is Wally_” which should be translated to its Italian counterpart “_Dove è Wally_”. In the transformer architecture, the **_encoder_** processes the input word by word, producing three different **_hidden states_**. + +Then, the **attention layer** produces a single fixed-size **_context vector_** from all the encoder hidden states (often with a weighted sum) and it represents the “_attention_” that must be given to that context when processing such input word. Here is when **_global_** and **_local_** attention comes into play. + +**Global attention** considers _all the hidden_ states in creating the **_context vector_**. When is applied, a lot of computation occurs. This is because all the hidden states must be taken into consideration, concatenated into a matrix, and processed by a **NN** to compute their weights. + +On the other hand, **local attention** considers _only a subset_ of all the hidden states in creating the **_context vector_**. The subset can be obtained in many different ways, such as with **Monotonic Alignment** and **Predictive Alignment**. + +![https://miro.medium.com/v2/resize:fit:1200/0*eOBSEK6zMxz-S5kK.jpeg](https://miro.medium.com/v2/resize:fit:1200/0*eOBSEK6zMxz-S5kK.jpeg) + +## What's the difference between _next-token-prediction_ vs _masked-language-modeling_ in LLM? + +Both are techniques for training large language models and involve predicting a word in a sequence of words. + +- **Next token prediction**: the model is given a sequence of words with the goal of predicting the next word. For example, given the phrase _Hannah is a \_\_\_\__, the model would try to predict: +- _Hannah is a sister_ +- _Hannah is a friend_ +- _Hannah is a marketer_ +- _Hannah is a comedian_ + +- **Masked-language-modeling**: the model is given a sequence of words with the goal of predicting a _masked_ word in the middle. For example, given the phrase, _Jako mask reading_, the model would try to fill the gap as, +- _Jacob fears reading_ +- _Jacob loves reading_ +- _Jacob enjoys reading_ +- _Jacon hates reading_ + +## Why a _Multi-Head Attention_ mechanism is needed in a _Transformer-based Architecture_? + +Take for example the sentence: + +> Bark is very cute and he is a dog. + +Here, if we take the word ‘`dog`’, grammatically we understand that the words ‘`Bark`’, ‘`cute`’, and ‘`he`’ should have some significance or relevance with the word ‘`dog`’. These words say that the dog’s name is Bark, it is a male dog, and that he is a cute dog. + +In simple terms, just one attention mechanism may not be able to correctly identify these three words as relevant to ‘`dog`’, and we can sense that three attentions are better here to signify the three words with the word ‘`dog`’. + +Therefore, to overcome some of the pitfalls of using **single attention**, **multi-head attention** is used. This reduces the load on one attention to find all significant words and also increases the chances of finding more relevant words easily. + +## Why would you use _Encoder-Decoder RNNs_ vs _plain sequence-to-sequence RNNs_ for _automatic translation_? + +A **plain sequence-to-sequence RNN** would start translating a sentence _immediately_ after reading the first word of a sentence, while an **Encoder-Decoder RNN** will first _read the whole sentence_ and then translate it. + +In general, if you translate a sentence one word at a time, the result will be terrible. For example, the french sentence "_Je vous en prie_" means "_You are welcome_" but if you translate it one word at a time using _plain sequence-to-sequence RNN_, you get "_I you in pray_" which it does not have sense. So in automatic translation cases is much better to use _Encoder-Decoder RNNs_ to read the whole sentence first and then translate it. + +## Explain what is _Self-Attention_ mechanism in the Transformer architecture? + +We can think of **self-attention** as a mechanism that enhances the information content of an input embedding by _including information about the input’s context_. In other words, the self-attention mechanism enables the model to weigh the importance of different elements in an input sequence and dynamically adjust their influence on the output. + +Imagine that we have a text _x_, which we convert from raw text using an embedding algorithm. To then apply the attention, we map a query (_q_) as well as a set of key-value pairs (_k_, _v_) to our output _x_. Both _q_, _k_, as well as v, are vectors. The result _z_ is called the attention-head and is then sent along a simple feed-forward neural network. + +![https://sebastianraschka.com/images/blog/2023/self-attention-from-scratch/transformer.png](https://sebastianraschka.com/images/blog/2023/self-attention-from-scratch/transformer.png) + +## How does _Transfer Learning_ work in LLMs? + +**Transfer learning** in **LLMs** involves training a language model on a _large corpus of text_, and then **fine-tuning** the model on a specific task. The process involves several steps: + +1. **Pre-training**: In this step, the language model is trained on a large corpus of text to learn the patterns and relationships within the language. The goal of pre-training is to generate a set of **general-purpose** language representations that can be used for a wide range of NLP tasks. + +2. **Fine-tuning**: Once the language model is pre-trained, it can be fine-tuned on a **specific NLP task**, such as sentiment analysis or text classification. Fine-tuning involves taking the pre-trained model and training it on a smaller dataset specific to the target task. The goal of fine-tuning is to adapt the general-purpose language representations to the specific task at hand. + +3. **Inference**: Once the model is fine-tuned, it can be used to make predictions on new data. During inference, the model takes in the input text and generates an output that represents the model’s prediction for the task at hand. + +![https://www.researchgate.net/publication/342547232/figure/fig4/AS:907957101404162@1593485250772/The-concept-of-the-transfer-learning.png](https://www.researchgate.net/publication/342547232/figure/fig4/AS:907957101404162@1593485250772/The-concept-of-the-transfer-learning.png) + +## How does an LLM _parameter_ relate to a _weight_ in a Neural Network? + +Yes, the parameters in a large language model (LLM) are similar to the weights in a standard neural network. In both LLMs and neural networks, these parameters are numerical values that start as random coefficients and are adjusted during training to minimize loss. These parameters include not only the weights that determine the strength of connections between neurons but also the biases, which affect the output of neurons. In a large language model (LLM) like GPT-4 or other transformer-based models, the term "parameters" refers to the numerical values that determine the behavior of the model. These parameters include weights and biases, which together define the connections and activations of neurons within the model. Here's a more detailed explanation: + +- **Weights**: Weights are numerical values that define the strength of connections between neurons across different layers in the model. In the context of LLMs, weights are primarily used in the attention mechanism and the feedforward neural networks that make up the model's architecture. They are adjusted during the training process to optimize the model's ability to generate relevant and coherent text. +- **Biases**: Biases are additional numerical values that are added to the weighted sum of inputs before being passed through an activation function. They help to control the output of neurons and provide flexibility in the model's learning process. Biases can be thought of as a way to shift the activation function to the left or right, allowing the model to learn more complex patterns and relationships in the input data. + +The training process involves adjusting these parameters (weights and biases) iteratively to minimize the loss function. This is typically done using gradient descent or a variant thereof, such as stochastic gradient descent or Adam optimizer. The loss function measures the difference between the model's predictions and the true values (e.g., the correct next word in a sentence). By minimizing the loss, the model learns to generate text that closely resembles the patterns in its training data. + +Researchers often use the term "parameters" instead of "weights" to emphasize that both weights and biases play a crucial role in the model's learning process. + +## What are some downsides of _fine-tuning_ LLMs? + +- **Fine-tuning** requires manually creating tons of data, and the model may memorize portions of the data you provide. +- Every time you want to add a new capability, instead of adding a few lines to a prompt, you need to create a bunch of fake data and then run the finetune process and then use the newly fine-tuned model. +- It is more expensive. + +## What is the difference between _Word Embedding_, _Position Embedding_ and _Positional Encoding_ in BERT? + +- A **_Word embedding_** is a learned lookup map i.e. _every word_ is given a _one hot encoding_ which then functions as an _index_, and the corresponding to this index is a `n` _dimensional vector_ where the coefficients are learned when training the model. +- A **Positional embedding** is similar to a **_word embedding_**. Except it is the _position_ in the _sentence_ is used as the index, rather than the one hot encoding. Those positions then correspond to a `n`\-vector whose coefficients come from the trained model. +- On the other hand, the **Positional encoding** is a **_static function_** that maps an integer input to real-valued vectors in a way that captures the inherent relationships among the positions. That is, it captures the fact that position `4` in input is more closely related to position `5` than it is to position `17`. In other words, **positional encoding** is not learned but a chosen mathematical function, $\mathbb{N}\rightarrow\mathbb{R}^n$. + +## What’s the difference between _Feature-based Transfer Learning_ vs. _Fine Tuning_ in LLMs? + +- In **Feature-based Transfer Learning**, you can train _word embeddings_ by running a **model `A`** and then using those features from model `A` (i.e. _word vectors_) on a **different task**, or model `B`. +- When **Fine Tuning**, you can use the exact **same model `A`** and just run it on a **different task**. Sometimes when fine-tuning, you can keep the model weights fixed and just add a new layer that you will train. Other times you can slowly unfreeze the layers one at a time. You can also use unlabelled data when pre-training, by masking words and trying to predict which word was masked. + +![](https://miro.medium.com/v2/resize:fit:828/format:webp/0*n49fDyrzIEeiUXqG.jpeg) + +## Why do transformers need _Positional Encodings_? + +Consider the input sentence - "_I am good_". + +In **RNNs**, we feed the sentence to the network _word by word_. That is, first the word "`I`" is passed as input, next the word "`am`" is passed, and so on. We feed the sentence word by word so that our network understands the sentence completely. + +But with the **transformer network**, we don't follow the recurrence mechanism. So, instead of feeding the sentence word by word, we feed all the words in the sentence **parallel** to the network. Feeding the words in parallel helps in decreasing the training time and also helps in learning the long-term dependency. + +When we feed the words parallel to the transformer, the word order (position of the words in the sentence) is important. So, we should give some information about the word order to the transformer so that it can understand the sentence. + +If we pass the input matrix directly to the transformer, it cannot understand the word order. So, instead of feeding the input matrix directly to the transformer, we need to add some information indicating the word order (position of the word) so that our network can understand the meaning of the sentence. To do this, we introduce a technique called **positional encoding**. Positional encoding, as the name suggests, is an **encoding** indicating the position of the word in a sentence (word order). + +![https://machinelearningmastery.com/wp-content/uploads/2022/01/PE6.png](https://machinelearningmastery.com/wp-content/uploads/2022/01/PE6.png) + +## What's the difference between _Monotonic alignment_ and _Predictive alignment_ in transformers? + +## What is OpenAI GPT model temperature? + +Temperature is a parameter of OpenAI ChatGPT, GPT-3 and GPT-4 models that governs the randomness and thus the creativity of the responses. + +It is always a number between 0 and 1. A temperature of 0 means the responses will be very straightforward, almost deterministic (meaning you almost always get the same response to a given prompt) A temperature of 1 means the responses can vary wildly. + +### How exactly does the temperature work? + +Under the hood, large language models try to predict the next best word given a prompt. One word at a time. They assign a probability to each word in their vocabulary, and then picks a word among those. + +A temperature of 0 means roughly that the model will always select the highest probability word. A higher temperature means that the model might select a word with slightly lower probability, leading to more variation, randomness and creativity. A very high temperature therefore increases the risk of “hallucination”, meaning that the AI starts selecting words that will make no sense or be offtopic. + +Since all characters count, the ratio of words to tokens is language dependent. + +### Rules of thumb for temperature choice + +Your choice of temperature should depend on the task you are giving GPT. + +For transformation tasks (extraction, standardization, format conversion, grammar fixes) prefer a temperature of 0 or up to 0.3. For writing tasks, you should juice the temperature higher, closer to 0.5. If you want GPT to be highly creative (for marketing or advertising copy for instance), consider values between 0.7 and 1. + +If you want to experiment and create many variations quickly, a high temperature is better. + +[How to use OpenAI GPT-3 temperature • Gptforwork.com](https://gptforwork.com/guides/openai-gpt3-temperature) + +## Links + +[MLStack.Cafe - Kill Your Next Machine Learning, Data Science & Python Interview. Find your next ML Job.](https://www.mlstack.cafe/blog/large-language-models-llms-interview-questions) + +[Top 20 Interview Questions & Answers on LLMs & Generative AI | by amirsina torfi | Machine Learning Mindset | Nov, 2023 | Medium](https://medium.com/machine-learning-mindset/top-20-interview-questions-answers-on-llms-generative-ai-42a2e16cb276) diff --git a/docs/ai/llm/llm-building.md b/docs/ai/llm/llm-building.md index 62f00281b09..cad1a2a8e1e 100644 --- a/docs/ai/llm/llm-building.md +++ b/docs/ai/llm/llm-building.md @@ -96,28 +96,30 @@ chainlit hello chainlit run document_qa.py ``` -### HuggingFace +### Ollama -#### About +The easiest way to get up and running with large language models locally. -[How to choose a Sentence Transformer from Hugging Face | Weaviate - Vector Database](https://weaviate.io/blog/how-to-choose-a-sentence-transformer-from-hugging-face) +```bash +docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama -- Blue - the **dataset** it was trained on -- Green - the **language** of the dataset -- White or Purple - **additional details** about the model +docker exec -it ollama ollama run llama2 -#### Models +docker exec -it ollama ollama run llama2-uncensored -- [GitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.](https://github.com/huggingface/transformers) -- [Hugging Face – The AI community building the future.](https://huggingface.co/) +docker exec -it ollama ollama run mistral -## Evaluation +>>> /? # for help +``` -- [Alpaca Eval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) -- [GitHub - tatsu-lab/alpaca\_eval: An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.](https://github.com/tatsu-lab/alpaca_eval) +[Docker](https://hub.docker.com/r/ollama/ollama) ## Resources +[Development with Large Language Models Tutorial – OpenAI, Langchain, Agents, Chroma - YouTube](https://www.youtube.com/watch?v=xZDB1naRUlk&t=1032s) + +![document-based-question-answering-system](../../media/Screenshot%202023-12-28%20at%208.00.43.PM.jpg) + - [GitHub - openai/openai-cookbook: Examples and guides for using the OpenAI API](https://github.com/openai/openai-cookbook) -- [Development with Large Language Models Tutorial – OpenAI, Langchain, Agents, Chroma - YouTube](https://www.youtube.com/watch?v=xZDB1naRUlk&t=1032s) - [Vector Embeddings Tutorial – Create an AI Assistant with GPT-4 & Natural Language Processing - YouTube](https://www.youtube.com/watch?v=yfHHvmaMkcA) +- [This new AI is powerful and uncensored… Let’s run it - YouTube](https://www.youtube.com/watch?v=GyllRd2E6fg&ab_channel=Fireship) diff --git a/docs/ai/llm/models.md b/docs/ai/llm/models.md new file mode 100644 index 00000000000..fdde500c639 --- /dev/null +++ b/docs/ai/llm/models.md @@ -0,0 +1,56 @@ +# Models + +- [openai/whisper-large-v3 · Hugging Face](https://huggingface.co/openai/whisper-large-v3) + - Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. + - [GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision](https://github.com/openai/whisper) +- Vicuna +- Bloom + +| Model | Parameters | Size | +| ------------------ | ---------- | ----- | +| Llama 2 | 7B | 3.8GB | +| Mistral | 7B | 4.1GB | +| Phi-2 | 2.7B | 1.7GB | +| Neural Chat | 7B | 4.1GB | +| Starling | 7B | 4.1GB | +| Code Llama | 7B | 3.8GB | +| [Llama 2 Uncensored](https://huggingface.co/georgesung/llama2_7b_chat_uncensored) | 7B | 3.8GB | +| Llama 2 13B | 13B | 7.3GB | +| Llama 2 70B | 70B | 39GB | +| Orca Mini | 3B | 1.9GB | +| Vicuna | 7B | 3.8GB | +| LLaVA | 7B | 4.5GB | + +Note: You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models. + +- [dolphin-mixtral-8x7b](https://erichartford.com/dolphin-25-mixtral-8x7b) +- [Ollama Library](https://ollama.ai/library) +- [Uncensored Models](https://erichartford.com/uncensored-models) + - aligned by an alignment team + - Remove refusals + +### HuggingFace + +#### About + +[How to choose a Sentence Transformer from Hugging Face | Weaviate - Vector Database](https://weaviate.io/blog/how-to-choose-a-sentence-transformer-from-hugging-face) + +- Blue - the **dataset** it was trained on +- Green - the **language** of the dataset +- White or Purple - **additional details** about the model + +#### Transformer Models + +- [GitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.](https://github.com/huggingface/transformers) +- [Hugging Face – The AI community building the future.](https://huggingface.co/) +- [sentence-transformers/all-MiniLM-L6-v2 · Hugging Face](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) + +## Evaluation + +- [Hugging Face Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) +- [Alpaca Eval Leaderboard](https://tatsu-lab.github.io/alpaca_eval/) +- [GitHub - tatsu-lab/alpaca\_eval: An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.](https://github.com/tatsu-lab/alpaca_eval) + +## Links + +[Should You Use Open Source Large Language Models? - YouTube](https://www.youtube.com/watch?v=y9k-U9AuDeM&ab_channel=IBMTechnology) diff --git a/docs/ai/llm/rag-retrieval-augmented-generation.md b/docs/ai/llm/rag-retrieval-augmented-generation.md index f73edd7f49c..7c7fc120b75 100644 --- a/docs/ai/llm/rag-retrieval-augmented-generation.md +++ b/docs/ai/llm/rag-retrieval-augmented-generation.md @@ -9,3 +9,5 @@ RAG is an AI framework for retrieving facts from an external knowledge base to g [What is Retrieval-Augmented Generation (RAG)? - YouTube](https://youtu.be/T-D1OfcDW1M?si=KoUb-NXATK50d3i7) [**Vector Search RAG Tutorial – Combine Your Data with LLMs with Advanced Search - YouTube**](https://www.youtube.com/watch?v=JEBDfGqrAUA) + +[GitHub - beaucarnes/vector-search-tutorial](https://github.com/beaucarnes/vector-search-tutorial) diff --git a/docs/ai/llm/readme.md b/docs/ai/llm/readme.md index c6c95f7648d..385908ee112 100644 --- a/docs/ai/llm/readme.md +++ b/docs/ai/llm/readme.md @@ -3,7 +3,9 @@ - [LLM Building](ai/llm/llm-building.md) - [Design Patterns](ai/llm/design-patterns.md) - [RAG Retrieval Augmented Generation](ai/llm/rag-retrieval-augmented-generation.md) +- [Models](ai/llm/models.md) - [ChatGPT Prompt Engineering](ai/courses/chatgpt-prompt-eng.md) +- [Interview Questions](ai/llm/interview-questions.md) MMLU - Massive Multitask Language Understanding @@ -76,6 +78,7 @@ Moving from information to knowledge age - [Generative AI in Jupyter. Jupyter AI, a new open source project… | by Jason Weill | Aug, 2023 | Jupyter Blog](https://blog.jupyter.org/generative-ai-in-jupyter-3f7174824862) - [Mem: Your AI knowledge assistant](https://get.mem.ai/) - [GitHub - Nutlope/aicommits: A CLI that writes your git commit messages for you with AI](https://github.com/Nutlope/aicommits) +- [ShareGPT: Share your wildest ChatGPT conversations with one click.](https://sharegpt.com/) ![ai-tools](../../media/image.png) @@ -100,13 +103,6 @@ Moving from information to knowledge age - Amazon EC2 P4d/P4de instances - Powered by NVIDIA A100 Tensor Core GPUs - Amazon EC2 G5 instances - Powered by NVIDIA A10G Tensor Core GPUs -## Models - -- [openai/whisper-large-v3 · Hugging Face](https://huggingface.co/openai/whisper-large-v3) - - Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. - - [GitHub - openai/whisper: Robust Speech Recognition via Large-Scale Weak Supervision](https://github.com/openai/whisper) -- [sentence-transformers/all-MiniLM-L6-v2 · Hugging Face](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) - ## Links - [ChatGPT Prompt Engineering for Developers](ai/courses/chatgpt-prompt-eng.md) @@ -121,3 +117,6 @@ Moving from information to knowledge age - [The True Cost of Compute - YouTube](https://www.youtube.com/watch?v=MNFeJNUu074) - [Welcome to State of AI Report 2023](https://www.stateof.ai/) - [State of AI Report 2023 - ONLINE - Google Slides](https://docs.google.com/presentation/d/156WpBF_rGvf4Ecg19oM1fyR51g4FAmHV3Zs0WLukrLQ/edit) +- [[#29] AI Evolution: From AlexNet to Generative AI – Redefining the Paradigm of Software Development](https://bizit.substack.com/p/29-ai-evolution-from-alexnet-to-generative) +- [Google "We Have No Moat, And Neither Does OpenAI"](https://www.semianalysis.com/p/google-we-have-no-moat-and-neither) +- [AGI-Proof Jobs: Navigating the Impending Obsolescence of Human Labor in the Age of AGI - YouTube](https://www.youtube.com/watch?v=Ahh92qtRwos&ab_channel=DavidShapiro) diff --git a/docs/ai/readme.md b/docs/ai/readme.md index 9c83afc4216..1ba0c75d7f6 100755 --- a/docs/ai/readme.md +++ b/docs/ai/readme.md @@ -18,6 +18,8 @@ ## AGI (Artificial General Intelligence) +AGI will be intellectually, morally, ethically and creatively superior to humans in every conceivable way + [Ben Goertzel: Singularity, Sparks of AGI in GPT - YouTube](https://www.youtube.com/watch?v=27zHyw_oHSI) [Ben Goertzel - Open Ended vs Closed Minded Conceptions of Superintelligence - YouTube](https://www.youtube.com/watch?v=4mEsJ3z6T_E) diff --git a/docs/algorithms/complex-systems/game-theory.md b/docs/algorithms/complex-systems/game-theory.md index 56ef5039387..6a3234f3267 100755 --- a/docs/algorithms/complex-systems/game-theory.md +++ b/docs/algorithms/complex-systems/game-theory.md @@ -110,3 +110,5 @@ https://www.freecodecamp.org/news/introduction-to-evolutionary-game-theory [Simulating the Evolution of Teamwork - YouTube](https://youtu.be/TZfh8hpJIxo?si=-7deIyM-Ub4MR2ic) ![](../../media/Screenshot%202023-12-24%20at%202.49.25.PM.jpg) + +[Round-robin tournament - Wikipedia](https://en.wikipedia.org/wiki/Round-robin_tournament) diff --git a/docs/book-summaries/the-art-of-impossible.md b/docs/book-summaries/the-art-of-impossible.md index a44bb41c929..dc31f6329bd 100755 --- a/docs/book-summaries/the-art-of-impossible.md +++ b/docs/book-summaries/the-art-of-impossible.md @@ -18,7 +18,7 @@ Drive, or motivation, nudges us toward action. But what creates drive in the fir But drive doesn't consist of just one thing. There are various drivers, like fear, curiosity, and passion. All of these provide psychological fuel --⁠ and you can harness them to achieve the impossible. -All psychological drivers are divided into one of two categories:**extrinsicorintrinsic**. Extrinsic drivers are things like **money, fame, food, and sex.** They're external to ourselves, and we chase them to win the evolutionary survival game. Intrinsic drivers, on the other hand, are within us. They're psychological and emotional concepts like curiosity, passion, meaning, and purpose. +All psychological drivers are divided into one of two categories:**extrinsic or intrinsic**. Extrinsic drivers are things like **money, fame, food, and sex.** They're external to ourselves, and we chase them to win the evolutionary survival game. Intrinsic drivers, on the other hand, are within us. They're psychological and emotional concepts like curiosity, passion, meaning, and purpose. Psychological research tells us that extrinsic drivers only motivate us to a certain extent. They get weaker once we have enough money to pay for food, clothing, and shelter. After we've met these basic needs, it's the intrinsic motivators that take over. It's those intrinsic drivers we'll be focusing on in these blinks. @@ -46,7 +46,7 @@ As you complete goals, your brain rewards your persistence with dopamine. Grit w ## Growth mindset -## Use the five-step process to learn almost anything +### Use the five-step process to learn almost anything - Reading - Notes @@ -92,6 +92,6 @@ It's no wonder that flow makes us feel like we can achieve the impossible. So ho ## Final summary -## Follow the 80/20 rule +### Follow the 80/20 rule When you're trying to master a skill, try the 80/20 approach popularized by the author and investor Tim Ferriss. The idea is that 80 percent of your results stem from 20 percent of your actions. Say you want to learn to play a new musical instrument. Most pop songs consist of just four or five chords, so mastering those chords alone can take you a long way. When you're trying to master a skill, be sure to focus on the 20 percent that really matters. diff --git a/docs/cloud/aws/security-identity-compliance/iam.md b/docs/cloud/aws/security-identity-compliance/iam.md index 85b980ee517..e04a6960e81 100755 --- a/docs/cloud/aws/security-identity-compliance/iam.md +++ b/docs/cloud/aws/security-identity-compliance/iam.md @@ -175,7 +175,7 @@ https://www.freecodecamp.org/news/aws-iam-explained ## IAM > Roles > Identity Providers and Federation -If you already manage user identities outside of AWS, you can use IAMidentity providersinstead of creating IAM users in your AWS account. With an identity provider (IdP), you can manage your user identities outside of AWS and give these external user identities permissions to use AWS resources in your account. This is useful if your organization already has its own identity system, such as a corporate user directory. It is also useful if you are creating a mobile app or web application that requires access to AWS resources. +If you already manage user identities outside of AWS, you can use IAM identity providers instead of creating IAM users in your AWS account. With an identity provider (IdP), you can manage your user identities outside of AWS and give these external user identities permissions to use AWS resources in your account. This is useful if your organization already has its own identity system, such as a corporate user directory. It is also useful if you are creating a mobile app or web application that requires access to AWS resources. When you use an IAM identity provider, you don't have to create custom sign-in code or manage your own user identities. The IdP provides that for you. Your external users sign in through a well-known IdP, such as Login with Amazon, Facebook, or Google. You can give those external identities permissions to use AWS resources in your account. IAM identity providers help keep your AWS account secure because you don't have to distribute or embed long-term security credentials, such as access keys, in your application. diff --git a/docs/cloud/aws/security-identity-compliance/intro.md b/docs/cloud/aws/security-identity-compliance/intro.md index efdd5c59363..7f6e8d5cbb2 100755 --- a/docs/cloud/aws/security-identity-compliance/intro.md +++ b/docs/cloud/aws/security-identity-compliance/intro.md @@ -96,7 +96,18 @@ https://github.com/nccgroup/ScoutSuite #### Running scoutsuite (Download scoutsuite repo) -`python scout.py aws --profile zen` +```bash +python scout.py aws --profile zen + +# View a summary of image vulnerabilities and recommendations +docker scout quickview clickhouse/clickhouse-server + +# View vulnerabilities +docker scout cves clickhouse/clickhouse-server + +# View base image update recommendations +docker scout recommendations clickhouse/clickhouse-server +``` https://github.com/tensult/cloud-reports @@ -133,6 +144,22 @@ SOC 2 compliance may help organizations avoid data breaches and the financial an SOC 2, aka **Service Organization Control Type 2**, is a cybersecurity compliance framework developed by the American Institute of Certified Public Accountants (AICPA). The primary purpose of SOC 2 is to ensure that third-party service providers store and process client data in a secure manner. +#### SOC 2 Type 1 vs Type 2 + +There are two main types of SOC 2 compliance: _Type 1_ and _Type 2_. + +Type 1 attests an organization’s use of compliant systems and processes at a _specific point in time._ Conversely, Type 2 is an attestation of compliance over a period (usually 12 months). + +A Type 1 report describes the controls in use by an organization, and confirms that the controls are properly designed and enforced. A Type 2 report includes everything that’s part of a Type 1 report, along with the attestation that the controls are _operationally effective_. + +#### SOC 1 vs SOC 2 vs SOC 3 + +There are three main types of SOC reports – SOC 1, SOC 2, and SOC 3. The first two are the most prevalent, with the second being most relevant to technology companies. + +SOC 1 revolves around financial reporting, whereas SOC 2 focuses more on compliance and business operations. SOC 3 is an adaptation of SOC 2, which reports SOC 2 results in a format that is understandable for the general public. + +[What is SOC 2: Principles, Types, Benefits | OneLogin](https://www.onelogin.com/learn/what-is-soc-2) + ## Others Open-source SAST tools such as [**Semgrep**](https://www.jit.io/blog/semgrep-to-uncover-log4j-vulnerabilities), Bandit, or  KICS can help you find vulnerabilities and compliance issues in your code. diff --git a/docs/cloud/others/firebase/readme.md b/docs/cloud/others/firebase/readme.md index 298a4bfc2c8..d22c9deb52f 100755 --- a/docs/cloud/others/firebase/readme.md +++ b/docs/cloud/others/firebase/readme.md @@ -98,8 +98,6 @@ With Remote Config personalization, you can automatically select Remote Config p - Firebase **AppCheck** - https://neilpatel.com/blog/google-analytics-4 - https://github.com/supabase/supabase -- [Learn Supabase, an Open-Source Firebase Alternative](https://www.freecodecamp.org/news/learn-supabase-open-source-firebase-alternative/) - -Firebase Open Source Alternative - + - Firebase Open Source Alternative + - [Learn Supabase, an Open-Source Firebase Alternative](https://www.freecodecamp.org/news/learn-supabase-open-source-firebase-alternative/) - https://counter.dev diff --git a/docs/computer-science/general/newsletter-learning-resources.md b/docs/computer-science/general/newsletter-learning-resources.md index 490a4dece87..75d86b590d3 100755 --- a/docs/computer-science/general/newsletter-learning-resources.md +++ b/docs/computer-science/general/newsletter-learning-resources.md @@ -93,6 +93,7 @@ Give tech talk - Learning + Note taking + Zettlekasten - Obsidian / Onenote + Do - AWS Architecture - Slack Engineering - Stripe Blog +- [White papers that are worth your time | Interview Ready](https://interviewready.io/blog/white-papers-worth-reading-for-software-engineers) ## Repositories diff --git a/docs/computer-science/general/standards.md b/docs/computer-science/general/standards.md index 1727a4f92c3..bdc5a6e02aa 100755 --- a/docs/computer-science/general/standards.md +++ b/docs/computer-science/general/standards.md @@ -8,13 +8,17 @@ ## base64 -In [computer science](https://en.wikipedia.org/wiki/Computer_science),Base64 is a group of [binary-to-text encoding](https://en.wikipedia.org/wiki/Binary-to-text_encoding) schemes that represent [binary data](https://en.wikipedia.org/wiki/Binary_data) in an [ASCII](https://en.wikipedia.org/wiki/ASCII) string format by translating it into a [radix](https://en.wikipedia.org/wiki/Radix)-64 representation. The term Base64 originates from a specific [MIME content transfer encoding](https://en.wikipedia.org/wiki/MIME#Content-Transfer-Encoding). Each Base64 digit represents exactly 6 bits of data. Three 8-bit bytes (i.e., a total of 24 bits) can therefore be represented by four 6-bit Base64 digits. +In [computer science](https://en.wikipedia.org/wiki/Computer_science), Base64 is a group of [binary-to-text encoding](https://en.wikipedia.org/wiki/Binary-to-text_encoding) schemes that represent [binary data](https://en.wikipedia.org/wiki/Binary_data) in an [ASCII](https://en.wikipedia.org/wiki/ASCII) string format by translating it into a [radix](https://en.wikipedia.org/wiki/Radix)-64 representation. The term Base64 originates from a specific [MIME content transfer encoding](https://en.wikipedia.org/wiki/MIME#Content-Transfer-Encoding). Each Base64 digit represents exactly 6 bits of data. Three 8-bit bytes (i.e., a total of 24 bits) can therefore be represented by four 6-bit Base64 digits. Common to all binary-to-text encoding schemes, Base64 is designed to carry data stored in binary formats across channels that only reliably support text content. Base64 is particularly prevalent on the World Wide Web its uses include the ability to embed [image files](https://en.wikipedia.org/wiki/Image_files) or other binary assets inside textual assets such as [HTML](https://en.wikipedia.org/wiki/HTML) and [CSS](https://en.wikipedia.org/wiki/CSS) files. The difference between Base64 and hex is really just how bytes are represented. Hex is another way of saying "Base16". **Hex will take two characters for each byte - Base64 takes 4 characters for every 3 bytes, so it's more efficient than hex.** Assuming you're using UTF-8 to encode the XML document, a 100K file will take 200K to encode in hex, or 133K in Base64. -https://en.wikipedia.org/wiki/Base64 +base64 is a costly instrument. It makes data about **33% larger** in terms of memory usage. So base64 is one of these little things that make software **slow**. That's why you should use it only when it's absolutely necessary. + +[Base64 - Wikipedia](https://en.wikipedia.org/wiki/Base64) + +[What is base64 Encoding and Why is it Necessary?](https://www.freecodecamp.org/news/what-is-base64-encoding/) ## Base64 vs UTF-8/UTF-16 diff --git a/docs/computer-science/links.md b/docs/computer-science/links.md index 57f44725d26..0ae3df4a2f1 100644 --- a/docs/computer-science/links.md +++ b/docs/computer-science/links.md @@ -98,6 +98,8 @@ 94. [Designing A Data-Intensive Future: Expert Talk • Martin Kleppmann & Jess...](https://youtu.be/P-9FwZxO1zE?si=WyoFSrXD_IR6faH6) 95. [How Does Linux Boot Process Work?](https://youtu.be/XpFsMB6FoOs?si=BMNajLZbSLdmU7Iz) 96. [It's a Noisy World Out There • Linda Rising • GOTO 2023](https://youtu.be/k8LxUKx85X0?si=nCXx1UvtJr7zWPwL) +97. [AI’s Hottest New Job Pays Up to $250K a Year. So I Applied. | WSJ - YouTube](https://www.youtube.com/watch?v=whkge1rEamU&ab_channel=TheWallStreetJournal) +98. [Technologies I'm learning to Make More Money in 2024 - YouTube](https://www.youtube.com/watch?v=C_qhRtuxZzU&ab_channel=HarkiratSingh) ## Podcasts diff --git a/docs/computer-science/others/computer-graphics.md b/docs/computer-science/others/computer-graphics.md index 98f1da80a12..139fae3fbc8 100755 --- a/docs/computer-science/others/computer-graphics.md +++ b/docs/computer-science/others/computer-graphics.md @@ -1,37 +1,31 @@ # Computer Graphics -1. Rasterization +1. Vertex Shading +2. Rasterization +3. Fragment Shading +4. Ray Tracing +5. Shader program +6. Subsurface scattering +7. BVH Tree Traversals +8. Ray triangle intersection +9. Binning +10. Bounding Voluming Hierarchy -2. Ray Tracing - -3. Shader program - -4. Subsurface scattering - -5. BVH Tree Traversals +[TU Wien Rendering / Ray Tracing Course](https://www.youtube.com/playlist?list=PLujxSBD-JXgnGmsn7gEyN28P1DnRZG7qi) (2 Minute Papers) -6. Ray triangle intersection +[How do Video Game Graphics Work? - YouTube](https://youtu.be/C8YtdC8mxTU?si=QHa-SGSlaXNevkSB) -7. Binning +![video-game-graphics](../../media/Screenshot%202023-12-26%20at%2012.43.35.PM.jpg) -8. Bounding Voluming Hierarchy -[TU Wien Rendering / Ray Tracing Course](https://www.youtube.com/playlist?list=PLujxSBD-JXgnGmsn7gEyN28P1DnRZG7qi) (2 Minute Papers) -Pipeline +### Pipeline 1. Deferred shading - 2. Direct shadows - 3. Lighting - 4. Reflections - 5. Global Illuminations - 6. Ambient Occlusion - 7. Transparency and Transluency - 8. Post processing ## OpenGL diff --git a/docs/computer-science/security/firewall-waf.md b/docs/computer-science/security/firewall-waf.md index 5d5931b8ade..48f46a98a00 100755 --- a/docs/computer-science/security/firewall-waf.md +++ b/docs/computer-science/security/firewall-waf.md @@ -38,3 +38,19 @@ https://developers.cloudflare.com/firewall/cf-firewall-rules/actions DDoS blackhole routing/filtering (sometimes called blackholing), is a countermeasure to mitigate a [DDoS attack](https://www.cloudflare.com/learning/ddos/what-is-a-ddos-attack/) in which network traffic is routed into a "black hole," and is lost. When blackhole filtering is implemented without specific restriction criteria, both legitimate and malicious network traffic is routed to a null route or black hole and dropped from the network. When using protocols that are connectionless such as [UDP](https://www.cloudflare.com/learning/ddos/glossary/user-datagram-protocol-udp/), no notification of the dropped data will be returned to the source. With connection oriented protocols like [TCP](https://www.cloudflare.com/learning/ddos/glossary/tcp-ip/), which require a handshake to connect with the target system, a notification will be returned if the data is dropped. https://www.freecodecamp.org/news/protect-against-ddos-attacks + +## Intrusion Detection System + +### OSSEC + +**OSSEC (Open Source HIDS SECurity)** is a [free](https://en.wikipedia.org/wiki/Free_software "Free software"), [open-source](https://en.wikipedia.org/wiki/Open-source_software "Open-source software") [host-based intrusion detection system](https://en.wikipedia.org/wiki/Host-based_intrusion_detection_system) (HIDS). It performs [log analysis](https://en.wikipedia.org/wiki/Log_analysis "Log analysis"), integrity checking, [Windows registry](https://en.wikipedia.org/wiki/Windows_registry "Windows registry") monitoring, [rootkit](https://en.wikipedia.org/wiki/Rootkit "Rootkit") detection, time-based alerting, and active response. It provides intrusion detection for most operating systems, including [Linux](https://en.wikipedia.org/wiki/Linux "Linux"), [OpenBSD](https://en.wikipedia.org/wiki/OpenBSD "OpenBSD"), [FreeBSD](https://en.wikipedia.org/wiki/FreeBSD "FreeBSD"), [OS X](https://en.wikipedia.org/wiki/OS_X "OS X"), [Solaris](https://en.wikipedia.org/wiki/Solaris_(operating_system) "Solaris (operating system)") and [Windows](https://en.wikipedia.org/wiki/Microsoft_Windows "Microsoft Windows"). OSSEC has a centralized, cross-platform architecture allowing multiple systems to be easily monitored and managed. OSSEC has a log analysis engine that is able to correlate and analyze logs from multiple devices and formats. + +[OSSEC - Wikipedia](https://en.wikipedia.org/wiki/OSSEC) + +### Wazuh + +Wazuh is a free and open source security platform that unifies XDR and SIEM capabilities. It protects workloads across on-premises, virtualized, containerized, and cloud-based environments. + +[Wazuh - Open Source XDR. Open Source SIEM.](https://wazuh.com/) + +[Getting started with Wazuh · Wazuh documentation](https://documentation.wazuh.com/current/getting-started/index.html) diff --git a/docs/computer-science/system-design/api-gateway.md b/docs/computer-science/system-design/api-gateway.md index 33ba77d1e52..ba2a6d32acd 100755 --- a/docs/computer-science/system-design/api-gateway.md +++ b/docs/computer-science/system-design/api-gateway.md @@ -23,6 +23,8 @@ 15. Rate Limiting 16. Expose as API service +![api-gateway](../../media/Pasted%20image%2020231224193859.png) + ## Using an API gateway has the following benefits - Insulates the clients from how the application is partitioned into microservices diff --git a/docs/computer-science/system-design/intro.md b/docs/computer-science/system-design/intro.md index 1e9ce039c86..14c52c9cff9 100755 --- a/docs/computer-science/system-design/intro.md +++ b/docs/computer-science/system-design/intro.md @@ -20,9 +20,11 @@ Systems design is the process of defining the [architecture](https://en.wikipedi 1. Features 2. Define APIs 3. CAP Theorem - - Consistency - - Availability, and - - Partition Tolerance + +- Consistency +- Availability, and +- Partition Tolerance + 4. Latency Performance (If customer facing application, then latency matters) 5. Scalability (Add more users and requests) 6. Durability (Data is not lost or compromised) diff --git a/docs/computer-science/system-design/message-queues.md b/docs/computer-science/system-design/message-queues.md new file mode 100644 index 00000000000..7df1e53ee14 --- /dev/null +++ b/docs/computer-science/system-design/message-queues.md @@ -0,0 +1,32 @@ +# Message Queues + +`IBM MQ -> RabbitMQ -> Kafka -> Pulsar`, How do message queue architectures evolve? + +## IBM MQ + +IBM MQ was launched in 1993. It was originally called MQSeries and was renamed WebSphere MQ in 2002. It was renamed to IBM MQ in 2014. IBM MQ is a very successful product widely used in the financial sector. Its revenue still reached 1 billion dollars in 2020. + +## RabbitMQ + +RabbitMQ architecture differs from IBM MQ and is more similar to Kafka concepts. The producer publishes a message to an exchange with a specified exchange type. It can be direct, topic, or fanout. The exchange then routes the message into the queues based on different message attributes and the exchange type. The consumers pick up the message accordingly. + +## Kafka + +In early 2011, LinkedIn open sourced Kafka, which is a distributed event streaming platform. It was named after Franz Kafka. As the name suggested, Kafka is optimized for writing. It offers a high-throughput, low-latency platform for handling real-time data feeds. It provides a unified event log to enable event streaming and is widely used in internet companies. + +Kafka defines producer, broker, topic, partition, and consumer. Its simplicity and fault tolerance allow it to replace previous products like AMQP-based message queues. + +## Pulsar + +Pulsar, developed originally by Yahoo, is an all-in-one messaging and streaming platform. Compared with Kafka, Pulsar incorporates many useful features from other products and supports a wide range of capabilities. Also, Pulsar architecture is more cloud-native, providing better support for cluster scaling and partition migration, etc. + +There are two layers in Pulsar architecture: the serving layer and the persistent layer. Pulsar natively supports tiered storage, where we can leverage cheaper object storage like AWS S3 to persist messages for a longer term. + +![message-queues](../../media/Pasted%20image%2020231225102216.png) + +## Others + +- [BullMQ - libraries](frontend/others/nodejs/libraries.md) +- [Redis Streams](databases/nosql-databases/redis/redis-streams-pubsub.md) +- [AWS SQS](cloud/aws/aws-sqs.md) +- [Publish Subscribe Pattern](networking/mqtt/publish-subscribe-pattern.md) diff --git a/docs/databases/concepts/isolation-levels.md b/docs/databases/concepts/isolation-levels.md index 958fb6d3a16..7bc30282e0f 100755 --- a/docs/databases/concepts/isolation-levels.md +++ b/docs/databases/concepts/isolation-levels.md @@ -165,3 +165,5 @@ https://dev.mysql.com/doc/refman/5.7/en/innodb-transaction-isolation-levels.html https://dbmsmusings.blogspot.com/2019/08/an-explanation-of-difference-between.html http://dbmsmusings.blogspot.com/2019/10/introducing-slog-cheating-low-latency.html + +[Transaction Isolation in Postgres, explained](https://www.thenile.dev/blog/transaction-isolation-postgres) diff --git a/docs/databases/concepts/types-of-databases.md b/docs/databases/concepts/types-of-databases.md index 2f4275f1ecb..776ee5ebf1f 100755 --- a/docs/databases/concepts/types-of-databases.md +++ b/docs/databases/concepts/types-of-databases.md @@ -180,6 +180,8 @@ Vector databases are designed to handle critical query and algorithmic styles se [A Comprehensive Guide to Vector Databases - KDB.AI](https://kdb.ai/blog/a-comprehensive-guide-to-vector-databases/) +- pinecone + ## Others ### Chroma diff --git a/docs/databases/nosql-databases/clickhouse.md b/docs/databases/nosql-databases/clickhouse.md index e2b985eb2a6..d956fc00421 100644 --- a/docs/databases/nosql-databases/clickhouse.md +++ b/docs/databases/nosql-databases/clickhouse.md @@ -130,6 +130,66 @@ ClickHouse supports both virtual views and materialized views. The materialized [Modern SQL in 2023 - ClickHouse - YouTube](https://www.youtube.com/watch?v=zhrOYQpgvkk) +["Building for Fast" by Alexey Milovidov, Amsterdam, June 2022 - YouTube](https://www.youtube.com/watch?v=CAS2otEoerM&ab_channel=ClickHouse) + +[Why is ClickHouse so fast? | ClickHouse Docs](https://clickhouse.com/docs/en/concepts/why-clickhouse-is-so-fast) + +## Commands + +```bash +docker pull clickhouse/clickhouse-server + +docker run -d -p 8123:8123 -p 9000:9000 --network=test-net --name some-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server + +docker run -d --name some-clickhouse-server -p 8123:8123 -p 9000:9000 --network=test-net -e CLICKHOUSE_DB=my_database -e CLICKHOUSE_USER=username -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 -e CLICKHOUSE_PASSWORD=password --ulimit nofile=262144:262144 clickhouse/clickhouse-server + +# https://hub.docker.com/r/clickhouse/clickhouse-server/ + +echo 'SELECT version()' | curl 'http://localhost:8123/' --data-binary @- +``` + +[Visual Interfaces from Third-party Developers | ClickHouse Docs](https://clickhouse.com/docs/en/interfaces/third-party/gui) + +[GitHub - ClickHouse/metabase-clickhouse-driver: ClickHouse database driver for the Metabase business intelligence front-end](https://github.com/ClickHouse/metabase-clickhouse-driver) + +```sql +-- https://clickhouse.com/docs/en/getting-started/quick-start + +clickhouse client + +CREATE TABLE my_first_table +( + user_id UInt32, + message String, + timestamp DateTime, + metric Float32 +) +ENGINE = MergeTree +PRIMARY KEY (user_id, timestamp) + +INSERT INTO my_first_table (user_id, message, timestamp, metric) VALUES + (101, 'Hello, ClickHouse!', now(), -1.0 ), + (102, 'Insert a lot of rows per batch', yesterday(), 1.41421 ), + (102, 'Sort your data based on your commonly-used queries', today(), 2.718 ), + (101, 'Granules are the smallest chunks of data read', now() + 5, 3.14159 ) + + SELECT * + FROM my_first_table + ORDER BY timestamp + +SELECT + passenger_count, + avg(toFloat32(total_amount)) +FROM s3( + 'https://datasets-documentation.s3.eu-west-3.amazonaws.com/nyc-taxi/trips_0.gz', + 'TabSeparatedWithNames' +) +GROUP BY passenger_count +ORDER BY passenger_count; +``` + +[New York Taxi Data | ClickHouse Docs](https://clickhouse.com/docs/en/getting-started/example-datasets/nyc-taxi) + ## Links [Fast Open-Source OLAP DBMS - ClickHouse](https://clickhouse.com/) @@ -140,4 +200,6 @@ ClickHouse supports both virtual views and materialized views. The materialized [What Is ClickHouse? | ClickHouse Docs](https://clickhouse.com/docs/en/intro) -Used by - Zerodha +[Getting Started with ClickHouse](https://clickhouse.com/company/events/getting-started-with-clickhouse?loc=getting-started) + +Used by - Zerodha, Cloudflare, Uber, Ebay (moved from postgres, elk, druid to clickhouse) diff --git a/docs/decentralized-applications/coins-tokens-chains/readme.md b/docs/decentralized-applications/coins-tokens-chains/readme.md index 691557dbbc3..723ffab29ae 100755 --- a/docs/decentralized-applications/coins-tokens-chains/readme.md +++ b/docs/decentralized-applications/coins-tokens-chains/readme.md @@ -124,6 +124,7 @@ A wrapped token is a tokenized version of another cryptocurrency which is pegged - [What is Terra? UST and LUNA explained with Animations - YouTube](https://www.youtube.com/watch?v=U9lrH0loAns) - [How To Get Started With Constellation Network](https://milkroad.com/invest/constellation-network/) - [Aptos Labs - Opening the gateway to the Aptos Network](https://aptoslabs.com/) + - [Aptos Blockchain Deep Dive | Aptos Docs](https://aptos.dev/concepts/blockchain/) #### Chainlink (Oracle) - LINK diff --git a/docs/decentralized-applications/ethereum/solidity.md b/docs/decentralized-applications/ethereum/solidity.md index 501611809ec..898056b0a81 100755 --- a/docs/decentralized-applications/ethereum/solidity.md +++ b/docs/decentralized-applications/ethereum/solidity.md @@ -87,3 +87,5 @@ You will hack custom smart contracts we deploy to the main net (and get a little ## Others [Slither – a Solidity static analysis framework | Trail of Bits Blog](https://blog.trailofbits.com/2018/10/19/slither-a-solidity-static-analysis-framework/) + +[Introduction - The Move Book](https://move-language.github.io/move/) diff --git a/docs/devops/docker/docker-commands.md b/docs/devops/docker/docker-commands.md index 3b06354d6b4..8dfc933879f 100755 --- a/docs/devops/docker/docker-commands.md +++ b/docs/devops/docker/docker-commands.md @@ -7,9 +7,9 @@ sudo apt-get update sudo apt-get -y install docker.io -~~--add-host="" : Add a line to /etc/hosts (host:IP)~~ +--add-host="" : Add a line to /etc/hosts (host:IP) -https://docs.docker.com/engine/install/ubuntu +# https://docs.docker.com/engine/install/ubuntu sudo service docker start @@ -60,6 +60,10 @@ docker run --rm -it -p=8080:8080 inventree/inventree - [docker kill](https://docs.docker.com/engine/reference/commandline/kill) sends a SIGKILL to a running container - [docker attach](https://docs.docker.com/engine/reference/commandline/attach) will connect to a running container +```bash +docker start **CONTAINER ID** +``` + ## Info - [docker ps](https://docs.docker.com/engine/reference/commandline/ps) shows running containers. @@ -144,30 +148,31 @@ docker tag azure-vote-front gcr.io/example-data-archiver/azure-vote-front:v1 ## Network ```bash -- docker network connect - Connect a container to a network -- docker network create - Create a network +docker network connect - Connect a container to a network +docker network create - Create a network docker network create --subset=172.18.0.0/16 example-docker -- docker network disconnect - Disconnect a container from a network -- docker network inspect - Display detailed information on one or more networks -- docker network ls - List networks -- docker network prune - Remove all unused networks -- docker network rm - Remove one or more networks +docker network disconnect - Disconnect a container from a network +docker network inspect - Display detailed information on one or more networks +docker network ls - List networks +docker network prune - Remove all unused networks +docker network rm - Remove one or more networks ``` ## Volumes ```bash cd /var/lib/docker/volumes/druid-volume/_data/segment-cache +``` -| [docker volume create](https://docs.docker.com/engine/reference/commandline/volume_create/) | Create a volume | +| | | |------------------------|------------------------------------------------| +| [docker volume create](https://docs.docker.com/engine/reference/commandline/volume_create/) | Create a volume | | [docker volume inspect](https://docs.docker.com/engine/reference/commandline/volume_inspect/) | Display detailed information on one or more volumes | | [docker volume ls](https://docs.docker.com/engine/reference/commandline/volume_ls/) | List volumes | | [docker volume prune](https://docs.docker.com/engine/reference/commandline/volume_prune/) | Remove all unused local volumes | | [docker volume rm](https://docs.docker.com/engine/reference/commandline/volume_rm/) | Remove one or more volumes | -``` ## Docker CLI @@ -176,29 +181,31 @@ cd /var/lib/docker/volumes/druid-volume/_data/segment-cache ## Cleanup Commands ```bash -- docker stop $(docker ps -aq) #stop all running containers -- docker rmi $(docker images -q) #Delete all images -- docker rm -f $(docker ps -a -q) #Delete all containers -- docker-compose config #Check if environment variables are loaded in source -- docker ps -aq -f status=exited #Show all stopped containers +docker stop $(docker ps -aq) #stop all running containers +docker rmi $(docker images -q) #Delete all images +docker rm -f $(docker ps -a -q) #Delete all containers +docker-compose config #Check if environment variables are loaded in source +docker ps -aq -f status=exited #Show all stopped containers + +docker rm $(docker ps -a -f status=exited -q) #docker-remove-exited-containers + +docker volume prune #remove docker volumes +docker volume rm $(docker volume ls -f dangling=true -q) #docker-remove-dangling-volumes -- **docker rm $(docker ps -a -f status=exited -q) #docker-remove-exited-containers** +docker images -qf dangling=true | xargs docker rmi #remove all dangling images -- docker volume prune #remove docker volumes -- docker volume rm $(docker volume ls -f dangling=true -q) #docker-remove-dangling-volumes -- docker images -qf dangling=true | xargs docker rmi #remove all dangling images +docker images -q | xargs docker rmi #remove all unused images -- **docker images -q | xargs docker rmi #remove all unused images** -- **docker builder prune -f** +docker builder prune -f -- docker system prune -a #clean all, Can kill container in kubernetes cluster +docker system prune -a #clean all, Can kill container in kubernetes cluster ## Kubernetes Cleanup Commands -- **docker system df #check volume status (docker sizes)** +docker system df #check volume status (docker sizes)** -- docker container prune -- docker image prune -a +docker container prune +docker image prune -a ``` https://github.com/onfido/k8s-cleanup @@ -206,22 +213,24 @@ https://github.com/onfido/k8s-cleanup ## Base Commands ```bash -- docker run --rm -it -v $PWD:/build ubuntu:18.04 #create a docker image of ubuntu:18.04 -- docker run --rm -it ubuntu:18.04 #create a docker image of ubuntu:18.04 -- docker run --rm -it exampletech/react-awscli:1.0.0 /bin/sh -- docker run --rm -it -p=8080:8080 volttron_docker_image /bin/bash -- docker commit suspicious_wescoff volttron_docker_image -- docker run -i -t volttron_docker_image /bin/bash -- Exit a container - CTRL + D -- docker run -it --network="host" --name mynodered nodered/node-red-docker #for binding docker to localhost, published ports doesn't work when --network="host" is used +docker run --rm -it -v $PWD:/build ubuntu:18.04 #create a docker image of ubuntu:18.04 +docker run --rm -it ubuntu:18.04 #create a docker image of ubuntu:18.04 +docker run --rm -it exampletech/react-awscli:1.0.0 /bin/sh +docker run --rm -it -p=8080:8080 volttron_docker_image /bin/bash +docker commit suspicious_wescoff volttron_docker_image +docker run -i -t volttron_docker_image /bin/bash + +Exit a container - CTRL + D + +docker run -it --network="host" --name mynodered nodered/node-red-docker #for binding docker to localhost, published ports doesn't work when --network="host" is used ``` ## Other Commands ```bash -- docker exec -it --user root temp-emqx /bin/sh #get inside docker container as user root -- whoami #get logged in user inside docker container -- sudo systemctl restart docker (When docker gets hanged) +docker exec -it --user root temp-emqx /bin/sh #get inside docker container as user root +whoami #get logged in user inside docker container +sudo systemctl restart docker (When docker gets hanged) ``` ## Scaling diff --git a/docs/devops/ides/mac.md b/docs/devops/ides/mac.md index bef6b4a0171..c7bdaef4a6d 100755 --- a/docs/devops/ides/mac.md +++ b/docs/devops/ides/mac.md @@ -92,21 +92,22 @@ Open two instance of same application - **open -n MQTT.fx.app** #### Shortcuts -| Action | Shortcuts | -|---|---| -| to cycle between tabs | Cmd + Shift + A + Enter | -| Jump to the next open tab | ⌘ + Option + Right arrow or ctrl + tab | -| Jump to the previous open tab | ⌘ + Option + Left arrow or ctrl + shift + tab | -| Open the previous page in your browsing history for the current tab | `⌘ + [ or ⌘ + Left arrow` | -| Open the next page in your browsing history for the current tab | `⌘ + ] or ⌘ + Right arrow` | -| Open the History page in a new tab | ⌘ + y | -| Open the Downloads page in a new tab | ⌘ + Shift + j | -| Open the Find Bar to search the current page | ⌘ + f | -| Jump to the next match to your Find Bar search | ⌘ + g | -| Jump to the previous match to your Find Bar search | ⌘ + Shift + g | -| When Find Bar is open, search for selected text | ⌘ + e | -| Open the JavaScript Console | ⌘ + Option + j | -| Search the web | ⌘ + Option + f | +| Action | Shortcuts | +| ------------------------------------------------------------------- | --------------------------------------------- | +| switch profiles | cmd + shift + m - enter | +| to cycle between tabs | Cmd + Shift + A + Enter | +| Jump to the next open tab | ⌘ + Option + Right arrow or ctrl + tab | +| Jump to the previous open tab | ⌘ + Option + Left arrow or ctrl + shift + tab | +| Open the previous page in your browsing history for the current tab | `⌘ + [ or ⌘ + Left arrow` | +| Open the next page in your browsing history for the current tab | `⌘ + ] or ⌘ + Right arrow` | +| Open the History page in a new tab | ⌘ + y | +| Open the Downloads page in a new tab | ⌘ + Shift + j | +| Open the Find Bar to search the current page | ⌘ + f | +| Jump to the next match to your Find Bar search | ⌘ + g | +| Jump to the previous match to your Find Bar search | ⌘ + Shift + g | +| When Find Bar is open, search for selected text | ⌘ + e | +| Open the JavaScript Console | ⌘ + Option + j | +| Search the web | ⌘ + Option + f | https://www.google.com/inputtools diff --git a/docs/devops/ides/youtube-podcasts-magazine-apps.md b/docs/devops/ides/youtube-podcasts-magazine-apps.md index 41a6f60960b..f93f3ce07a1 100755 --- a/docs/devops/ides/youtube-podcasts-magazine-apps.md +++ b/docs/devops/ides/youtube-podcasts-magazine-apps.md @@ -171,6 +171,7 @@ - RM Transit - The School of life - [Medlife Crisis - YouTube](https://youtube.com/@MedlifeCrisis?si=vmOmOUXV_TZoBNyW) +- [Architectural Digest - YouTube](https://www.youtube.com/@Archdigest) ## Yearly @@ -211,6 +212,7 @@ - Curiosity Stream - Discovery+ - NatGeo +- Nebula - The logistics of x ## General Science @@ -336,6 +338,8 @@ - The Science of Change [https://podcastaddict.com/podcast/the-science-of-change/4496415](https://podcastaddict.com/podcast/the-science-of-change/4496415) - Business Breakdowns [https://podcastaddict.com/podcast/business-breakdowns/4400688](https://podcastaddict.com/podcast/business-breakdowns/4400688) - The rational reminder podcast +- Aquired +- Empire https://www.freecodecamp.org/news/best-tech-podcasts-for-software-developers diff --git a/docs/devops/readme.md b/docs/devops/readme.md index 14c8f3c0608..67ee7a57c7f 100755 --- a/docs/devops/readme.md +++ b/docs/devops/readme.md @@ -7,3 +7,5 @@ - [Terminal Bash](terminal-bash/readme.md) - [IDEs](ides/readme.md) - [Others](others/readme.md) + +[DevOps Roadmap: Learn to become a DevOps Engineer or SRE](https://roadmap.sh/devops) diff --git a/docs/economics/mental-models/economics-mental-models.md b/docs/economics/mental-models/economics-mental-models.md index 4a43f3f0b20..30db68448ca 100755 --- a/docs/economics/mental-models/economics-mental-models.md +++ b/docs/economics/mental-models/economics-mental-models.md @@ -297,6 +297,12 @@ Baumol helps us understand not only today’s economy but also a likely path for The concept refers to the attempts to gain economic benefits without producing new wealth. Rather than creating a better product or offering a more efficient service, the actors in this scenario extracted vast wealth through manipulative practices. +## Jevons paradox + +In economics, the **Jevons paradox** (sometimes **Jevons effect**) occurs when [technological progress](https://en.wikipedia.org/wiki/Technological_change "Technological change") or government policy increases the [efficiency](https://en.wikipedia.org/wiki/Efficiency "Efficiency") with which a [resource](https://en.wikipedia.org/wiki/Resource_(economics) "Resource (economics)") is used (reducing the amount necessary for any one use), but the falling cost of use [induces increases in demand](https://en.wikipedia.org/wiki/Induced_demand) enough that resource use is increased, rather than reduced. Governments typically assume that efficiency gains will lower [resource consumption](https://en.wikipedia.org/wiki/Resource_consumption "Resource consumption"), ignoring the possibility of the paradox arising. + +[Jevons paradox - Wikipedia](https://en.wikipedia.org/wiki/Jevons_paradox) + ## Links - [Marginal Utility / Analysis](marginal-utility-analysis) diff --git a/docs/economics/mutual-funds/equity-mutual-funds.md b/docs/economics/mutual-funds/equity-mutual-funds.md index 93632ef14fd..5d90d01d9b2 100644 --- a/docs/economics/mutual-funds/equity-mutual-funds.md +++ b/docs/economics/mutual-funds/equity-mutual-funds.md @@ -97,3 +97,5 @@ In the multicap category, the allocation decision is left to the fund manager wh - Value fund [List of worst equity mutual funds (Jan 2013 to Dec 2023)](https://freefincal.com/list-of-worst-equity-mutual-funds-jan-2013-to-dec-2023/) + +[Flexi-cap funds vs. focused equity funds: Which is better? | Value Research](https://www.valueresearchonline.com/stories/53730/do-focused-funds-give-better-returns-than-flexi-cap-funds/) diff --git a/docs/frontend/others/nextjs/readme.md b/docs/frontend/others/nextjs/readme.md index 1992fba3340..714153adb14 100755 --- a/docs/frontend/others/nextjs/readme.md +++ b/docs/frontend/others/nextjs/readme.md @@ -114,3 +114,5 @@ https://www.freecodecamp.org/news/nextjs-tutorial [How to Integrate MongoDB Into Your Next.js App | MongoDB](https://www.mongodb.com/developer/languages/javascript/nextjs-with-mongodb/) [Learn Next.js by Building a Custom API with MongoDB](https://www.freecodecamp.org/news/full-stack-with-nextjs-and-appwrite-course/) + +[Full Stack JavaScript Course: Build a Hotel Management Site with Next.js, Sanity.io, and Tailwind CSS](https://www.freecodecamp.org/news/build-and-deploy-a-hotel-management-site/) diff --git a/docs/frontend/others/readme.md b/docs/frontend/others/readme.md index c671130663b..a707870581c 100755 --- a/docs/frontend/others/readme.md +++ b/docs/frontend/others/readme.md @@ -4,14 +4,14 @@ - [Frontend Frameworks](frontend-frameworks) - [Static Site Generators](static-site-generators) - [NextJS](nextjs/readme.md) - - [data-fetching](frontend/others/nextjs/data-fetching.md) - - [others](frontend/others/nextjs/others.md) + - [data-fetching](frontend/others/nextjs/data-fetching.md) + - [others](frontend/others/nextjs/others.md) - [Gatsby](gatsby) - [Docusaurus](docusaurus) - [Nodejs](nodejs/readme.md) - - [Nodejs Frameworks](nodejs/nodejs-frameworks) - - [npm Commands](nodejs/npm-commands) - - [Libraries](nodejs/libraries) + - [Nodejs Frameworks](nodejs/nodejs-frameworks) + - [npm Commands](nodejs/npm-commands) + - [Libraries](nodejs/libraries) - [NestJS](nestjs/readme.md) - - [Documentation](nestjs/documentation) + - [Documentation](nestjs/documentation) - [Angular JS](angularjs/readme.md) diff --git a/docs/knowledge/applied-sciences/chemical-engineering.md b/docs/knowledge/applied-sciences/chemical-engineering.md index 862b38e3b69..90895c52278 100755 --- a/docs/knowledge/applied-sciences/chemical-engineering.md +++ b/docs/knowledge/applied-sciences/chemical-engineering.md @@ -8,3 +8,5 @@ - Polymerization - It's important to think of processes as a whole, but it will be just as important to break down our chemical plant into unit operations when we get to the manufacturing phase - Biomass: renewable orgainic material that comes from plants and animals. This range from wood and leftover crops, to garbage and manure + +[Strychnine - Wikipedia](https://en.wikipedia.org/wiki/Strychnine) diff --git a/docs/knowledge/applied-sciences/civil-engineering.md b/docs/knowledge/applied-sciences/civil-engineering.md index 1a803d5b9e8..c9f9b0b81d0 100755 --- a/docs/knowledge/applied-sciences/civil-engineering.md +++ b/docs/knowledge/applied-sciences/civil-engineering.md @@ -46,6 +46,10 @@ Humans design buildings according to conditions they themselves have created but Habitat - 67 - [How a 23-Year-Old Solved Urban Sprawl - YouTube](https://www.youtube.com/watch?v=Qwbp9T-WS-I) +## Places + +[Doge's Palace - Wikipedia](https://en.wikipedia.org/wiki/Doge%27s_Palace) + ## Links - [Does Rebar Rust? - YouTube](https://www.youtube.com/watch?v=PLF18H9JGHs) @@ -56,3 +60,4 @@ Habitat - 67 - [How a 23-Year-Old Solved Urban Sprawl - YouTube](https://www.you - [Every Construction Machine Explained in 15 Minutes - YouTube](https://www.youtube.com/watch?v=6CmX4ZmhwPM) - [How to Control a Crowd - YouTube](https://www.youtube.com/watch?v=C_B09FZwSbA) - [Rare earths crunch? Why we need them and who has them | Business Beyond - YouTube](https://www.youtube.com/watch?v=cL4YXCKrJJ4) +- [Jackhammer - Wikipedia](https://en.wikipedia.org/wiki/Jackhammer) diff --git a/docs/knowledge/applied-sciences/mechanical-engineering.md b/docs/knowledge/applied-sciences/mechanical-engineering.md index 6969972caf9..9efdcf4951b 100755 --- a/docs/knowledge/applied-sciences/mechanical-engineering.md +++ b/docs/knowledge/applied-sciences/mechanical-engineering.md @@ -228,3 +228,4 @@ Isotropic refers to the properties of a material which is independent of the dir - [How do Mechanical Pencils Work? #shorts - YouTube](https://www.youtube.com/watch?v=vpR8piWPdKo) - [How do Soap Bottle Pumps Work? || Inside Animation of a Soap Pump Dispenser - YouTube](https://www.youtube.com/watch?v=9kzC4CpPxSQ) - [Photoelectric light controls are weirder than they seem - YouTube](https://www.youtube.com/watch?v=Khp3wb0QMpQ) +- [Autoclave - Wikipedia](https://en.m.wikipedia.org/wiki/Autoclave) diff --git a/docs/knowledge/biology/muscles.md b/docs/knowledge/biology/muscles.md index ff3f557ffb6..ac7b9d2f5ff 100755 --- a/docs/knowledge/biology/muscles.md +++ b/docs/knowledge/biology/muscles.md @@ -9,3 +9,5 @@ - **Ligaments** are tough, fibrous bands of tissue that bind bone to bone, or bone to cartilage, at a joint, allowing a safe range of movement. An example is the anterior cruciate ligament (ACL), one of five ligaments that collectively control knee movements. The ACL is responsible for keeping the knee joint from rotating too far or allowing your shin to move out in front of your thigh. As many sports enthusi- asts know, some to their sorrow, the ACL is often injured in sports like soccer and skiing that demand forceful actions involving quick stops or turns. [What actually ARE muscle cramps? - YouTube](https://www.youtube.com/watch?v=WsWPI18RmVU) + +[Deltoid muscle - Wikipedia](https://en.wikipedia.org/wiki/Deltoid_muscle) diff --git a/docs/knowledge/biology/nutrition/ecology.md b/docs/knowledge/biology/nutrition/ecology.md index f53155ee447..dffcdef329f 100755 --- a/docs/knowledge/biology/nutrition/ecology.md +++ b/docs/knowledge/biology/nutrition/ecology.md @@ -32,3 +32,4 @@ https://en.wikipedia.org/wiki/Pseudocereal ## Botany - [How We ALL Get Our Energy (Plants & Ecosystems): Crash Course Botany #13 - YouTube](https://www.youtube.com/watch?v=oZtUFJAJ3lU) +- [Pome - Wikipedia](https://en.wikipedia.org/wiki/Pome) diff --git a/docs/knowledge/biology/nutrition/food.md b/docs/knowledge/biology/nutrition/food.md index 2c7525ac89b..dce41c887b6 100755 --- a/docs/knowledge/biology/nutrition/food.md +++ b/docs/knowledge/biology/nutrition/food.md @@ -165,6 +165,8 @@ https://www.onlymyhealth.com/how-gain-weight-in-days-1326451658 https://www.healthline.com/nutrition/list-of-vegetables +Mushroom - White Truffle [Tuber magnatum - Wikipedia](https://en.wikipedia.org/wiki/Tuber_magnatum) + ## Eight principles of low-glycemic eating 1. Eat a lot of non-starchy vegetables, beans, and fruits such as apples, pears, peaches, and berries. Even tropi- cal fruits like bananas, mangoes, and papayas tend to have a lower glycemic index than typical desserts @@ -326,6 +328,18 @@ Unlike the others, vitamin D can be produced by our bodies, provided our skin is Try to get 10–20 minutes of sunlight ideally before noon, 3-4 times a week. Darker skin needs more sun than lighter skin; so adjust your sun exposure based on your need. +## Infused Water + +**Tooth Enamel Erosion** - Many infused water recipes include citrus fruits, which are known to wear away tooth enamel due to their acidic content. Frequently drinking citrus-infused water could lead to tooth sensitivity or cavities, so be sure to infuse your water with other fruits from time to time. + +**Shelf Life** - Over time, fruit-infused water will go bad, but it will last longer when stored in the refrigerator. If you leave it out at room temperature, it’s best to consume it within four hours. But, if you’re storing it in the refrigerator, it will stay fresh for about three days. + +[Are There Health Benefits to Drinking Infused Water? Pros and Cons, Nutrition, and More](https://www.webmd.com/diet/health-benefits-infused-water) + +[To Infuse or Not to Infuse? | BestFoodFacts.org](https://www.bestfoodfacts.org/vitamins-infused-water/) + +[Health benefits of infused water (and common myths debunked) - Fad Free Nutrition Blog](https://fadfreenutrition.nutricionalgrano.com/health-benefits-of-infused-water/) + ## Farming - Sustenance farming diff --git a/docs/knowledge/biology/others.md b/docs/knowledge/biology/others.md index caf2e0f6fd9..7bc42fa6e26 100755 --- a/docs/knowledge/biology/others.md +++ b/docs/knowledge/biology/others.md @@ -62,3 +62,7 @@ Hydra don't age ![image](../../media/Others-image3-biology.jpg) [The science of super longevity | Dr. Morgan Levine - YouTube](https://www.youtube.com/watch?v=B_CqKVU19ec) + +## Links + +[The more fertile you are, the sooner you may die](https://www.msn.com/en-in/lifestyle/whats-hot/the-more-fertile-you-are-the-sooner-you-may-die-study/ar-AA1lA8ML) diff --git a/docs/knowledge/links.md b/docs/knowledge/links.md index a5895ef3ecf..8e39db558c8 100644 --- a/docs/knowledge/links.md +++ b/docs/knowledge/links.md @@ -1243,5 +1243,17 @@ 1233. [Lasting Lessons from Charlie Munger.](https://youtu.be/3lcRdAmHgKc?si=DWlltEQn9noAwDFk) 1234. [Why India is a Rising Power](https://youtu.be/s2X_nAOYbYY?si=g68v9KN0G68oh9rv) 1235. [Why Greece Is Suddenly Doing Really Well - YouTube](https://youtu.be/aPnu-VPvgb4?si=eofaNvdskJ97-1QJ) +1236. [Why Nigeria Is Suddenly Betting on Solar - YouTube](https://www.youtube.com/watch?v=NDxsgA50OYk&ab_channel=BloombergOriginals) +1237. [Inside The White House With President Joe Biden - YouTube](https://www.youtube.com/watch?v=Jzqu48uup54&ab_channel=ArchitecturalDigest) +1238. [I'VE BEEN ROBBED - YouTube](https://www.youtube.com/watch?v=byfWscC87Vg&ab_channel=JerryRigEverything) +1239. [Seductive AI has no rules. It will change your life (or end it). - YouTube](https://www.youtube.com/watch?v=lgwrRn1uHkg&ab_channel=DigitalEngine) +1240. [Visiting the Closest Town to Space (Life with 50% less oxygen) - YouTube](https://www.youtube.com/watch?v=KqaGdcQh5jA) +1241. [I Flew Into US Navy's Largest Stealth Ship - YouTube](https://www.youtube.com/watch?v=8MlM6B2OMlE) +1242. [2023, in 7 minutes - YouTube](https://www.youtube.com/watch?v=sJs_64OUpEs&ab_channel=Vox) +1243. [Why visas don’t apply to Elon Musk & other billionaires | On the Money | CRED - YouTube](https://www.youtube.com/watch?v=BT6uULSRGpk&ab_channel=CRED) +1244. [5 years after IIT in 5 minutes - YouTube](https://www.youtube.com/watch?v=fZGnOPnvBp8&ab_channel=HarkiratSingh) +1245. [Huge Bank of England Update - What Happens Next? - YouTube](https://www.youtube.com/watch?v=2I1z6Ok0DUo&ab_channel=KatyWilsonProperty) +1246. [India's unbelievable speed in 2023 - YouTube](https://www.youtube.com/watch?v=eEkaB77pp8k&ab_channel=Johnny%27sDesk) +1247. [Why Apple’s iPhone Maker Is Turning to Cars - YouTube](https://www.youtube.com/watch?v=8RlRLHFiVWI&ab_channel=BloombergOriginals) ## End diff --git a/docs/knowledge/others/climate-change.md b/docs/knowledge/others/climate-change.md index 12f477d92fe..652a38707be 100755 --- a/docs/knowledge/others/climate-change.md +++ b/docs/knowledge/others/climate-change.md @@ -27,3 +27,7 @@ Earth overshoot day - 27 July ## Environmental Sciences https://www.pmfias.com/carbon-nitrogen-phosphorus-sulphur-cycle + +## Links + +[The Fourth Big Force: Climate Change](https://www.linkedin.com/pulse/fourth-big-force-climate-change-ray-dalio-vmt5e/) diff --git a/docs/knowledge/others/mythology.md b/docs/knowledge/others/mythology.md new file mode 100644 index 00000000000..72875f4e795 --- /dev/null +++ b/docs/knowledge/others/mythology.md @@ -0,0 +1,7 @@ +# Mythology + +## Akshay Patra + +**Akshaya Patra** (inexhaustible vessel) is a legendary copper vessel featured in the [Hindu epic](https://en.wikipedia.org/wiki/Hindu_epic "Hindu epic") [Mahabharata](https://en.wikipedia.org/wiki/Mahabharata "Mahabharata"). It is a divine vessel given to [Yudhishthira](https://en.wikipedia.org/wiki/Yudishtira "Yudishtira") by [Surya](https://en.wikipedia.org/wiki/Surya "Surya"), which offered a never-depleting supply of food to the [Pandavas](https://en.wikipedia.org/wiki/Pandavas "Pandavas") every day. + +[Akshaya Patra - Wikipedia](https://en.wikipedia.org/wiki/Akshaya_Patra) diff --git a/docs/knowledge/others/readme.md b/docs/knowledge/others/readme.md index ba8970452ae..87106f53a3b 100755 --- a/docs/knowledge/others/readme.md +++ b/docs/knowledge/others/readme.md @@ -11,6 +11,7 @@ - [Chemistry Outline](chemistry-outline) - [News](knowledge/others/news-problems.md) - [Clothes / Textiles](knowledge/others/clothes-textiles.md) +- [Mythology](knowledge/others/mythology.md) ## Art diff --git a/docs/languages/others/other-languages.md b/docs/languages/others/other-languages.md index b225e1bc069..2eff6039abc 100755 --- a/docs/languages/others/other-languages.md +++ b/docs/languages/others/other-languages.md @@ -236,3 +236,9 @@ https://vlang.io [Home ⚡ Zig Programming Language](https://ziglang.org/) [GitHub - zigzap/zap: blazingly fast backends in zig](https://github.com/zigzap/zap) + +## Others + +- [Solidity](decentralized-applications/ethereum/solidity.md) +- [GitHub - move-language/move](https://github.com/move-language/move) + - [Introduction - The Move Book](https://move-language.github.io/move/) diff --git a/docs/languages/sql/intro.md b/docs/languages/sql/intro.md index 36a003768b3..67b15d7f4ff 100755 --- a/docs/languages/sql/intro.md +++ b/docs/languages/sql/intro.md @@ -86,3 +86,4 @@ https://launchbylunch.com/posts/2014/Feb/16/sql-naming-conventions - https://www.codecademy.com/courses/learn-sql/lessons/manipulation/exercises/sql - https://www.codecademy.com/articles/what-is-rdbms-sql - https://www.codecademy.com/articles/sql-commands?r=master +- [SQL 50 - Study Plan - LeetCode](https://leetcode.com/studyplan/top-sql-50/) diff --git a/docs/management/decision-making/ethics.md b/docs/management/decision-making/ethics.md index 3cd2684c8f5..7e1559054b7 100755 --- a/docs/management/decision-making/ethics.md +++ b/docs/management/decision-making/ethics.md @@ -57,3 +57,9 @@ Double Exponential Progress - [The A.I. Dilemma - March 9, 2023 - YouTube](https - A-Z testing of everything - Synthetic relationships - AlphaPersuade + +## Medicine + +[Should Doctors Help People End Their Lives? - YouTube](https://www.youtube.com/watch?v=l-IO6_cU5jM&ab_channel=MedlifeCrisis) + +[Euthanasia in India - Wikipedia](https://en.wikipedia.org/wiki/Euthanasia_in_India) diff --git a/docs/media/Pasted image 20231224193859.png b/docs/media/Pasted image 20231224193859.png new file mode 100644 index 00000000000..364f2fff26f Binary files /dev/null and b/docs/media/Pasted image 20231224193859.png differ diff --git a/docs/media/Pasted image 20231225102216.png b/docs/media/Pasted image 20231225102216.png new file mode 100644 index 00000000000..227d7a8a246 Binary files /dev/null and b/docs/media/Pasted image 20231225102216.png differ diff --git a/docs/media/Screenshot 2023-12-26 at 12.39.39.PM.jpg b/docs/media/Screenshot 2023-12-26 at 12.39.39.PM.jpg new file mode 100644 index 00000000000..ed81440bb52 Binary files /dev/null and b/docs/media/Screenshot 2023-12-26 at 12.39.39.PM.jpg differ diff --git a/docs/media/Screenshot 2023-12-26 at 12.43.35.PM.jpg b/docs/media/Screenshot 2023-12-26 at 12.43.35.PM.jpg new file mode 100644 index 00000000000..06dc4cb8830 Binary files /dev/null and b/docs/media/Screenshot 2023-12-26 at 12.43.35.PM.jpg differ diff --git a/docs/media/Screenshot 2023-12-28 at 8.00.43.PM.jpg b/docs/media/Screenshot 2023-12-28 at 8.00.43.PM.jpg new file mode 100644 index 00000000000..4a88ee38848 Binary files /dev/null and b/docs/media/Screenshot 2023-12-28 at 8.00.43.PM.jpg differ diff --git a/docs/psychology/biases-fallacies.md b/docs/psychology/biases-fallacies.md index f0bce7f7dd7..9e7cfa3cf1d 100755 --- a/docs/psychology/biases-fallacies.md +++ b/docs/psychology/biases-fallacies.md @@ -107,12 +107,9 @@ https://en.wikipedia.org/wiki/Loss_aversion - The certainty effect says individuals prefer certain outcomes over probable ones, while the isolation effect says individuals cancel out similar information when making a decision. - Money 1 - 100% chance of winning $200 (people choose this) - -Money 2 - 20% chance of winning $1000 - +- Money 2 - 20% chance of winning $1000 - Money 1 - 100% chance of losing $800 - -Money 2 - 80% change of losing $1000 (people choose this) +- Money 2 - 80% change of losing $1000 (people choose this) https://www.investopedia.com/terms/p/prospecttheory.asp @@ -397,6 +394,80 @@ And that misconception is called the broken window fallacy. - Lollapalooza effect - When multiple bias are used to sway someone's decision +## Fallacies + +- Ad Hominem +- Hasty Generalization +- Red Herring +- Tu QuoQue +- Slippery Slope +- Special Pleading +- Loaded Question +- False Dilemma +- Strawman +- Circular Reasoning +- Appeal to Authority +- Appeal to Nature +- Composition Fallacy +- Division Fallacy +- Affirming the Consequent +- Anecdotal Fallacy +- Appeal to Emotion +- Burden of Proof Fallacy +- No True Scotsman +- Texas Sharpshooter +- Suppressed Correlative +- Personal Incredulity +- Ambiguity Fallacy +- Genetic Fallacy +- Middle-Ground Fallacy +- Affirming the Disjunct +- Appeal to Tradition +- Sunk Cost Fallacy +- Appeal to Ignorance +- Continuum Fallacy +- Equivocation +- Faulty Analogy +- Denying the Antecedent +- False Cause +- Definist Fallacy +- Ecological Fallacy +- Etymological Fallacy +- Quoting out of Context +- False Equivalence +- Historian's Fallacy +- Inflation of Conflict +- Incomplete Comparison +- Ludic Fallacy +- Moralistic Fallacy +- Nirvana Fallacy +- Proof by Assertion +- Cherry Picking +- Psychologist's Fallacy +- Reification Fallacy +- Retrospective Determinism +- Thought Terminating Cliché +- Fallacy of the single cause +- Appeal to the Stone +- Ignoratio Elenchi +- Circumnstantial ad Hominem +- Tone Policing +- Association Fallacy +- Appeal to Accomplishment +- Courtier's Reply +- Appeal to Consequences +- Appeal to Novelty +- Bulverism +- Chronological Snobbery +- Entitled to my Opinion Fallacy +- Two wrongs make a right +- Vacuous Truth +- Fallacy Fallacy + +[Every Logical Fallacy Explained in 11 Minutes - YouTube](https://www.youtube.com/watch?v=pCg-SNOteQQ&ab_channel=ThePaintExplainer) + +[Every Logical Fallacy Explained in 4 Minutes - YouTube](https://www.youtube.com/watch?v=VZbYecAkGo0&ab_channel=Explainem) + ## Links [LessWrong - Wikipedia](https://en.wikipedia.org/wiki/LessWrong) diff --git a/docs/psychology/mental-models.md b/docs/psychology/mental-models.md index 2d6bbd3df0f..70adfc01710 100755 --- a/docs/psychology/mental-models.md +++ b/docs/psychology/mental-models.md @@ -66,6 +66,14 @@ The dark forest hypothesis is the conjecture that many alien civilizations exist [Dark forest hypothesis - Wikipedia](https://en.wikipedia.org/wiki/Dark_forest_hypothesis) +## as-judged-by-themselves + +The “as-judged-by-themselves” principle is a standard that focuses on people's preferences and the importance of informed choice. It can justify a nudge-based intervention if the nudge aligns people's choices with what they already prefer. + +According to Thaler and Sunstein, nudges should be used to enrich people's lives “as judged by themselves”. They also establish that nudges should be transparent and come with the option to opt out. + +A nudge is a mechanism that influences the behavior of a group of individuals through providing indirect suggestions. The purpose of nudging is to subtly direct people towards positive behavioral choices. + ## Links - [Law of Triviality](soft-skills/prioritization-leverage#law-of-triviality) diff --git a/docs/psychology/personality.md b/docs/psychology/personality.md index f8ed88665b1..18a4f672a35 100755 --- a/docs/psychology/personality.md +++ b/docs/psychology/personality.md @@ -2,7 +2,9 @@ Class is permanent and form is temporary -MBTI - Myers-Briggs Type Indicator for personality test +## MBTI + +Myers-Briggs Type Indicator for personality test https://www.16personalities.com/free-personality-test @@ -14,6 +16,12 @@ https://www.16personalities.com/free-personality-test [Is There An Atheist Personality Type? (Psychology of Atheism Part 2)](https://www.youtube.com/watch?v=xejfuTNov7Y) +## Four temperaments + +The **four temperament theory** is a proto-psychological theory which suggests that there are four fundamental personality types: **sanguine**, **choleric**, **[melancholic](https://en.wikipedia.org/wiki/Melancholic "Melancholic")**, and **phlegmatic**. + +[Four temperaments - Wikipedia](https://en.wikipedia.org/wiki/Four_temperaments) + ## Big 5 Theories of Personality ### 1. Extraversion diff --git a/docs/psychology/soft-skills/disagreement-feedback.md b/docs/psychology/soft-skills/disagreement-feedback.md index db7a815d905..a2a46a7edc9 100755 --- a/docs/psychology/soft-skills/disagreement-feedback.md +++ b/docs/psychology/soft-skills/disagreement-feedback.md @@ -82,10 +82,24 @@ Remember, feedback is not a one-size-fits-all approach. Trust your own judgement ![hierarchy-of-disagreement](../../media/Pasted%20image%2020230628225903.png) +What is your sample size? + [How To Argue - The Hierarchy of Disagreement - Adioma](https://blog.adioma.com/how-to-argue-pg-hierarchy-of-disagreement/) [Hierarchy of disagreement - RationalWiki](https://rationalwiki.org/wiki/Hierarchy_of_disagreement) +#### Reductio ad absurdum + +- Reductio ad absurdum - reduction to absurdity +- argumentum ad absurdum - argument to absurdity +- apagogical arguments + +Is the form of argument that attempts to establish a claim by showing that the opposite scenario would lead to absurdity or contradiction. + +A form of the reductio ad absurdum [argument](https://www.britannica.com/topic/argument-logic), known as [indirect proof](https://www.britannica.com/topic/indirect-proof) or reductio ad impossibile, is one that proves a proposition by showing that its denial conjoined with other propositions previously proved or accepted leads to a contradiction. In common speech the term reductio ad absurdum refers to anything pushed to **absurd extremes**. + +[Reductio ad absurdum - Wikipedia](https://en.wikipedia.org/wiki/Reductio_ad_absurdum) + ## Why You Believe The Things You Do I remember reading an article years ago about a father in Yemen who lost a son to starvation, only to have another child fall dangerously ill. Desperate, he turned to tribal elders who recommended a folk remedy: Shove a burning stick through the sick child’s chest to drain the illness. The father agreed. diff --git a/docs/python/django/overview.md b/docs/python/django/overview.md index 6546a48b601..fa0460432c2 100755 --- a/docs/python/django/overview.md +++ b/docs/python/django/overview.md @@ -117,3 +117,26 @@ https://www.freecodecamp.org/news/django-for-everybody-learn-the-popular-python- - Inline formsets https://www.freecodecamp.org/news/create-a-social-media-app-with-django + +## Interview Questions + +1. Explain Django’s architecture - MVT +2. What is Jinja templating? +3. What is serialization in Django? +4. What is context in Django? +5. Explain user authentication in Django + 1. Users + 2. Permissions + 3. Groups + 4. Password Hashing System + 5. Forms Validation + 6. A pluggable backend system +6. What is the difference between CharField and TextField in Django? +7. What is Django Rest Framework +8. Explain the use of Middlewares in Django +9. What are signals? +10. Explain the Django Response lifecycle + +[Top 50 Django Interview Questions and Answers - GeeksforGeeks](https://www.geeksforgeeks.org/django-interview-questions/#adv) + +[30+ Django Interview Questions (2023) - InterviewBit](https://www.interviewbit.com/django-interview-questions/) diff --git a/docs/python/django/readme.md b/docs/python/django/readme.md index 296018f7df2..766b694b6e6 100755 --- a/docs/python/django/readme.md +++ b/docs/python/django/readme.md @@ -5,7 +5,7 @@ - [Django Template Language](django-template-language) - [Commands](commands) - [Unicode](unicode) -- [Models](models) +- [Models](python/django/models.md) - [QuerySet and Query Expressions](queryset-and-query-expressions) - [Content Types](content-types) - [Admin site](admin-site) diff --git a/docs/technologies/apache/apache-hadoop/others.md b/docs/technologies/apache/apache-hadoop/others.md index d08d5f5237f..18be63c8d60 100755 --- a/docs/technologies/apache/apache-hadoop/others.md +++ b/docs/technologies/apache/apache-hadoop/others.md @@ -1,3 +1,7 @@ # Others Traditionally, Hadoop saves its data internally in flat sequence files, which is a binary storage format for key value pairs. It has the benefit of being more compact than text and fits well the map-reduce output format. Sequence files can be compressed on value, or block level, to improve its IO profile further. Unfortunately, sequence files are not an optimal solution for Hive since it saves a complete row as a single binary value. Consequently, Hive has to read a full row and decompress it even if only one column is being requested. + +## [Azkaban](https://azkaban.github.io/) + +**Azkaban** is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows. diff --git a/docs/technologies/apache/apache-projects.md b/docs/technologies/apache/apache-projects.md index 639e5c3b7c2..02880c3a75d 100755 --- a/docs/technologies/apache/apache-projects.md +++ b/docs/technologies/apache/apache-projects.md @@ -225,25 +225,24 @@ https://hudi.apache.org ## I - [Isis](http://isis.apache.org/) -- [Ignite](http://ignite.apache.org/) +- [Ignite](http://ignite.apache.org/) - In-memory data fabric +- [Impala](http://impala.apache.org/) - Distributed SQL on YARN - In-memory data fabric +### Iceberg -- [Impala](http://impala.apache.org/) - - Distributed SQL on YARN + Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, and Hive to safely work with the same tables, at the same time. -- **Iceberg** +- Expressive SQL +- Full Schema Evolution +- Hidden Partitioning +- Time Travel and Rollback +- Data Compaction - Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, and Hive to safely work with the same tables, at the same time. +![apache-iceberg](../../media/Screenshot%202023-12-26%20at%2012.39.39.PM.jpg) - - Expressive SQL - - Full Schema Evolution - - Hidden Partitioning - - Time Travel and Rollback - - Data Compaction +https://iceberg.apache.org - https://iceberg.apache.org +[Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines - YouTube](https://youtu.be/x6WX_JE96bU?si=hvcH0LIdafH6ZNBU) ## J diff --git a/docs/technologies/brokers/others-brokers.md b/docs/technologies/brokers/others-brokers.md deleted file mode 100755 index 1e42c800fb0..00000000000 --- a/docs/technologies/brokers/others-brokers.md +++ /dev/null @@ -1,86 +0,0 @@ -# Others - Brokers - -## NSQ - A realtime distributed messaging platform - -NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. - -https://nsq.io/overview/internals.html - -https://nsq.io/overview/performance.html - -https://pynsq.readthedocs.io/en/latest - -## Scribe - -Scribe processes logs with an input rate that can exceed 2.5 terabytes per second and an output rate that can exceed 7 terabytes per second. - -https://engineering.fb.com/data-infrastructure/scribe - -## Brooklin - -Brooklin is a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale. Designed for multitenancy, Brooklin can simultaneously power hundreds of data pipelines across different systems and can easily be extended to support new sources and destinations. - -https://engineering.linkedin.com/blog/2019/brooklin-open-source - -https://github.com/linkedin/Brooklin - -## NATS - -NATSis an [open-source](https://en.wikipedia.org/wiki/Open-source_software) messaging system (sometimes called [message-oriented middleware](https://en.wikipedia.org/wiki/Message-oriented_middleware)). The NATS server is written in the [Go programming language](https://en.wikipedia.org/wiki/Go_(programming_language)). Client libraries to interface with the server are available for dozens of major programming languages. The core design principles of NATS are performance, scalability, and ease of use. - -[NATS](https://nats.io/) is a mature, seven-year-old open source messaging technology that implements the publish/subscribe, request/reply and distributed queue patterns to help create a performant and secure method of InterProcess Communication (IPC). Simplicity, performance, scalability and security are the core tenets of NATS. - -The project consists of a family of open source components that are tightly integrated but can be deployed independently. NATS is based on a client-server architecture with servers that can be clustered to operate as a single entity -- clients connect to these clusters to exchange data encapsulated in messages. - -"While most messaging systems provide a mechanism to persist messages and ensure message delivery, NATS does this through log based streaming -- which we've found to be an easier way to store and replay messages," "NATS is a simple yet powerful messaging system written to support modern cloud native architectures. Because complexity does not scale, NATS is designed to be easy to use while acting as a central nervous system for building distributed applications." - -NATS Streaming subscribers can retrieve messages published when they were offline, or replay a series of messages. Streaming inherently provides a buffer in the distributed application ecosystem, increasing stability. This allows applications to offload local message caching and buffering logic into NATS and ensures a message is never lost. - -## Main features - -- Pure pub-sub -- Clustered mode server -- Auto-pruning of subscribers -- Text-based protocol -- Multiple qualities of service (QoS) -- Durable subscriptions -- Event streaming service -- Last/Initial value caching - -## NATS and NATS Streaming - -For context, [NATS](https://nats.io/) and [NATS Streaming](https://nats.io/documentation/streaming/nats-streaming-intro/) are two different things. NATS Streaming is a log-based streaming system built on top of NATS, and NATS is a lightweight pub/sub messaging system. NATS was originally built (and then open sourced) as the control plane for Cloud Foundry. NATS Streaming was built in response to the community's ask for higher-level guarantees - durability, at-least-once delivery, and so forth - beyond what NATS provided. It was built as a separate layer on top of NATS. I tend to describe NATS as a dial tone - ubiquitous and always on - perfect for "online" communications. NATS Streaming is the voicemail - leave a message after the beep and someone will get to it later. There are, of course, more nuances than this, but that's the gist. - -The key point here is that NATS and NATS Streaming are distinct systems with distinct protocols, distinct APIs, and distinct client libraries. In fact, NATS Streaming was designed to essentially act as aclientto NATS. As such, clients don't talk to NATS Streaming directly, rather all communication goes through NATS. However, the NATS Streaming binary can be configured to either embed NATS or point to a standalone deployment. - -![image](../../media/Technologies-Others-Others-Brokers-image1.jpg) - -https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system - -https://en.wikipedia.org/wiki/NATS_Messaging - -https://github.com/nats-io - -https://www.cncf.io/blog/2018/03/15/cncf-to-host-nats - -https://nats.io/documentation - -Used by - Zerodha - -internal streaming technology has changed from a custom TCP protocol to ZeroMQ to Nanomsg/Mango to NATS finally, which is rock solid and highly scalable - -## LiftBridge - -Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the [NATS messaging system](https://nats.io/). It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. - -https://github.com/liftbridge-io/liftbridge - -## LSD Live Streaming Daemon - -- extremely high availability, reliability and local buffering capabilities, limited only by producer's disk storage capacity -- lightweight text-based data format -- simple configuration, deployment and support -- ability to create complex routing schemes for different data streams -- in general you don't need any specific library to send events (just write to plain files) - -https://github.com/badoo/lsd diff --git a/docs/technologies/brokers/others.md b/docs/technologies/brokers/others.md index 13bf2eaa607..50d532e5f8b 100755 --- a/docs/technologies/brokers/others.md +++ b/docs/technologies/brokers/others.md @@ -1,8 +1,93 @@ # Others +## NSQ - A realtime distributed messaging platform + +NSQ is a realtime distributed messaging platform designed to operate at scale, handling billions of messages per day. It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee. + +https://nsq.io/overview/internals.html + +https://nsq.io/overview/performance.html + +https://pynsq.readthedocs.io/en/latest + +## Scribe + +Scribe processes logs with an input rate that can exceed 2.5 terabytes per second and an output rate that can exceed 7 terabytes per second. + +https://engineering.fb.com/data-infrastructure/scribe + +## Brooklin + +Brooklin is a distributed system intended for streaming data between various heterogeneous source and destination systems with high reliability and throughput at scale. Designed for multitenancy, Brooklin can simultaneously power hundreds of data pipelines across different systems and can easily be extended to support new sources and destinations. + +https://engineering.linkedin.com/blog/2019/brooklin-open-source + +https://github.com/linkedin/Brooklin + +## NATS + +NATSis an [open-source](https://en.wikipedia.org/wiki/Open-source_software) messaging system (sometimes called [message-oriented middleware](https://en.wikipedia.org/wiki/Message-oriented_middleware)). The NATS server is written in the [Go programming language](https://en.wikipedia.org/wiki/Go_(programming_language)). Client libraries to interface with the server are available for dozens of major programming languages. The core design principles of NATS are performance, scalability, and ease of use. + +[NATS](https://nats.io/) is a mature, seven-year-old open source messaging technology that implements the publish/subscribe, request/reply and distributed queue patterns to help create a performant and secure method of InterProcess Communication (IPC). Simplicity, performance, scalability and security are the core tenets of NATS. + +The project consists of a family of open source components that are tightly integrated but can be deployed independently. NATS is based on a client-server architecture with servers that can be clustered to operate as a single entity -- clients connect to these clusters to exchange data encapsulated in messages. + +"While most messaging systems provide a mechanism to persist messages and ensure message delivery, NATS does this through log based streaming -- which we've found to be an easier way to store and replay messages," "NATS is a simple yet powerful messaging system written to support modern cloud native architectures. Because complexity does not scale, NATS is designed to be easy to use while acting as a central nervous system for building distributed applications." + +NATS Streaming subscribers can retrieve messages published when they were offline, or replay a series of messages. Streaming inherently provides a buffer in the distributed application ecosystem, increasing stability. This allows applications to offload local message caching and buffering logic into NATS and ensures a message is never lost. + +## Main features + +- Pure pub-sub +- Clustered mode server +- Auto-pruning of subscribers +- Text-based protocol +- Multiple qualities of service (QoS) +- Durable subscriptions +- Event streaming service +- Last/Initial value caching + +## NATS and NATS Streaming + +For context, [NATS](https://nats.io/) and [NATS Streaming](https://nats.io/documentation/streaming/nats-streaming-intro/) are two different things. NATS Streaming is a log-based streaming system built on top of NATS, and NATS is a lightweight pub/sub messaging system. NATS was originally built (and then open sourced) as the control plane for Cloud Foundry. NATS Streaming was built in response to the community's ask for higher-level guarantees - durability, at-least-once delivery, and so forth - beyond what NATS provided. It was built as a separate layer on top of NATS. I tend to describe NATS as a dial tone - ubiquitous and always on - perfect for "online" communications. NATS Streaming is the voicemail - leave a message after the beep and someone will get to it later. There are, of course, more nuances than this, but that's the gist. + +The key point here is that NATS and NATS Streaming are distinct systems with distinct protocols, distinct APIs, and distinct client libraries. In fact, NATS Streaming was designed to essentially act as aclientto NATS. As such, clients don't talk to NATS Streaming directly, rather all communication goes through NATS. However, the NATS Streaming binary can be configured to either embed NATS or point to a standalone deployment. + +![image](../../media/Technologies-Others-Others-Brokers-image1.jpg) + +https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-5-sketching-a-new-system + +https://en.wikipedia.org/wiki/NATS_Messaging + +https://github.com/nats-io + +https://www.cncf.io/blog/2018/03/15/cncf-to-host-nats + +https://nats.io/documentation + +Used by - Zerodha + +internal streaming technology has changed from a custom TCP protocol to ZeroMQ to Nanomsg/Mango to NATS finally, which is rock solid and highly scalable + +## LiftBridge + +Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the [NATS messaging system](https://nats.io/). It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. Use Liftbridge as a simpler and lighter alternative to systems like Kafka and Pulsar or use it to add streaming semantics to an existing NATS deployment. + +https://github.com/liftbridge-io/liftbridge + +## LSD Live Streaming Daemon + +- extremely high availability, reliability and local buffering capabilities, limited only by producer's disk storage capacity +- lightweight text-based data format +- simple configuration, deployment and support +- ability to create complex routing schemes for different data streams +- in general you don't need any specific library to send events (just write to plain files) + +https://github.com/badoo/lsd + ## Message DB -A fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Evented Microservices applications. +A fully-featured event store and message store implemented in PostgreSQL for Pub/Sub, Event Sourcing, Messaging, and Event Microservices applications. Message DB supports: @@ -20,6 +105,6 @@ Message DB supports: http://docs.eventide-project.org/user-guide/message-db -Others +## Others -- Apache Pulsar +- [apache-pulsar](technologies/apache/apache-pulsar.md) diff --git a/docs/technologies/brokers/readme.md b/docs/technologies/brokers/readme.md index 3aa0f60c5b0..a74185c0091 100755 --- a/docs/technologies/brokers/readme.md +++ b/docs/technologies/brokers/readme.md @@ -9,5 +9,4 @@ - [Commands](vernemq/commands) - [sMAP](smap) - [Volttron](volttron) -- [Other Brokers](others-brokers) -- [Others](technologies/brokers/others.md) +- [Other](technologies/brokers/others.md)