updated docs

docs/ai/llm/readme.md

@@ -44,6 +44,7 @@ Moving from information to knowledge age
 - [AI Voice Generator: Versatile Text to Speech Software | Murf AI](https://murf.ai/)
 - [Tammy AI](https://tammy.ai/)
 - captions (Android App)
+- [MeetGeek | Record, Transcribe & Share Meeting Notes](https://meetgeek.ai/)
 
 #### [Midjourney](https://www.midjourney.com/)
 

docs/databases/data-warehousing/11-dw-databases.md

@@ -1,8 +1,8 @@
 # DW - Databases
 
 1. SnowFlake
-2. AWS Redshift
-3. Snowflake
+2. Clickhouse
+3. AWS Redshift
 4. AWS Athena
 5. Google BigQuery
 6. Elastic
@@ -13,7 +13,9 @@
 11. FireBolt
 12. [Databricks](technologies/apache/databricks/readme.md)
 
-## A new class of cloud data warehouses built for AWS
+## Firebolt
+
+A new class of cloud data warehouses built for AWS
 
 Firebolt has completely redesigned the cloud data warehouse to deliver a super fast, incredibly efficient analytics experience at any scale
 

docs/databases/nosql-databases/clickhouse.md

@@ -0,0 +1,143 @@
+# ClickHouse
+
+ClickHouse is an open source column-oriented database management system capable of realtime generation of analytical data reports using SQL queries.
+
+### Key Features
+
+- True column-oriented storage
+- Vectorized query execution
+- Data compression
+- Parallel and distributed query execution
+- Real time query processing
+- Real time data ingestion
+- On-disk locality of reference
+- Cross-datacenter replication
+- High availability
+- SQL support
+- Local and distributed joins
+- Pluggable external dimension tables
+- Arrays and nested data types
+- Approximate query processing
+- Probabilistic data structures
+- Full support of IPv6
+- Features for web analytics
+- State-of-the-art algorithms
+- Detailed documentation - Clean documented code
+
+#### History 
+
+ClickHouse is developed by a Russian company called Yandex. It is designed for multiple projects within Yandex. Yandex needed a DBMS to analyze large amounts of data, thus they began to develop their own column-oriented DBMS. The prototype of ClickHouse appeared in 2009 and it was released to open-source in 2016.
+
+#### Compression 
+
+[Dictionary Encoding](https://dbdb.io/browse?compression=dictionary-encoding) [Delta Encoding](https://dbdb.io/browse?compression=delta-encoding) [Naïve (Page-Level)](https://dbdb.io/browse?compression=naive-page-level)
+
+In addition to general-purpose encoding with LZ4 (default) or Zstd, ClickHouse supports dictionary encoding via LowCardinality data type, as well as delta, double-delta and Gorilla encodings via column codecs.
+
+#### Concurrency Control
+
+[Not Supported](https://dbdb.io/browse?concurrency-control=not-supported)
+
+ClickHouse does not support multi-statement transactions.
+
+#### Data Model 
+
+[Relational](https://dbdb.io/browse?data-model=relational)
+
+ClickHouse uses the relational database model.
+
+#### Foreign Keys
+
+[Not Supported](https://dbdb.io/browse?foreign-keys=not-supported)
+
+ClickHouse does not support foreign keys.
+
+#### Indexes 
+
+[Log-Structured Merge Tree](https://dbdb.io/browse?indexes=log-structured-merge-tree)
+
+ClickHouse supports primary key indexes. The indexing mechanism is called a sparse index. In the MergeTree, data are sorted by primary key lexicographically in each part. Then ClickHouse selects some marks for every Nth row, where N is chosen adaptively by default. Together these marks serve as a sparse index, which allows efficient range queries.
+
+#### Joins 
+
+[Hash Join](https://dbdb.io/browse?joins=hash-join)
+
+ClickHouse uses hash join by default, which is done by placing the right part of data in a hash table in memory. If there's not enough memory for hash join it falls back to merge join.
+
+#### Logging 
+
+[Physical Logging](https://dbdb.io/browse?logging=physical-logging)
+
+ClickHouse replicates its data on multiple nodes and monitors data synchronicity on replicas. It recovers after failures by syncing data from other replica nodes.
+
+#### Parallel Execution 
+
+[Intra-Operator (Horizontal)](https://dbdb.io/browse?parallel-execution=intra-operator) [Inter-Operator (Vertical)](https://dbdb.io/browse?parallel-execution=inter-operator)
+
+ClickHouse utilizes half cores for single-node queries and one replica of each shard for distributed queries by default. It could be tuned to utilize only one core, all cores of the whole cluster or anything in between.
+
+#### Query Compilation 
+
+[Code Generation](https://dbdb.io/browse?query-compilation=code-generation)
+
+ClickHouse supports runtime code generation. The code is generated for every kind of query on the fly, removing all indirection and dynamic dispatch. Runtime code generation can be better when it fuses many operations together and fully utilizes CPU execution units.
+
+#### Query Execution 
+
+[Vectorized Model](https://dbdb.io/browse?query-execution=vectorized-model)
+
+#### Query Interface 
+
+[Custom API](https://dbdb.io/browse?query-interface=custom-api) [SQL](https://dbdb.io/browse?query-interface=sql) [HTTP / REST](https://dbdb.io/browse?query-interface=http-rest) [Command-line / Shell](https://dbdb.io/browse?query-interface=command-line-shell)
+
+ClickHouses provides two types of parsers: a full SQL parser and a data format parser. It uses SQL parser for all types of queries and the data format parser only for INSERT queries. Beyond the query language, it provides multiple user interfaces, including HTTP interface, JDBC driver, TCP interface, command-line client, etc.
+
+#### Storage Architecture
+
+[Disk-oriented](https://dbdb.io/browse?storage-architecture=disk-oriented) [In-Memory](https://dbdb.io/browse?storage-architecture=in-memory) [Hybrid](https://dbdb.io/browse?storage-architecture=hybrid)
+
+ClickHouse has multiple types of table engines. The type of the table engine determines where the data is stored, concurrent level, whether indexes are supported and some other properties. Key table engine family for production use is a [MergeTree](https://clickhouse.tech/docs/en/engines/table_engines/mergetree_family/mergetree/) that allows for resilient storage of large volumes of data and supports replication. There's also a [Log family](https://clickhouse.tech/docs/en/engines/table_engines/log_family/log_family/) for lightweight storage of temporary data and [Distributed engine](https://clickhouse.tech/docs/en/engines/table_engines/special/distributed/) for querying a cluster.
+
+#### Storage Model 
+
+[Decomposition Storage Model (Columnar)](https://dbdb.io/browse?storage-model=decomposition-storage-model-columnar)
+
+ClickHouse is a column-oriented DBMS and it stores data by columns.
+
+#### Storage Organization
+
+[Indexed Sequential Access Method (ISAM)](https://dbdb.io/browse?storage-organization=indexed-sequential-access-method-isam) [Sorted Files](https://dbdb.io/browse?storage-organization=sorted-files)
+
+#### Stored Procedures 
+
+[Not Supported](https://dbdb.io/browse?stored-procedures=not-supported)
+
+Currently, stored procedures and UDF are listed as open issues in ClickHouse.
+
+#### System Architecture 
+
+[Shared-Nothing](https://dbdb.io/browse?system-architecture=shared-nothing)
+
+ClickHouse system in a distributed setup is a cluster of shards. It uses asynchronous multimaster replication and there is no single point of contention across the system.
+
+#### Views 
+
+[Virtual Views](https://dbdb.io/browse?views=virtual-views) [Materialized Views](https://dbdb.io/browse?views=materialized-views)
+
+ClickHouse supports both virtual views and materialized views. The materialized views store data transformed by corresponding SELECT query. The SELECT query can contain DISTINCT, GROUP BY, ORDER BY, LIMIT, etc.
+
+## Internals
+
+[Modern SQL in 2023 - ClickHouse - YouTube](https://www.youtube.com/watch?v=zhrOYQpgvkk)
+
+## Links
+
+[Fast Open-Source OLAP DBMS - ClickHouse](https://clickhouse.com/)
+
+[GitHub - ClickHouse/ClickHouse: ClickHouse® is a free analytics DBMS for big data](https://github.com/ClickHouse/ClickHouse)
+
+[ClickHouse - YouTube](https://www.youtube.com/c/ClickHouseDB)
+
+[What Is ClickHouse? | ClickHouse Docs](https://clickhouse.com/docs/en/intro)
+
+Used by - Zerodha

docs/databases/nosql-databases/comparisions.md

@@ -0,0 +1,14 @@
+# Comparisions
+
+### Clickhouse vs Snowflake
+
+ClickHouse is designed for real-time data analytics and exploration at scale. Snowflake is a cloud data warehouse that is well-optimized for executing long-running reports and ad-hoc data analysis. When it comes to real-time analytics, ClickHouse shines with faster queries at a fraction of the cost.
+
+- Cost: ClickHouse is cost-effective. ClickHouse Cloud is 3-5x more cost-effective than Snowflake.
+- Performance: ClickHouse has faster queries. ClickHouse Cloud querying speeds are over 2x faster than Snowflake.
+- Data compression: ClickHouse Cloud results in 38% better data compression than Snowflake.
+- Architecture: ClickHouse uses Shared-Nothing Architecture by default, but also supports Shared-Disk Architecture.
+- Querying: ClickHouse uses SQL for querying, with support for SQL joins.
+- Integration: ClickHouse integrates with some common tools for visual analytics, including Superset, Grafana and Tableau.
+
+[ClickHouse vs Snowflake](https://clickhouse.com/comparison/snowflake)

docs/databases/nosql-databases/druid/architecture.md

@@ -1,45 +1,75 @@
 # Architecture
 
 Druid has a multi-process, distributed architecture that is designed to be cloud-friendly and easy to operate. Each Druid process type can be configured and scaled independently, giving you maximum flexibility over your cluster. This design also provides enhanced fault tolerance: an outage of one component will not immediately affect other components.
+
 Druid's process types are:
 
-- [**Historical**](http://druid.io/docs/latest/design/historical.html) processes are the workhorses that handle storage and querying on "historical" data (including any streaming data that has been in the system long enough to be committed). Historical processes download segments from deep storage and respond to queries about these segments. They don't accept writes.
-  - Each Historical process serves up data that has been partitioned into segments. These segments are assigned to Historical by the Coordinator via ZooKeeper
-  - When a Historical process is assigned a segment, it will copy the file from deep storage to its local storage
-  - When a query is received from the Broker process, the Historical process returns the results- [**MiddleManager**](http://druid.io/docs/latest/design/middlemanager.html) processes handle ingestion of new data into the cluster. They are responsible for reading from external data sources and publishing new Druid segments.
-  - The MiddleManager process is a worker process that executes submitted tasks. Middle Managers forward tasks to Peons that run in separate JVMs. The reason we have separate JVMs for tasks is for resource and log isolation. Each [Peon](https://druid.apache.org/docs/latest/design/peons.html) is capable of running only one task at a time, however, a MiddleManager may have multiple Peons.
-  - During real-time ingestion, the MiddleManager also serves queries on real-time data before it has been pushed to deep storage.
-  - When a query is received from the Broker process, the MiddleManager process executes that query on real-time data and returns results.- [**Broker**](http://druid.io/docs/latest/design/broker.html) processes receive queries from external clients and forward those queries to Historicals and MiddleManagers. When Brokers receive results from those subqueries, they merge those results and return them to the caller. End users typically query Brokers rather than querying Historicals or MiddleManagers directly.
-  - Broker process is responsible for knowing the internal state of the cluster (from the ZooKeeper)
-  - The broker finds out information from ZooKeeper about the Druid cluster
-    - Which Historical processes are serving which segments
-    - Which MiddleManager processes are serving which tasks' data
-    - When a query is run, the Broker will figure out which process to contact- [**Coordinator**](http://druid.io/docs/latest/design/coordinator.html) processes watch over the Historical processes. They are responsible for assigning segments to specific servers, and for ensuring segments are well-balanced across Historicals.
-  - Segment management and distribution
-  - It communicates with the Historical nodes to:
-    - **Load -** Copy a segment from deep storage and start serving it
-    - **Drop -** Delete a segment from its local copy and stop serving it- [**Overlord**](http://druid.io/docs/latest/design/overlord.html) processes watch over the MiddleManager processes and are the controllers of data ingestion into Druid. They are responsible for assigning ingestion tasks to MiddleManagers and for coordinating segment publishing.
-  - Accepting ingestion supervisors and tasks
-  - Coordinating which servers run which tasks
-  - Managing locks so tasks don't conflict with each other
-  - Returning supervisor and task status to callers- [**Router**](http://druid.io/docs/latest/development/router.html) processes areoptionalprocesses that provide a unified API gateway in front of Druid Brokers, Overlords, and Coordinators. They are optional since you can also simply contact the Druid Brokers, Overlords, and Coordinators directly.
+### [Historical](http://druid.io/docs/latest/design/historical.html)
+
+Historical processes are the workhorses that handle storage and querying on "historical" data (including any streaming data that has been in the system long enough to be committed). Historical processes download segments from deep storage and respond to queries about these segments. They don't accept writes.
+
+- Each Historical process serves up data that has been partitioned into segments. These segments are assigned to Historical by the Coordinator via ZooKeeper
+- When a Historical process is assigned a segment, it will copy the file from deep storage to its local storage
+- When a query is received from the Broker process, the Historical process returns the results
+
+### [MiddleManager](http://druid.io/docs/latest/design/middlemanager.html)
+
+MiddleManager processes handle ingestion of new data into the cluster. They are responsible for reading from external data sources and publishing new Druid segments.
+
+- The MiddleManager process is a worker process that executes submitted tasks. Middle Managers forward tasks to Peons that run in separate JVMs. The reason we have separate JVMs for tasks is for resource and log isolation. Each [Peon](https://druid.apache.org/docs/latest/design/peons.html) is capable of running only one task at a time, however, a MiddleManager may have multiple Peons.
+- During real-time ingestion, the MiddleManager also serves queries on real-time data before it has been pushed to deep storage.
+- When a query is received from the Broker process, the MiddleManager process executes that query on real-time data and returns results.
+
+### [Broker](http://druid.io/docs/latest/design/broker.html)
+
+Broker processes receive queries from external clients and forward those queries to Historicals and MiddleManagers. When Brokers receive results from those subqueries, they merge those results and return them to the caller. End users typically query Brokers rather than querying Historicals or MiddleManagers directly.
+
+- Broker process is responsible for knowing the internal state of the cluster (from the ZooKeeper)
+- The broker finds out information from ZooKeeper about the Druid cluster
+  - Which Historical processes are serving which segments
+  - Which MiddleManager processes are serving which tasks' data
+  - When a query is run, the Broker will figure out which process to contact
+
+### [Coordinator](http://druid.io/docs/latest/design/coordinator.html)
+
+Coordinator processes watch over the Historical processes. They are responsible for assigning segments to specific servers, and for ensuring segments are well-balanced across Historicals.
+
+- Segment management and distribution
+- It communicates with the Historical nodes to:
+  - **Load -** Copy a segment from deep storage and start serving it
+  - **Drop -** Delete a segment from its local copy and stop serving it
+
+### [Overlord](http://druid.io/docs/latest/design/overlord.html)
+
+Overlord processes watch over the MiddleManager processes and are the controllers of data ingestion into Druid. They are responsible for assigning ingestion tasks to MiddleManagers and for coordinating segment publishing.
+
+- Accepting ingestion supervisors and tasks
+- Coordinating which servers run which tasks
+- Managing locks so tasks don't conflict with each other
+- Returning supervisor and task status to callers
+
+### [Router](http://druid.io/docs/latest/development/router.html)
+
+Router processes are optional processes that provide a unified API gateway in front of Druid Brokers, Overlords, and Coordinators. They are optional since you can also simply contact the Druid Brokers, Overlords, and Coordinators directly.
 
 ![image](../../../media/Druid_Architecture-image1.jpg)
 
 <https://docs.imply.io/cloud/design>
+
 Druid processes can be deployed individually (one per physical server, virtual server, or container) or can be colocated on shared servers. One common colocation plan is a three-type plan:
 
 1. **"Data"** servers run Historical and MiddleManager processes.
-
 2. **"Query"** servers run Broker and (optionally) Router processes.
-
 3. **"Master"** servers run Coordinator and Overlord processes. They may run ZooKeeper as well.
+
 In addition to these process types, Druid also has three external dependencies. These are intended to be able to leverage existing infrastructure, where present.
 
 - **[Deep storage](http://druid.io/docs/latest/design/index.html#deep-storage),** shared file storage accessible by every Druid server. This is typically going to be a distributed object store like S3 or HDFS, cassandra, Google Cloud Storage or a network mounted filesystem. Druid uses this to store any data that has been ingested into the system.
 - [**Metadata store**](http://druid.io/docs/latest/design/index.html#metadata-storage), shared metadata storage. This is typically going to be a traditional RDBMS like PostgreSQL or MySQL.
 - [**ZooKeeper**](http://druid.io/docs/latest/design/index.html#zookeeper) is used for internal service discovery, coordination, and leader election.
+
 The idea behind this architecture is to make a Druid cluster simple to operate in production at scale. For example, the separation of deep storage and the metadata store from the rest of the cluster means that Druid processes are radically fault tolerant: even if every single Druid server fails, you can still relaunch your cluster from data stored in deep storage and the metadata store.
+
 The following diagram shows how queries and data flow through this architecture:
 
 ![image](../../../media/Druid_Architecture-image2.jpg)