Skip to content

Commit

Permalink
updated docs
Browse files Browse the repository at this point in the history
  • Loading branch information
deepaksood619 committed Feb 12, 2024
1 parent 8ce2c0a commit 31e65b1
Show file tree
Hide file tree
Showing 24 changed files with 1,728 additions and 1,655 deletions.
2 changes: 1 addition & 1 deletion docs/ai/llm/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Moving from information to knowledge age
- [Gamma App](https://gamma.app/)
- [Presentation Software | Basic to Beautiful in Minutes with Beautiful.ai](https://www.beautiful.ai/)
- [Decktopus | Create Better Presentations | AI Presentation Assistant](https://app.decktopus.com/dashboard/decks)
- [Plus AI for Google Slides™ - Google Workspace Marketplace](https://workspace.google.com/u/0/marketplace/app/plus_ai_for_google_slides/214277172452)
- [Plus AI for Google Slides™ - Google Workspace Marketplace](https://wvorkspace.google.com/u/0/marketplace/app/plus_ai_for_google_slides/214277172452)
- [AudioPen - The easiest way to convert messy thoughts into clear text.](https://audiopen.ai/)
- [Instantly.ai - Cold Email Software](https://instantly.ai/)
- [Text-to-meme | Supermeme.ai](https://app.supermeme.ai/text-to-meme)
Expand Down
4 changes: 4 additions & 0 deletions docs/cloud/aws/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,7 @@
- [Others](cloud/aws/others.md)

![Cloud Infra 30K Feet Design](../../media/Pasted%20image%2020240205224442.jpg)

### Others

[AWS Financial Services Competency Partners](https://aws.amazon.com/financial-services/partner-solutions/)
2 changes: 2 additions & 0 deletions docs/cloud/others/imagekit.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
- [How to bulk upload images in ImageKit? | ImageKit Help Center](https://help.imagekit.io/en/articles/2558077-how-to-bulk-upload-images-in-imagekit)
- [Amazon S3 bucket - ImageKit.io Docs](https://docs.imagekit.io/integration/configure-origin/amazon-s3-bucket-origin)
- [S3 Compatible External Storages - ImageKit.io Docs](https://docs.imagekit.io/integration/configure-origin/s3-compatible-external-storages)
- [How it works? - ImageKit.io Docs](https://docs.imagekit.io/integration/how-it-works)
- [Optimize and resize images in AWS S3 in real-time with ImageKit](https://imagekit.io/blog/image-optimization-resize-aws-s3-imagekit/)

#### Question

Expand Down
7 changes: 6 additions & 1 deletion docs/computer-science/operating-system/dram.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
# DRAM

Dynamic random-access memory(DRAM) is a type of [random access](https://en.wikipedia.org/wiki/Random-access_memory)[semiconductor memory](https://en.wikipedia.org/wiki/Semiconductor_memory) that stores each [bit](https://en.wikipedia.org/wiki/Bit) of data in a [memory cell](https://en.wikipedia.org/wiki/Memory_cell_(computing)) consisting of a tiny [capacitor](https://en.wikipedia.org/wiki/Capacitor) and a [transistor](https://en.wikipedia.org/wiki/Transistor), both typically based on [metal-oxide-semiconductor](https://en.wikipedia.org/wiki/Metal-oxide-semiconductor)(MOS) technology. The capacitor can either be charged or discharged; these two states are taken to represent the two values of a bit, conventionally called 0 and 1. The [electric charge](https://en.wikipedia.org/wiki/Electric_charge) on the capacitors slowly leaks off, so without intervention the data on the chip would soon be lost. To prevent this, DRAM requires an external [memory refresh](https://en.wikipedia.org/wiki/Memory_refresh) circuit which periodically rewrites the data in the capacitors, restoring them to their original charge. This refresh process is the defining characteristic of dynamic random-access memory, in contrast to [static random-access memory](https://en.wikipedia.org/wiki/Static_random-access_memory)(SRAM) which does not require data to be refreshed. Unlike [flash memory](https://en.wikipedia.org/wiki/Flash_memory), DRAM is [volatile memory](https://en.wikipedia.org/wiki/Volatile_memory)(vs.[non-volatile memory](https://en.wikipedia.org/wiki/Non-volatile_memory)), since it loses its data quickly when power is removed. However, DRAM does exhibit limited [data remanence](https://en.wikipedia.org/wiki/Data_remanence).
Dynamic random-access memory (DRAM) is a type of [random access](https://en.wikipedia.org/wiki/Random-access_memory)[semiconductor memory](https://en.wikipedia.org/wiki/Semiconductor_memory) that stores each [bit](https://en.wikipedia.org/wiki/Bit) of data in a [memory cell](https://en.wikipedia.org/wiki/Memory_cell_(computing)) consisting of a tiny [capacitor](https://en.wikipedia.org/wiki/Capacitor) and a [transistor](https://en.wikipedia.org/wiki/Transistor), both typically based on [metal-oxide-semiconductor](https://en.wikipedia.org/wiki/Metal-oxide-semiconductor)(MOS) technology. The capacitor can either be charged or discharged; these two states are taken to represent the two values of a bit, conventionally called 0 and 1. The [electric charge](https://en.wikipedia.org/wiki/Electric_charge) on the capacitors slowly leaks off, so without intervention the data on the chip would soon be lost. To prevent this, DRAM requires an external [memory refresh](https://en.wikipedia.org/wiki/Memory_refresh) circuit which periodically rewrites the data in the capacitors, restoring them to their original charge. This refresh process is the defining characteristic of dynamic random-access memory, in contrast to [static random-access memory](https://en.wikipedia.org/wiki/Static_random-access_memory)(SRAM) which does not require data to be refreshed. Unlike [flash memory](https://en.wikipedia.org/wiki/Flash_memory), DRAM is [volatile memory](https://en.wikipedia.org/wiki/Volatile_memory)(vs.[non-volatile memory](https://en.wikipedia.org/wiki/Non-volatile_memory)), since it loses its data quickly when power is removed. However, DRAM does exhibit limited [data remanence](https://en.wikipedia.org/wiki/Data_remanence).

DRAM typically takes the form of an [integrated circuit](https://en.wikipedia.org/wiki/Integrated_circuit) chip, which can consist of dozens to billions of DRAM memory cells. DRAM chips are widely used in [digital electronics](https://en.wikipedia.org/wiki/Digital_electronics) where low-cost and high-capacity [computer memory](https://en.wikipedia.org/wiki/Computer_memory) is required. One of the largest applications for DRAM is the [main memory](https://en.wikipedia.org/wiki/Main_memory)(colloquially called the "RAM") in modern [computers](https://en.wikipedia.org/wiki/Computer) and [graphics cards](https://en.wikipedia.org/wiki/Graphics_card)(where the "main memory" is called thegraphics memory). It is also used in many portable devices and [video game](https://en.wikipedia.org/wiki/Video_game) consoles. In contrast, SRAM, which is faster and more expensive than DRAM, is typically used where speed is of greater concern than cost and size, such as the [cache memories](https://en.wikipedia.org/wiki/CPU_cache) in [processors](https://en.wikipedia.org/wiki/Central_processing_unit).

Due to its need of a system to perform refreshing, DRAM has more complicated circuitry and timing requirements than SRAM, but it is much more widely used. The advantage of DRAM is the structural simplicity of its [memory cells](https://en.wikipedia.org/wiki/Memory_cell_(computing)): only one [transistor](https://en.wikipedia.org/wiki/Transistor) and a capacitor are required per bit, compared to four or six transistors in SRAM. This allows DRAM to reach very high [densities](https://en.wikipedia.org/wiki/Computer_storage_density), making DRAM much cheaper per bit. The transistors and capacitors used are extremely small; billions can fit on a single memory chip. Due to the dynamic nature of its [memory cells](https://en.wikipedia.org/wiki/Memory_cell_(computing)), DRAM consumes relatively large amounts of power, with different ways for managing the power consumption.
DRAM had a 47% increase in the price-per-bit in 2017, the largest jump in 30 years since the 45% percent jump in 1988, while in recent years the price has been going down.

https://en.wikipedia.org/wiki/Dynamic_random-access_memory

In [dynamic RAM](https://en.wikipedia.org/wiki/Dynamic_RAM)(DRAM), each [bit](https://en.wikipedia.org/wiki/Bit) of stored data occupies a separate memory cell that is electrically implemented with one [capacitor](https://en.wikipedia.org/wiki/Capacitor) and one [transistor](https://en.wikipedia.org/wiki/Transistor). The charge state of a capacitor (charged or discharged) is what determines whether a DRAM cell stores "1" or "0" as a [binary value](https://en.wikipedia.org/wiki/Binary_value). Huge numbers of DRAM memory cells are packed into [integrated circuits](https://en.wikipedia.org/wiki/Integrated_circuit), together with some additional logic that organises the cells for the purposes of reading, writing, and [refreshing](https://en.wikipedia.org/wiki/Memory_refresh) the data.

Memory cells (blue squares in the illustration) are further organised into [matrices](https://en.wikipedia.org/wiki/Matrix_(mathematics)) and addressed through rows and columns. A memory address applied to a matrix is broken into the row address and column address, which are processed by the row and column [address decoders](https://en.wikipedia.org/wiki/Address_decoder)(in the illustration, vertical and horizontal green rectangles, respectively). After a row address selects the row for a read operation (the selection is also known as [row activation](https://en.wikipedia.org/wiki/Row_activation)), bits from all cells in the row are transferred into the [sense amplifiers](https://en.wikipedia.org/wiki/Sense_amplifier) that form the row buffer (red squares in the illustration), from which the exact bit is selected using the column address. Consequently, read operations are of a destructive nature because the design of DRAM requires memory cells to be rewritten after their values have been read by transferring the cell charges into the row buffer. Write operations decode the addresses in a similar way, but as a result of the design entire rows must be rewritten for the value of a single bit to be changed.

As a result of storing data bits using capacitors that have a natural discharge rate, DRAM memory cells lose their state over time and require periodic [rewriting](https://en.wikipedia.org/wiki/Memory_refresh) of all memory cells, which is a process known as refreshing.As another result of the design, DRAM memory is susceptible to random changes in stored data, which are known as [soft memory errors](https://en.wikipedia.org/wiki/Soft_error) and attributed to [cosmic rays](https://en.wikipedia.org/wiki/Cosmic_ray#Effect_on_electronics) and other causes. There are different techniques that counteract soft memory errors and improve the reliability of DRAM, of which [error-correcting code (ECC) memory](https://en.wikipedia.org/wiki/ECC_memory) and its advanced variants (such as [lockstep memory](https://en.wikipedia.org/wiki/Lockstep_memory)) are most commonly used.

![image](../../media/DRAM-image1.jpg)
Expand Down
2 changes: 2 additions & 0 deletions docs/computer-science/operating-system/memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,3 +173,5 @@ CPU is considered a "compressible" resource. If your app starts hitting your CPU
- **[Disk Cache memory](http://en.wikipedia.org/wiki/Page_cache):** This are chunks of the physical memory, the RAM, used to store files. That way when a program needs to read the file, it's fetched from memory instead of the hard disk. This is done because memory is way faster.

- **[Swap](http://en.wikipedia.org/wiki/Paging):** This is a place on the hard disk (usually a dedicated partition) that is used to store programs or data that can't fit in memory, like when a program grows more than the available memory. SWAP is way slower than RAM, so when you hit swap the computer gets slower, but at least the program can work. In linux swap is also used to hibernate, or to move low used program out of memory to allow more space to the disk cache.

![Types of Memory and Storage](../../media/Pasted%20image%2020240213011810.png)
10 changes: 10 additions & 0 deletions docs/courses/chatgpt-prompt-eng.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,16 @@ Prompt: "I want to learn about `{insert topic}`. Identify and share the most imp

I'm currently learning about snowflake. Ask me a series of 50 questions, one at a time, that will test my knowledge. Wait for my response before proceeding to the next question, ask the next question after your explanation of the answers. Identify knowledge gaps in my answers and give me better answers to fill those gaps. When finish show me the quantity of correct answers and the quantity of failed answers

##### Presentation

Create a presentation for presenting to top management of the company on title "Data Engineering". I as an owner to data engineering vertical working in service based company, have to tell the management about how we can create a Data Engineering vertical.

Start with importance of data and about the data industry. 1 one slide on what are different areas in Data like Data Engineering, Data Science, Data Analytics, ML, AI, etc. Then deep dive into data engineering. Also add on how we can start the vertical by building case studies, finding clients, checking competitive landscape. How much to invest and ROI, whom to hire and how many people to start with. What tools and technologies to focus on.

Presentation should be 30 mins long with 10-15 slides. I am aiming for a script that is persuasive, highlighting the different areas that can be tapped in Data Engineering, and how can we sell these as a service to other companies, and solve other company problems.

ChatGPT, could you aid me in crafting a compelling proposal presentation script for a project centered around integrating AI customer service solutions for Client's Name? I am aiming for a script that is persuasive, highlighting the advantages of our proposed solutions distinctively.

#### Principle 1: Write clear and specific instructions

##### Tactic 1: Use delimiters to clearly indicate distinct parts of the input
Expand Down
10 changes: 5 additions & 5 deletions docs/databases/concepts/types-of-databases.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,11 @@ https://rethinkdb.com
## Hierarchical database / Graph based database (Entities, Relationships)

- Dgraph
- Nebula-graph

https://nebula-graph.io

- Nebula-graph - https://nebula-graph.io
- [Alibaba Graph Database](https://cn.aliyun.com/product/gdb)- A real-time, reliable, cloud-native graph database service that supports property graph model.
- [Amazon Neptune](https://aws.amazon.com/neptune/)- Fully-managed graph database service.
- [ArangoDB](https://github.com/ArangoDB-Community/arangodb-tinkerpop-provider)- OLTP Provider for ArangoDB.
- [Ultimate Scalable Graph Database: ArangoDB for Real-World Use Cases](https://arangodb.com/)
- [GitHub - arangodb/arangodb: 🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.](https://github.com/arangodb/arangodb)
- [Bitsy](https://github.com/lambdazen/bitsy/wiki)- A small, fast, embeddable, durable in-memory graph database.
- [Blazegraph](https://github.com/blazegraph/tinkerpop3)- RDF graph database with OLTP support.
- [CosmosDB](https://docs.microsoft.com/en-us/azure/cosmos-db/graph-introduction)- Microsoft's distributed OLTP graph database.
Expand Down Expand Up @@ -130,6 +128,8 @@ Examples
- Facebook Parse
- Amazon DynamoDB
- Amazon Aurora
- [One-stop Generative AI Stack to Build Production-ready Apps | DataStax](https://www.datastax.com/)
- [Astra DB | DataStax](https://www.datastax.com/products/datastax-astra)

## Object Storage

Expand Down
4 changes: 3 additions & 1 deletion docs/databases/nosql-databases/snowflake/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -470,7 +470,9 @@ When memory spillage is the issue, you can convert your existing warehouse to a

If you want to try resolving the spillage of a query without adjusting the warehouse that runs it, use the [Query Profile](https://docs.snowflake.com/en/user-guide/ui-snowsight-activity.html#label-snowsight-query-profile) to identify which operation nodes are causing the spillage.

[Article Detail](https://community.snowflake.com/s/article/Performance-impact-from-local-and-remote-disk-spilling)
[Performance impact from local and remote disk spilling](https://community.snowflake.com/s/article/Performance-impact-from-local-and-remote-disk-spilling)

[Resolving memory spillage | Snowflake Documentation](https://docs.snowflake.com/en/user-guide/performance-query-warehouse-memory)

### [Cost Management](https://docs.snowflake.com/en/guides-overview-cost "Cost Management")

Expand Down
10 changes: 10 additions & 0 deletions docs/databases/nosql-databases/snowflake/optimizations.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,20 @@ It's worth noting that while clustering can significantly improve performance fo

#### Maintenance Considerations

There are no indices in Snowflake. Instead you can use Data Clustering.

Clustering is not a one-time operation. It's an ongoing process, and the system will automatically reorganize data based on the clustering key during certain operations, such as data loading.

[Micro-partitions & Data Clustering | Snowflake Documentation](https://docs.snowflake.com/en/user-guide/tables-clustering-micropartitions)

##### Clustering Information Maintained for Micro-partitions

Snowflake maintains clustering metadata for the micro-partitions in a table, including:

- The total number of micro-partitions that comprise the table.
- The number of micro-partitions containing values that overlap with each other (in a specified subset of table columns).
- The depth of the overlapping micro-partitions.

## Data compression and metadata storage

### Columnar Storage
Expand Down
Loading

0 comments on commit 31e65b1

Please sign in to comment.