Skip to content

Ehsan Ahmadi's Personal Docs

Bigger Neural Network

Initializing search

Ehsan Ahmadi's Personal Docs

🚀 Cheatsheet
🚀 Cheatsheet
- CI/CD
- Data Science / AI
- Docker
- .Net
- Elasticsearch
- Git
- Go
- Kubernetes
- Linux
- MongoDB
- Mysql
- Nginx
- Postgresql
- Python
- Rabbitmq
- Wordpress
🧪 Data Science
🧪 Data Science
- Overview
  Overview
- 🔍 Features & Data
  🔍 Features & Data
  - Data cleaning
    Data cleaning
    
    Handling Imbalanced Data
    
    Handling Missing Values
    
    Handling Noisy Data
    
    Handling Outliers
    
    Removing Duplicates
  - Data sampling
    Data sampling
    
    Overview
    
    Importance Sampling
    
    Reservoir Sampling
    
    Simple Random Sampling
    
    Stratified Sampling
    
    Weighted Sampling
  - Data size
    Data size
    
    Overview
    
    Aggregation
    
    Augmentation
    
    Clustering
    
    Compression
    
    Generalization
  - Feature engineering extraction
    Feature engineering extraction
    
    Overview
    
    Construction
    Construction
    
    Interaction Terms
    
    Polynomial Expansion
    
    Encoding
    Encoding
    
    Bin-Counting
    
    Dummy Coding
    
    Effect Coding
    
    Feature Hashing
    
    One-Hot Encoding
    
    TF-IDF
    
    Transforming Nominal Attributes
    
    Transforming Ordinal Attributes
    
    Scaling
    Scaling
    
    Log Transformation
    
    Mean Normalization
    
    Normalization (Min-Max)
    
    Power Transformation [Box-Cox]
    
    Robust Scaling
    
    Standardization (Z-score)
    
    Vs (Normalization & Standardization)
    
    Transformation
    Transformation
    
    Binarization
    
    Binning (Quantization)
    
    Rounding
  - Feature selection reduction
    Feature selection reduction
    
    Cheatsheet
    
    Overview
    
    Dimensionality reduction linear
    Dimensionality reduction linear
    
    Overview
    
    Factor Analysis
    
    Linear Discriminant Analysis (LDA)
    
    Principal Component Analysis (PCA)
    
    Vs (PCA & LDA)
    
    Dimensionality reduction manifold
    Dimensionality reduction manifold
    
    Overview
    
    Hessian Eigenmapping (HLLE)
    
    Isometric Feature Mapping (Isomap)
    
    Locally Linear Embedding (LLE)
    
    Multi-dimensional Scaling (MDS)
    
    Spectral Embedding (Laplacian Eigenmaps)
    
    T-Distributed Stochastic Neighbor Embedding (T-SNE)
    
    Uniform Manifold Approximation and Projection (UMAP)
    
    Filter
    Filter
    
    Cheatsheet
    
    Overview
    
    ANOVA [Normal] [Continuous vs Categorical]
    
    Chi-Squared [Not Normal] [2 Categorical]
    
    Mutual Information (Information Gain) [Not Normal] [Continuous or Categorical]
    
    Pearson Correlation Coefficient (R) [Normal] [2 Continuous]
    
    Point-Biserial Correlation Coefficient (Rpb) [Normal] [Continuous vs Binary]
    
    Spearman Correlation Coefficient (\(Rs\) or \(\rho\)) [Not Normal] [2 Continuous]
    
    T-Test [Normal] [Continuous vs Categorical]
    
    Variance Threshold [Not Normal] [1 Continuous]
    
    Wrapper
    Wrapper
    
    Overview
    
    Recursive Feature Elimination (RFE)
- 🧠 AI
  🧠 AI
  - 🦾 Machine Learning
    🦾 Machine Learning
    
    Overview
    
    Classification
    Classification
    
    Decision Trees [Tree]
    
    K-Nearest Neighbors (KNN)
    
    Logistic Regression
    
    Multinomial Logistic Regression
    
    Naive Bayes
    
    Random Forest [Tree]
    
    Support Vector Machine (SVM)
    
    XGBoost (eXtreme Gradient Boosting) [Tree]
    
    Share (Information Gain Technique) [Tree]
    
    Vs (SVM & Logistic Regression)
    
    Clustering
    Clustering
    
    Overview
    
    Anomaly
    
    HDBSCAN
    
    K-Means
    
    VADER [Sentiment Analysis]
    
    Feature extraction
    Feature extraction
    
    FastText
    
    Word2Vec
    
    Recommender
    Recommender
    
    Collaborative Filtering [Unsup]
    
    Content-Based Filtering [Unsup]
    
    Simple Recommendation [Unsup]
    
    Regression
    Regression
    
    Linear Regression (Least Squares) [Sup]
    
    Polynomial Regression [Sup]
    
    Reinforcement
    Reinforcement
    
    Reinforcement Learning [Unsup]
    
    Topic modeling
    Topic modeling
    
    Overview
    
    BERTopic
    
    LDA
    
    NMF
  - 🤖 Neural Network
    🤖 Neural Network
    
    Activation functions
    Activation functions
    
    Linear [\(-\infty\) to \(+\infty\)]
    
    ReLU [\(0\) to \(+\infty\)]
    
    Sigmoid [0 to 1]
    
    Softmax [0 to 1]
    
    Softplus [\(0\) to \(+\infty\)]
    
    TanH [-1 to 1]
    
    Threshold [0 or 1]
    
    Cost functions
    Cost functions
    
    Overview
    
    Cosine Similarity [Embedding]
    
    Cross-Entropy Loss [Multi-Class Classification]
    
    Distortion [Signal Processing]
    
    Log Loss [Binary Classification]
    
    Multiple Negative Ranking (MNR) (InfoNCE7) (NTXentLoss) [Embedding]
    
    Squared Error Loss [Regression]
    
    Fine tuning
    Fine tuning
    
    Overview
    
    Contrastive Learning
    
    Freezing Layers
    
    LoRA [QLoRA]
    
    Masked Language Modeling (MLM)
    
    Parameter-Efficient Fine-Tuning (PEFT)
    
    SetFit
    
    Learning rate techniques
    Learning rate techniques
    
    Adam Algorithm
    
    ADOPT Algorithm
    
    Batch & Mini-batch
    
    Gradient Descent With Momentum
    
    Hyperparameters Tuning
    
    Learning Rate Decay (lrDecay)
    
    Weight Initialization
    
    Network types
    Network types
    
    Auto-Encoders (AE) [Dimensionality Reduction]
    
    Bi-Encoders (Dense Retrieval)
    
    Convolutional (CNN)
    
    Cross-Encoders (Reranking)
    
    Feed Forward (FFNN)
    
    Generative Adversarial (GAN) [Data Augmentation]
    
    Mamba [NLP]
    
    Recurrent (RNN) [Sequential Data]
    
    RWKV [NLP]
    
    Time Series Models [Correlated Data]
    
    Transformers [NLP]
    
    Tasks
    Tasks
    
    Fill-Mask
    
    Named Entity Recognition (NER)
    
    Neural Machine Translation
    
    Part of Speech Tagging (POS)
    
    Question Answering
    
    Sentiment Analysis
    
    Text Generation
    
    Text Summarization
    
    Zero-Shot Classification
  - 🗣️ LLM
    🗣️ LLM
    
    Chain types
    Chain types
    
    Map-Reduce
    
    Map-Rerank
    
    Refine
    
    Search Type
    
    Stuffing
    
    Hyperparameters
    Hyperparameters
    
    max_token
    
    Presence Penalty
    
    Quantization
    
    Repetition Penalty
    
    temperature
    
    top_k
    
    top_p
    
    Vs (top_p & temperature)
    
    Jailbreaking
    Jailbreaking
    
    Overview
    
    Encoding
    
    Many-Shot
    
    Prompt Iinjection
    
    Prompting
    Prompting
    
    Chain-of-Thought (CoT)
    
    Few-Shot
    
    Generated Knowledge
    
    Meta
    
    Prompt Chaining
    
    Retrieval Augmented Generation (RAG)
    
    Self-Consistency
    
    Structured
    
    Tree-of-Thoughts (ToT)
    
    Zero-Shot
    
    Zero-Shot Chain-of-Thought
    
    Vs (Generated Knowledge & RAG)
    
    Vs (Meta & Few-Shot)
    
    Tokenization
    Tokenization
    
    Outputs
    Outputs
    
    Byte Tokens
    
    Character Tokens
    
    Subword Tokens
    
    Word Tokens
    
    Strategies
    Strategies
    
    Byte Pair Encoding (BPE)
    
    SentencePiece
    
    WordPiece
  - Evaluation
    Evaluation
    
    Data Distribution
    
    Orthogonalization
  - Fitting problems
    Fitting problems
    
    Overview
    
    Ensemble methods
    Ensemble methods
    
    Overview
    
    Bagging (Bootstrap Aggregating)
    
    Boosting
    
    Gradient Boosting
    
    Stacking
    
    For neural networks
    For neural networks
    
    Bigger Neural Network Bigger Neural Network
    Table of contents
    
    Description
    
    Dropout Regularization
    
    Early Stopping Regularization
    
    Hyperparameter tuning
    Hyperparameter tuning
    
    Training CV
    
    Imbalanced data
    Imbalanced data
    
    Cost-Sensitive Learning
    
    Regularization
    Regularization
    
    Overview
    
    LASSO (L1)
    
    Ridge (L2)
    
    Vs (LASSO & Ridge)
    
    Splitting data
    Splitting data
    
    Holdout Method (Train/Test Split)
    
    K-Fold
    
    Leave-P-Out
    
    Stratified K-Fold
    
    Time Series Cross-Validation
  - Learning types
    Learning types
    
    Overview
    
    Autoregressive Language Modeling [Self-Sup]
    
    Co-Training [Semi-Sup]
    
    Label Propagation [Semi-Sup]
    
    Masked Language Modeling (MLM) [Self-Sup]
- 📈 Charts
  📈 Charts
  - Categorical
    Categorical
    
    Bar
    
    Confusion Matrix (Error Matrix)
    
    Receiver Operator Characteristic (ROC)
  - Continuous
    Continuous
    
    Histogram
    
    Scatter
- ⚖️ Metrics & Evaluation
  ⚖️ Metrics & Evaluation
  - Categorical
    Categorical
    
    Accuracy [Balance]
    
    F1 [Imbalance]
    
    Precision (PPV) [Imbalance]
    
    Recall (TPR) [Imbalance]
    
    Vs (Recall & Precision)
  - Continuous
    Continuous
    
    Mean Absolute Error (MAE)
    
    Mean Squared Error (MSE)
    
    Coefficient of Determination (\(R^2\))
    
    Root Mean Squared Error (RMSE)
    
    Vs (MSE & RMSE)
👨‍💻 Software
👨‍💻 Software
- 📜 Languages
  📜 Languages
  - Overview
    Overview
    
    Programming Idioms
  - C#
    C#
    
    Architecture
    Architecture
    
    Architecture
    
    Memory Management
    
    Namespace
    
    Solution & Project
    
    Asp
    Asp
    
    Area
    
    Background Tasks
    
    Controller
    
    Data annotations
    
    DI
    
    Identity
    
    Routing
    
    Structure
    
    View
    
    Class
    Class
    
    Class
    
    Delegate & Event
    
    Enumeration
    
    Field
    
    Interface
    
    Method
    
    Property
    
    Record
    
    Structure
    
    Collections
    Collections
    
    Overview
    
    Array
    
    ArrayList
    
    Dictionary
    
    Hashset
    
    Hashtable
    
    LINQ
    
    List
    
    Queue
    
    SortedList
    
    Stack
    
    Share (Dictionary & SortedList & Hashtable)
    
    Share (List & Hashset & ArrayList & Stack & Queue)
    
    Ef
    Ef
    
    Overview & Installation
    
    CMD
    
    Command & Query
    
    DB Context & DB Set
    
    Model & Fields
    
    Fundamentals
    Fundamentals
    
    Async & Await
    
    Character Encoding
    
    Command Line Arguments
    
    Control Statements
    
    Data Types
    
    DateTime
    
    dotnet CLI
    
    Exception Handling
    
    IO
    
    Keywords
    
    Math
    
    Operators
    
    Serialization
    
    String
  - Go
    Go
    
    Passed by Value/Reference
  - Rust
    Rust
    
    Ownership
    
    String
  - Uml
    Uml
    
    Overview
    
    Aggregation (Constructor List)
    
    Association (Constructor Access)
    
    Composition (Constructor Create)
    
    Dependency (Method Access)
    
    Generalization (Extend)
    
    Realization (Implementation)
- 👷 DevOps
  👷 DevOps
  - AWS Services
  - DNS
  - FASS
  - Github Actions
  - Linux
- 📢 Protocols
  📢 Protocols
  - Authentication protocols
    Authentication protocols
    
    Oauth
    
    OAuth2 [OIDC]
    
    SAML
  - Message brockers
    Message brockers
    
    Kafka
  - Protocol rules
    Protocol rules
    
    Resource Naming
    
    RESTful API [HTTP1]
  - Underlying protocols
    Underlying protocols
    
    gRPC [HTTP2]
    
    OSI Model
    
    WebRTC [UDP]
    
    WebSocket [TCP]
- 🗃️ Storages & Data
  🗃️ Storages & Data
  - overview
    overview
    
    Glossary
    
    Storage Estimation
    
    Storage Selection
  - Cache
    Cache
    
    Cache Types
    
    System Memory Hierarchy
  - Data structures
    Data structures
    
    Overview
    
    Array
    
    AVL Tree (Balanced Binary Search Tree)
    
    B-Tree (Balanced Tree)
    
    Binary Search Tree
    
    Binary Tree
    
    Bloom Filter
    
    Characters
    
    Graph
    
    Hash Table (Hash Map)
    
    Heap
    
    Linked List
    
    Queue
    
    Red-Black Tree (Balanced Binary Search Tree)
    
    Set
    
    Skip List
    
    Stack
    
    Trie
    
    Vs (Array & Linked List)
  - Db
    Db
    
    overview
    overview
    
    DB Types
    
    DB Use Cases
    
    Acid transaction
    Acid transaction
    
    Atomicity
    
    Consistency
    
    Durability
    
    Isolation
    
    Data warehouse
    Data warehouse
    
    Overview
    
    Data Lake
    
    Data Mart
    
    Data Mesh
    
    Data Pipeline
    
    Data Warehouse
    
    Delta Lake
    
    Star Schema
    
    Nosql
    Nosql
    
    Data Locality
    
    LSM-Tree Storeage Structure
    
    Elasticsearch
    Elasticsearch
    
    Glossary
    
    DSL (Query Clauses)
    
    DSL (Query Contexts)
    
    Mapping
    
    Replication
    Replication
    
    Overview
    
    Approach: Leaderless Replication
    
    Approach: Multi-Leader Replication
    
    Approach: Single-Leader Replication
    
    Replication Lag Problem
    
    Vs (Single-Leader & Multi-Leader)
    
    Sql
    Sql
    
    B-Tree Storeage Structure
    
    Concurrency Control
    
    Row-Based vs Column-Based
    
    SQL Commands
    
    Standard/Virtual View
  - Log and metric
    Log and metric
    
    Logging
    
    Metrics
    
    Tracing
    
    Vs (Logging & Tracing & Metrics)
  - Object storages
    Object storages
    
    Ceph
    
    MinIO
- System design
  System design
  - Algorithms and patterns
    Algorithms and patterns
    
    API Gateway
    
    Bulkhead
    
    Circuit Breaker
    
    CQRS
    
    Event-Driven
    
    Outbox
    
    Saga
    
    Service Mesh
    
    Service Registry
    
    Sidecar
    
    Strangler
  - Architectures
    Architectures
    
    Overview
    
    Clean Architecture
    
    DDD
    
    Hexagonal Architecture
    
    Micro Services
  - Maintainability
    Maintainability
    
    Overview
    
    Evolvability
    
    Operability
    
    Simplicity
  - Reliability
    Reliability
    
    Overview
    
    Fault-Tolerant (Resilient)
  - Scalability
    Scalability
    
    Overview
    
    Points
- System development
  System development
⚾ Soft Skill
⚾ Soft Skill
🧮 Mathematics
🧮 Mathematics
- Fundamentals
  Fundamentals
  - Functions
  - Logic
  - Math Idioms
  - Sets
- Linear algebra
  Linear algebra
  - Matrix
  - Subset
  - Subspace
  - Transformation
  - Vector
- Probability theory and statistics
  Probability theory and statistics

Table of contents

Description

Bigger Neural Network

Description