--- language: - en library_name: transformers license: mit tags: - sentiment-analysis - emotion-classification - text-classification - multi-label-classification - deberta-v3 - nlp - pytorch pipeline_tag: text-classification --- # Sentimental Analyzer V3 **Sentimental Analyzer V3** is a multi-task Natural Language Processing (NLP) model designed to analyze English text through two complementary tasks: 1. **5-class sentiment classification** 2. **28-class multi-label emotion classification** The model uses `microsoft/deberta-v3-base` as its transformer backbone and includes a frozen production calibration pipeline for emotion predictions. ## Model Overview | Property | Value | |---|---| | Model | Sentimental Analyzer V3 | | Backbone | `microsoft/deberta-v3-base` | | Framework | PyTorch | | Maximum sequence length | 256 | | Sentiment classes | 5 | | Emotion classes | 28 | | Sentiment task | Single-label classification | | Emotion task | Multi-label classification | | Production calibration | `HYBRID_ISOTONIC_GLOBAL` | | Calibrated emotion classes | 11 | | Raw emotion classes | 17 | | Global emotion threshold | 0.35 | | Fallback strategy | `highest_calibrated_probability` | ## What the Model Does Given an English text or comment, the model produces: - One sentiment class from five possible sentiment categories. - One or more emotion labels from 28 possible emotion categories, with a fallback selection when no class crosses the production threshold. - A probability distribution over the five sentiment classes. - A probability for each of the 28 emotion classes. The model is designed for text analytics, exploratory NLP, research, educational demonstrations, and applications that analyze user feedback. ## Sentiment Classification The model predicts one of five sentiment categories: | ID | Sentiment | |---:|---| | 0 | Negative | | 1 | Neutral | | 2 | Positive | | 3 | Very Negative | | 4 | Very Positive | Sentiment classification is a single-label classification task. The sentiment probabilities are normalized using softmax. ## Emotion Classification The model performs multi-label emotion classification. | ID | Emotion | |---:|---| | 0 | admiration | | 1 | amusement | | 2 | anger | | 3 | annoyance | | 4 | approval | | 5 | caring | | 6 | confusion | | 7 | curiosity | | 8 | desire | | 9 | disappointment | | 10 | disapproval | | 11 | disgust | | 12 | embarrassment | | 13 | excitement | | 14 | fear | | 15 | gratitude | | 16 | grief | | 17 | joy | | 18 | love | | 19 | nervousness | | 20 | neutral | | 21 | optimism | | 22 | pride | | 23 | realization | | 24 | relief | | 25 | remorse | | 26 | sadness | | 27 | surprise | Multiple emotions can be associated with the same text. ## Architecture The model uses `microsoft/deberta-v3-base` as the transformer encoder. ```text Input Text | v DeBERTa-v3-base Encoder | v Attention Pooling | v Shared Representation / \ / \ v v Sentiment Head Emotion Head | | v v 5 Sentiments 28 Emotions ``` ### Attention Pooling The sequence-level hidden representations produced by DeBERTa are aggregated using learned attention pooling. Padding tokens are masked before the attention softmax so that padded positions do not contribute to the pooled representation. ### Shared Representation The pooled representation is passed through: - Linear projection - GELU activation - Layer normalization - Dropout The resulting representation is shared by the two task-specific prediction heads. ### Sentiment Head The sentiment head produces five logits. Softmax is applied during inference to obtain the sentiment probability distribution. ### Emotion Head The emotion head produces 28 logits. Sigmoid probabilities are used because emotion classification is multi-label. ## Production Calibration The production emotion pipeline uses the frozen strategy: `HYBRID_ISOTONIC_GLOBAL` ```text Raw Emotion Logits | v Sigmoid Probabilities | v Class-wise Isotonic Calibration | v Global Threshold = 0.35 | v Final Emotion Labels ``` ### Calibration Configuration - Calibration strategy: `HYBRID_ISOTONIC_GLOBAL` - Calibrated emotion classes: 11 - Raw emotion classes: 17 - Minimum calibration support: 10 - Global threshold: 0.35 - Fallback: `highest_calibrated_probability` The following 11 emotion classes use isotonic calibration: - admiration - anger - annoyance - approval - disappointment - disapproval - disgust - excitement - joy - love - neutral The following 17 rare classes retain their raw sigmoid probabilities: - amusement - caring - confusion - curiosity - desire - embarrassment - fear - gratitude - grief - nervousness - optimism - pride - realization - relief - remorse - sadness - surprise Calibration parameters and the threshold were selected using validation data. The final test set was not used for calibration or threshold selection. ## Evaluation The final V3 checkpoint was evaluated on an untouched real-world test split containing **253 examples**. ### Sentiment Results | Metric | Score | |---|---:| | Accuracy | 0.8024 | | Macro F1 | 0.6499 | | Weighted F1 | 0.7864 | ### Emotion Results | Metric | Score | |---|---:| | Micro F1 | 0.5770 | | Macro F1 | 0.3026 | | Weighted F1 | 0.5968 | | Macro Precision | 0.2791 | | Macro Recall | 0.4198 | These results correspond to the frozen V3 checkpoint and production calibration configuration. ## Evaluation Interpretation The sentiment task shows stronger performance than the 28-class emotion task. The emotion task is more challenging because: - It is multi-label. - Some emotions are substantially rarer than others. - Emotion boundaries can be subjective. - Multiple emotions can coexist in the same text. - Some classes have limited calibration support. Macro-level emotion metrics are particularly sensitive to performance on rare classes. ## Dataset and Labeling Limitation The real-world adaptation dataset used during development contains **LLM/pseudo-labeled examples rather than a fully human-annotated gold-standard dataset**. Therefore, the reported real-world adaptation results should be interpreted as evaluation against the project's labeling methodology rather than definitive human-gold benchmark performance. The model may perform differently on text that differs substantially from the adaptation data, including different domains, writing styles, topics, vocabulary, slang, cultural contexts, levels of context, or forms of sarcasm and implicit meaning. ## Intended Use Sentimental Analyzer V3 is intended for: - Sentiment analysis of English comments and short text. - Emotion analysis of user feedback. - NLP research and experimentation. - Educational demonstrations. - Exploratory analysis of text datasets. - Aggregate analysis of customer or user feedback. - Integration into applications through an inference API. The model can be used as an inference component behind a web application or API. ## Limitations This model should not be considered a perfect representation of human emotion. Important limitations include: - Emotion classification can be subjective. - Some emotion classes are substantially rarer than others. - Multi-label emotion annotation can be difficult to validate consistently. - Sarcasm and implicit meaning may be challenging. - Domain shift may reduce performance. - Rare emotion classes have limited calibration support. - The real-world adaptation labels are not fully human-gold. - The model has primarily been developed and evaluated for English text. - Model predictions are statistical estimates and should not be treated as objective facts about a person's feelings. - Predictions should not be interpreted as psychological, medical, or clinical assessments. ## Example ### Input ```text I absolutely love this product! It is amazing. ``` ### Example Production Output ```json { "sentiment": "Very Positive", "sentiment_confidence": 0.987, "emotions": [ "joy" ] } ``` The exact probability values depend on the frozen checkpoint and production calibration artifacts. ## Model Artifacts ```text Sentimental-Analyzer-V3/ │ ├── README.md ├── config.json │ ├── model/ │ └── best_v3_model.pt │ ├── calibration/ │ └── v3_isotonic_calibrators.pkl │ └── labels/ ├── sentiment_labels.json └── emotion_labels.json ``` ### Artifact Descriptions - `model/best_v3_model.pt`: Frozen V3 PyTorch checkpoint containing the trained model parameters. - `calibration/v3_isotonic_calibrators.pkl`: Frozen emotion calibration parameters used by the production inference pipeline. - `labels/sentiment_labels.json`: Mapping between sentiment class IDs and sentiment labels. - `labels/emotion_labels.json`: Mapping between emotion class IDs and emotion labels. - `config.json`: Frozen production configuration containing model, class, calibration, decision, selection, and checkpoint integrity information. ## Checkpoint Integrity The published checkpoint corresponds to the frozen V3 production candidate. SHA-256: ```text 4d3f5a97d352460d41fa884fd337b2de6f58549139135cad1b4b79beefcb2078 ``` The checkpoint was verified against this SHA-256 hash before publication. ## Frozen Production Configuration ```text Version: V3_FINAL_CANDIDATE Model: microsoft/deberta-v3-base Maximum sequence length: 256 Sentiment classes: 5 Emotion classes: 28 Calibration strategy: HYBRID_ISOTONIC_GLOBAL Calibrated emotion classes: 11 Raw emotion classes: 17 Minimum calibration support: 10 Global emotion threshold: 0.35 Fallback: highest_calibrated_probability Selection dataset: validation Test used for selection: false Threshold source: validation Calibration source: validation ``` No additional training or calibration is performed during production inference. ## Inference Behavior ```text Input text | v Tokenization | v DeBERTa-v3-base | v Shared representation | +-------------------------+ | | v v Sentiment logits Emotion logits | | v v Softmax Sigmoid | | v v Sentiment label Frozen calibration | v Threshold = 0.35 | v Emotion labels ``` The model is loaded in evaluation mode and its parameters are frozen during inference. ## API Integration The model can be exposed through a FastAPI service. A production API can provide: ```text POST /predict ``` Example request: ```json { "text": "I absolutely love this product!" } ``` Example response: ```json { "sentiment": "Very Positive", "sentiment_confidence": 0.98, "sentiment_probabilities": { "Negative": 0.00, "Neutral": 0.00, "Positive": 0.01, "Very Negative": 0.00, "Very Positive": 0.98 }, "emotions": [ "joy" ], "emotion_probabilities": { "joy": 0.76 } } ``` The API layer can be deployed separately from this model repository so that API users do not need to download the model checkpoint directly. ## Reproducibility The V3 production candidate is frozen. The following components are fixed: - Model checkpoint - Model architecture - Tokenizer configuration - Maximum sequence length - Sentiment label mapping - Emotion label mapping - Emotion ordering - Emotion calibration artifacts - Global emotion threshold - Fallback strategy - Production configuration The checkpoint can be verified using the published SHA-256 hash. ## Responsible Use This model produces statistical predictions from textual patterns learned during training and adaptation. Its outputs should not be presented as certain facts about: - A person's actual emotional state - A person's intentions - A person's personality - A person's psychological condition - A person's medical or clinical status Applications using this model should provide appropriate context and should avoid presenting model predictions as definitive judgments. ## Project Status **V3 production candidate — frozen.** The model, calibration configuration, and production inference behavior have been verified before publication. Training and calibration are not performed during production inference. ## License MIT