About Project

SentifyFlow is Automation of Scraping & Sentiment Analysis. SentifyFlow is an end-to-end data pipeline for performing the tasks below.

Crawling Twitter (X) data
Sentiment Analysis using fine-tuned BERT
Sentiment Weighting & Scoring
Data persistence to PostgreSQL
Data lake storage in MinIO
Alerting to Telegram & Email
Fully orchestrated using Apache Airflow
Fully containerized using Docker

This project is designed as a production-ready sentiment pipeline that is modular, scalable, and maintainable.

Tech Stack

Orchestration: Apache Airflow (Dockerized)

Containerization: Docker

Object Storage: MinIO

Database: PostgreSQL

Sentiment Model: Fine-Tuned BERT

Alerting: Telegram Bot + Email

Configuration: YAML

Language: Python

Architecture Overview

					┌──────────────────┐
					│  Apache Airflow  │
					│   (Dockerized)   │
					└─────────┬────────┘
							  │
		┌─────────────────────┴───────────────────┐
		│                                         │
   DAG Crawler                                DAG Analyzer
		│                                         │
		▼                                         ▼
  Docker: Crawler                         Docker: Analyzer
		│                                         │
		▼                                         ▼
	  MinIO  ─ ─ ─ ─ ─► PostgreSQL ◄─ ─ ─ ─ ─ ─ MinIO
		│                                         │
		└────────► Telegram & Email Alert ◄───────┘

Project Structure

SentifyFlow/
│   .env
│   .env.example
│   .gitignore
│   docker-compose.yaml
│   Dockerfile
│   README.md
│   requirements.txt
│
├───airflow
│   ├───config
│   │       airflow.cfg
│   │       catalog.yml
│   │       config.yml
│   │       credential.json.example
│   │
│   ├───dags
│   │   │   dag_analyzer.py
│   │   │   dag_crawler.py
│   │
│   ├───logs
│   │   │   airflow.log
│   │   │   sentifyflow.log
│   │ 
│   └───script
│       │   analyzer.py
│       │   crawler.py
│       │   helpers.py
│
├───analyzer
│   │   docker-compose.yaml
│   │   Dockerfile
│   │   requirements.txt
│   │
│   ├───config
│   │       catalog.yml
│   │
│   ├───model
│   │   └───sentiment-classifier-fine-tuned
│   │           config.json
│   │           pytorch_model.bin
│   │           special_tokens_map.json
│   │           tokenizer_config.json
│   │           vocab.txt
│   │
│   └───script
│           helpers.py
│           model.py
│           predict.py
│
└───crawler
	│   .env
	│   docker-compose.yaml
	│   Dockerfile
	│   requirements.txt
	│
	├───config
	│       catalog.yml
	│
	└───script
			crawling_data.py
			find_username.py
			get_token.py
			helpers.py

Pipeline Overview

SentifyFlow consists of two main DAGs:
dag_crawler: Retrieve Twitter data and save it

dag_analyzer: Performing sentiment analysis & alerting

NB: The DAG Analyzer runs after the DAG Crawler finishes.

DAG Crawler

DAG Crawler is used to retrieve Twitter data, enhance the data (username) from results of crawling, and store it in a bucket storage & database. Below is the task flow of this DAG.

check_credentials
        ↓
crawling_tweets (Docker)
        ↓
find_username (Docker)
        ↓
insert_tweets
        ↓
export_raw_tweets

DAG Analyzer

DAG Analyzer is used to perform sentiment analysis using fine-tuned BERT, perform weighting, create a summary of the analysis results, and send alert via Telegram & Email. Below is the task flow of this DAG.

check_credentials
        ↓
sentiment_tweets (Docker)
        ↓
listing_file
        ↓
weighting_sentiment
        ↓
insert_sentiment
        ↓
send_alert
        ↓
housekeeping_file

Architecture of Storage

SentifyFlow uses two storage methods for the results of each process for each pipeline. SentifyFlow uses MinIO as object storage and PostgreSQL as a database.

MinIO (Object Storage)

The following is the structure of the bucket storage that is used.

sentifyflow-bucket/
│   
├───auth
│
├───labeled
|
└───raw

NB:

auth: Stores JSON files containing credentials.

labeled: Stores CSV files containing labeled data.

raw: CSV files containing raw data.

PostgreSQL (Database)

The following is the structure schema of the database that is used.

tweets_raw: Stores data containing raw data.

tweets_sentiment: Stores data containing labeled data.

Containerization

All components in this SentifyFlow project is running on Docker.

Airflow: Docker

Crawler: Docker

Analyzer: Docker

MinIO: Docker

PostgreSQL: Docker

DockerOperator is used to run crawling and sentiment analysis processes with the following objectives.

Isolated environment
Secure dependencies
Scalable

Alert Output

Here is an example of the alert output of the analysis results that will be sent via Telegram and email.

📊 SentifyFlow Market Sentiment Alert (2026-02-20)
Period: 2026-02-13 — 2026-02-19

💡 Market Signal:
Index: 100.00/100 → 🟢 Bullish
Weighted Score: +1.0000
Net Score: +1.0000

📌 Sentiment Summary:
Positive: 59 (100.00%)
Neutral: 0 (0%)
Negative: 0 (0%)

⚖️ Pressure:
Bullish Pressure: 100.00%
Bearish Pressure: 0.00%

🎯 Model Confidence:
Avg Probability: 39.56%

🚀 Top Market Movers (highest weight):
1. @BudiBukanIntel | positive | weight: 6.3742
https://x.com/undefined/status/2024105000167723113
2. @TradingDiary2 | positive | weight: 5.6419
https://x.com/undefined/status/2023075414802137324
3. @TradingDiary2 | positive | weight: 2.3514
https://x.com/undefined/status/2024127800916021428

🔗 Link Task:
http://localhost:8080/dags/DEVdaganalyzer/grid?dagrunid=scheduled2026-02-20T07:02:00+00:00&taskid=sendalert

Key Features

Fully automated
End-to-end pipeline
Fine-tuned BERT model
Engagement-based sentiment weighting
Dockerized & modular
Production-style orchestration
Alerting system
Database consistency control
Storage housekeeping

Future Improvements

Dashboard Visualization (Streamlit / Superset)
Real-time streaming (Kafka)
Model retraining DAG
MLflow integration
Data quality validation (Great Expectations)
API layer untuk expose sentiment index

Conclusion

SentifyFlow is an end-to-end data pipeline implementation that combines data ingestion from Twitter (X), sentiment analysis using fine-tuned BERT, and automatic alerting in an architecture orchestrated with Apache Airflow and containerized using Docker. With a clean split between DAG Crawler and DAG Analyzer, the system is designed to be modular, scalable, and production-ready. The integration of MinIO as object storage and PostgreSQL as a persistent database ensures data consistency and reliability at every stage of the pipeline. Overall, SentifyFlow represents a real-world application of Data Engineering and MLOps practices in building a structured, automated sentiment analytics system that is ready for further development to meet data-driven monitoring needs.

Project Impact:

End-to-End ML Pipeline: From data crawling, preprocessing, BERT inference, weighting, to alerting, everything runs automatically.
Data Engineering Pipeline: Implementation of ingestion, storage, transformation, and persistence in a structured architecture.
Production-Oriented Architecture: Orchestration using Airflow with containerized execution (DockerOperator) for environment isolation and scalability.
MLOps Implementation: Integration of model inference in scheduled workflows with object storage, database persistence, and monitoring alerts.
Alerting System: Automatically deliver sentiment summaries and weighting to Telegram and email as decision-support signals.
Real-World Use Case: It can be used for market sentiment monitoring, brand tracking, and financial signal intelligence.

License

The project is licensed under the MIT license - see the LICENSE file for more details.

More About This Project

You can visit the repository of this project by click here.