Skip to content

Ashx87/AI_Health_Assistant_System

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

84 Commits
backend
backend
 
 
frontend
frontend
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 

Repository files navigation

Smart Workplace Health Assistant

AI-powered real-time behaviour analysis system that uses computer vision to detect extended sitting, posture issues, and dehydration — delivering wellness reminders via voice and desktop notifications.

Table of Contents

Overview

In today's digital workplace, prolonged computer use leads to widespread health concerns including sedentary behaviours, poor posture, and inadequate hydration. This project develops a Smart Workplace Health Assistant that uses AI-powered computer vision to monitor user behaviours in real-time.

The system detects extended sitting periods, incorrect posture, and signs of dehydration, then delivers personalised wellness reminders through voice and desktop notifications — transforming the computer from a source of health problems into a proactive wellness companion.

Objectives

  • Investigate workplace health challenges related to prolonged sedentary behaviour, poor posture, and inadequate hydration, and identify limitations of existing solutions.
  • Develop an AI-powered real-time behaviour analysis system using computer vision and deep learning for detecting sitting, posture, and drinking behaviour.
  • Design and evaluate an intelligent multi-modal reminder mechanism that delivers personalised wellness notifications through voice and text outputs.

Features

Feature Description
Real-time Action Recognition YOLO26-pose skeleton extraction + ST-GCN classifier detecting 6 action classes (sitting, standing, drinking, transitions)
Sitting Duration Tracking Tracks continuous sitting periods; triggers reminders after configurable threshold (default 30 min)
Drinking Detection YOLOv8 late-fusion pipeline detects drinking behaviour; reminds user to stay hydrated
Live Camera Feed MJPEG stream with action overlay available via /api/v1/camera/stream
Real-time Dashboard WebSocket-driven sitting status, activity charts, and health analytics
Desktop Notifications Windows toast notifications via PowerShell (winotify) for reminders
SSE Notification Stream Server-Sent Events push sitting/drinking alerts to the browser
REST API Full JWT-authenticated REST API for all features
Auto-start on Boot Backend can open camera and start detection automatically without any frontend interaction

System Architecture

┌──────────────────────────────────────────────────────────────────┐
│                          Frontend (React 19)                      │
│   Dashboard · Monitor · Analysis · Profile · Notifications        │
└──────────────────────┬───────────────────────────────────────────┘
                       │ REST API (JWT) + WebSocket (Socket.IO) + SSE
┌──────────────────────▼───────────────────────────────────────────┐
│                      Flask Backend (Python)                       │
│                                                                   │
│  ┌─────────────┐   ┌──────────────────────────────────────────┐  │
│  │ ServiceMgr  │──▶│            CameraService                 │  │
│  └─────────────┘   │  ┌─────────────┐  ┌──────────────────┐  │  │
│                    │  │FrameCapture │  │   AI Thread (OS)  │  │  │
│  ┌─────────────┐   │  │  (gevent)   │  │ YOLO26-pose      │  │  │
│  │SittingDura- │   │  └─────────────┘  │ ST-GCN Classify  │  │  │
│  │tionService  │   │                   │ YOLOv8 Drinking   │  │  │
│  └─────────────┘   │  ┌─────────────┐  └──────┬───────────┘  │  │
│                    │  │   Reminder  │          │ RealMailbox  │  │
│  ┌─────────────┐   │  │  Services   │◀─────────┘              │  │
│  │DrinkingDe-  │   │  └─────────────┘                         │  │
│  │tectionSvc   │   └──────────────────────────────────────────┘  │
│  └─────────────┘                                                  │
└──────────────────────────────────────────────────────────────────┘
                       │
              ┌────────▼────────┐
              │   PostgreSQL    │
              │   (+ SQLite     │
              │    fallback)    │
              └─────────────────┘

Concurrency Model

The backend runs gevent (monkey.patch_all()), meaning all web handlers, SSE, and Socket.IO are greenlets on a single OS thread. CPU-bound AI inference (YOLO26-pose + ST-GCN) runs on a genuine OS thread (RealThread) and communicates back to the gevent hub via RealMailbox — preventing the AI workload from freezing the event loop.

Tech Stack

Backend

Layer Technology
Language Python 3.10+
Framework Flask 3.0
WebSocket Flask-SocketIO 5 + gevent
Database PostgreSQL 16 (SQLite for dev)
ORM SQLAlchemy 2 + Flask-Migrate
Authentication Flask-JWT-Extended
Computer Vision OpenCV, Ultralytics (YOLO)
Deep Learning PyTorch 2+
Serialization Marshmallow
Production Server Gunicorn
Containerization Docker, Docker Compose

Frontend

Layer Technology
Language TypeScript
Framework React 19
Build Tool Vite 7
Styling Tailwind CSS v4
Routing React Router
State Context API
Real-time Socket.IO Client + EventSource (SSE)

AI Pipeline

Stage Model
Skeleton Extraction YOLO26-pose (17 COCO keypoints)
Action Classification ST-GCN (6 classes, 10 blocks)
Drinking Verification YOLOv8 (late-fusion)

Prerequisites

  • Python 3.10+ with Anaconda/Miniconda
  • Node.js 18+ and npm
  • PostgreSQL 16 (or Docker)
  • CUDA-capable GPU (optional; CPU inference supported)
  • Webcam connected to the machine

Installation

1. Clone the repository

git clone <repo-url>
cd AI_Health_Assistant_System

2. Backend setup

# Create and activate conda environment
conda create -n BD python=3.10
conda activate BD

# Install dependencies
cd backend
pip install -r requirements.txt

# Install PyTorch (choose based on your CUDA version)
# CUDA 11.8:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
# CPU only:
pip install torch torchvision

3. Configure environment

# Copy example and fill in values
cp env.example.txt .env

Edit .env:

FLASK_ENV=development
DATABASE_URL=postgresql://postgres:postgres@localhost:5432/health_assistant
SECRET_KEY=<your-secret-key>
JWT_SECRET_KEY=<your-jwt-secret-key>

See Configuration for all available variables.

4. Run database migrations

conda run -n BD flask db upgrade

5. Frontend setup

cd ../frontend
npm install

Running the System

Development (recommended)

Open two terminals:

Terminal 1 — Backend

cd backend
conda activate BD
python app.py
# Backend starts on http://localhost:5000
# Camera and detection auto-start on boot (configurable via AUTO_START_CAMERA)

Terminal 2 — Frontend

cd frontend
npm run dev
# Frontend starts on http://localhost:5173

Open http://localhost:5173 in your browser.

Makefile shortcuts (from backend/)

make test           # Run pytest
make test-cov       # Run pytest with HTML coverage report
make lint           # flake8 + isort check
make format         # black + isort
make db-migrate message="describe change"
make db-upgrade

Project Structure

AI_Health_Assistant_System/
├── backend/
│   ├── app/
│   │   ├── api/                    # Blueprint route handlers
│   │   │   ├── auth.py             # Registration, login, JWT
│   │   │   ├── camera.py           # Camera control, stream, SSE
│   │   │   ├── health.py           # Health profiles, BMI, symptoms
│   │   │   ├── sitting_duration.py # Sitting period tracking
│   │   │   ├── analytics.py        # Usage analytics
│   │   │   ├── dashboard.py        # Dashboard aggregates
│   │   │   └── notifications.py    # Notification history
│   │   ├── models/                 # SQLAlchemy models
│   │   ├── services/               # Business logic layer
│   │   │   ├── camera_service.py   # Frame capture + MJPEG stream
│   │   │   ├── action_recognition/ # YOLO26-pose + ST-GCN pipeline
│   │   │   ├── drinking_detection_service.py
│   │   │   ├── sitting_duration_service.py
│   │   │   ├── sitting_reminder_service.py
│   │   │   ├── realtime_primitives.py  # RealThread / RealMailbox
│   │   │   ├── thread_manager.py       # Worker lifecycle management
│   │   │   ├── service_manager.py      # Service coordinator
│   │   │   └── workers/            # Background worker threads
│   │   ├── config.py               # Dev / Test / Prod configs
│   │   └── extensions.py           # Flask extension singletons
│   ├── migrations/                 # Alembic migration files
│   ├── tests/                      # pytest test suite
│   ├── app.py                      # Dev entry point (socketio.run)
│   ├── wsgi.py                     # Production entry point (Gunicorn)
│   ├── docker-compose.yml
│   ├── Dockerfile
│   └── requirements.txt
│
├── frontend/
│   ├── src/
│   │   ├── components/             # Reusable UI components
│   │   ├── pages/                  # Route-level page components
│   │   ├── contexts/               # AuthContext (JWT management)
│   │   ├── services/               # API client + domain services
│   │   ├── layouts/                # MainLayout with navigation
│   │   └── App.tsx                 # Router configuration
│   ├── vite.config.ts
│   └── package.json
│
└── docs/                           # Additional documentation

AI Pipeline

Action Classes (6 total)

ID Label Description
0 drinking_sitting Drinking while seated
1 drinking_standing Drinking while standing
2 sitting Seated posture
3 sit_down Transition: stand → sit
4 standing Standing posture
5 stand_up Transition: sit → stand

"No person" is not a model class — it is decided upstream by the pose stage when YOLO26-pose detects no person or average keypoint confidence < 0.2.

Pipeline Stages

Webcam Frame
    │
    ▼
YOLO26-pose ──▶ 17 COCO keypoints (x, y, conf) × 30 frames
    │            Resampled to fixed-length clip via linear interpolation
    ▼
ST-GCN (10 blocks)
    │            Input tensor: (N, 3, 30, 17, 1)
    │            Graph: COCO 17-joint topology, 3-partition strategy
    │            Channels: 64 → 128 (stride-2 at block 5) → 256 (stride-2 at block 8)
    ▼
6-class softmax  ──▶  action label + confidence
    │
    ▼ (if drinking class detected)
YOLOv8 late-fusion verification

Model Files

The backend loads its models from backend/models/ (organized by stage). Each path is overridable via an environment variable:

Model Default path Env override
ST-GCN action classifier backend/models/HAR/STGCN_v2.pth STGCN_MODEL_PATH
YOLO26-pose skeleton extractor backend/models/selekton_extract/yolo26n-pose.pt YOLO_POSE_MODEL_PATH
YOLOv8 drinking detector backend/models/object_detection/yolov8n.pt YOLO_MODEL_PATH

Defaults are resolved relative to the backend/ directory, so no configuration is needed if the files sit in those locations. Training writes the ST-GCN checkpoint to Data_Preprocess/checkpoints/best_model_STGCN.pth — copy it into backend/models/HAR/ for runtime, or point STGCN_MODEL_PATH at it.

API Reference

All endpoints are prefixed with /api/v1/. Protected routes require Authorization: Bearer <token>.

Authentication

Method Endpoint Description
POST /auth/register Create account
POST /auth/login Login, receive JWT
POST /auth/refresh Refresh access token
POST /auth/logout Revoke token

Camera

Method Endpoint Description
POST /camera/start Start camera + detection
POST /camera/stop Stop camera
GET /camera/status Current camera/detection state
GET /camera/stream MJPEG video stream
GET /camera/notifications/stream SSE push stream for reminders

Sitting Duration

Method Endpoint Description
GET /sitting_duration/stats Sitting statistics
GET /sitting_duration/history Sitting period history
GET /sitting_duration/today Today's summary

Health Profile

Method Endpoint Description
GET /health/profile Get health profile
PUT /health/profile Update health profile
GET /health/bmi Calculate BMI

Dashboard & Analytics

Method Endpoint Description
GET /dashboard/summary Aggregated dashboard data
GET /analytics/weekly Weekly activity report
GET /analytics/trends Behaviour trend data

WebSocket Events (Socket.IO)

Event Direction Description
sitting_status Server → Client Live sitting status update
detection_update Server → Client Action recognition result

Configuration

All configuration is via environment variables in backend/.env:

Variable Default Description
FLASK_ENV development development / testing / production
DATABASE_URL SQLite PostgreSQL connection string
SECRET_KEY — Flask session secret (change in production)
JWT_SECRET_KEY — JWT signing secret (change in production)
SITTING_REMINDER_THRESHOLD_MINUTES 30 Minutes before sitting reminder fires
SITTING_REMINDER_COOLDOWN_MINUTES 5 Cooldown between repeated reminders
AUTO_START_CAMERA true Open camera + detection on backend boot
AUTO_START_CAMERA_INDEX 0 Capture device index
CORS_ORIGINS localhost:3000,localhost:5173 Allowed CORS origins
POSE_DETECTION_CONFIDENCE 0.7 YOLO pose confidence threshold
POSE_DETECTION_IOU 0.3 YOLO IOU threshold
STGCN_MODEL_PATH models/HAR/STGCN_v2.pth ST-GCN checkpoint location
YOLO_POSE_MODEL_PATH models/selekton_extract/yolo26n-pose.pt YOLO26-pose model location
YOLO_MODEL_PATH models/object_detection/yolov8n.pt YOLOv8 drinking detector location
LOG_LEVEL DEBUG Python logging level

Docker Deployment

cd backend
docker-compose up -d

Three services start automatically:

Service Port Description
api 5000 Flask application
db 5432 PostgreSQL 16
redis 6379 Redis 7 (rate limiting)

Run migrations inside the container:

docker-compose exec api flask db upgrade

Testing

cd backend
conda activate BD

# Run all tests
pytest

# Suppress ServiceManager log noise
pytest 2>&1 | grep -E "PASSED|FAILED|passed|failed"

# Single file
pytest -v tests/test_auth.py

# With coverage
pytest --cov=app --cov-report=term-missing

Test configuration is in backend/pytest.ini. The TestingConfig forces AUTO_START_CAMERA=false so tests never open a real camera.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages