NetSure: Network Intrusion Detection System (NIDS)

NetSure is a modular, production-ready machine learning pipeline orchestrating data ingestion, exploratory data analysis (EDA), data validation, feature transformation, training, hyperparameter optimization, and evaluation for the CICIDS2017 dataset. It uses DVC for data & pipeline versioning and MLflow for experiment and artifact tracking.

---

📊 Pipeline Architecture

The orchestration of the machine learning lifecycle in this repository is managed via DVC. Below is the workflow diagram of the pipeline stages:

graph TD
    A[data_ingestion] -->|combined_dataset.csv| B[eda]
    A -->|combined_dataset.csv| C[data_validation]
    A -->|combined_dataset.csv| D[data_transformation]
    
    C -->|status.txt| D
    
    D -->|X_train_preprocessed.csv| E[baseline_training]
    D -->|X_train_preprocessed.csv| F[random_search]
    D -->|X_train_preprocessed.csv| G[simulated_annealing]
    D -->|X_train_preprocessed.csv| H[optuna]
    
    E -->|baseline_model.pkl| I[evaluation]
    F -->|best_random_model.pkl| I
    G -->|best_sa_model.pkl| I
    H -->|best_optuna_model.pkl| I

Loading

---

🐳 Running inside Docker (Containerized Run)

Tip

Recommended Approach: Docker containerization simplifies environment setup. It encapsulates Python 3.10 and all ML requirements (XGBoost, Optuna, MLflow) inside standard containers, and mounts your local folders so that you can view logs, artifacts, and figures directly on the host machine.

1. Fast-Track Parameter Configuration

By default, the optimization stages run for 300 iterations, which can take hours. To run a fast validation check, open config/config.yaml and set n_iter / n_trials to 2:

random_search:
  n_iter: 2
simulated_annealing:
  n_iter: 2
optuna:
  n_trials: 2

2. Startup MLflow and Pipeline

To build the image and execute the training pipeline in tandem with a local MLflow tracking server:

docker-compose up --build

• Local directories (data/, artifacts/, reports/, and mlruns/) are mounted into the pipeline container. All output files and figures generated during training write directly onto your host computer filesystem.
• Open http://localhost:5000 in your browser to monitor parameters, validation F1 metrics, and comparison plots in real-time on your MLflow UI.

---

⚡ How to Reproduce Locally (No Docker)

If you prefer to run the DVC pipeline directly on your host machine:

1. Environment Setup

Install python dependencies:

pip install -r requirements.txt

2. Pipeline Execution

To run the entire pipeline:

dvc repro

To run a specific DVC stage:

dvc repro <stage_name>

DVC Stage Catalog

Stage Name	Command	Outputs Created
data_ingestion	dvc repro data_ingestion	Ingests raw CSV segments and compiles them to combined_dataset.csv
eda	dvc repro eda	Generates summary statistics and profiles class distribution figures in reports/
data_validation	dvc repro data_validation	Validates data schemas and flags issues to artifacts/data_validation/status.txt
data_transformation	dvc repro data_transformation	Scaler fitting, feature selection, and test/train partition splits
baseline_training	dvc repro baseline_training	Fast XGBoost baseline classifier trained with default parameters
random_search	dvc repro random_search	Runs Hyperparameter random search tuning
simulated_annealing	dvc repro simulated_annealing	Search space tuning using Heuristic Simulated Annealing
optuna	dvc repro optuna	Runs Bayesian search space optimization using Tree-structured Parzen Estimator
evaluation	dvc repro evaluation	Generates performance comparison charts, matrices, and metrics.json

---

📈 Experiment Tracking & MLflow

NetSure is preconfigured to log performance metrics, training parameters, models, and diagnostic plots (ROC/PR curves, confusion matrices, feature importances) to MLflow.

• If your remote MLflow Server is running (configured under the mlflow block in config/config.yaml), logs are pushed live.
• If the remote server is unreachable, the system automatically runs in offline mode, logging metrics and artifacts locally in ./mlruns/.