AO8 : Adversarial Robustness Testing for Security-Critical ML Models

Marvel Level 3 Project Yashaswini C Rao

• GitHub: github.com/yash-r04/AO8
• Drive (report, screenshots, extra docs): Link

---

Why this project exists

ML models are increasingly used in security-critical systems — intrusion detection, malware classification, fraud detection. On paper, these models often look highly accurate. But that accuracy is measured on clean, well-behaved data. Nobody typically tests what happens when someone intentionally tries to break the model.

It turns out you don't need to change an input much to break it. A tiny, often invisible perturbation to the data can flip a model's prediction completely — a technique known as an adversarial attack. Despite this being a well-documented risk, most deployed security ML models are never stress-tested against it before going live.

The core problem: there's no easy, accessible way for someone to just upload their model and dataset and see how it actually holds up under a real attack. Existing tools like CleverHans and Foolbox are attack libraries — you have to write code to use them, they aren't security-domain-focused, and they don't hand you defenses or a readable report out of the box.

AO8 was built to close that gap: point, click, and get a structured robustness report — with an option to harden the model afterward.

---

What AO8 actually does

1. You upload a trained model (TorchScript / ONNX / sklearn) and a CSV dataset.
2. You pick which attack(s) to run — FGSM, PGD, and/or C&W — all white-box attacks (the tool has full access to the model's weights and gradients, representing the worst-case, most-informed threat).
3. AO8 runs the attack(s) and compares clean accuracy vs. accuracy under attack, surfacing:
   • which specific samples got flipped (right → wrong, or vice versa)
   • how small a perturbation it took to fool each one
   • which input features got exploited the most
   • a per-attack risk score (0–100)
4. It also exports "safe values" — inputs the model never got fooled on — which can later be used to retrain or harden the model.
5. Optionally, you can click "Harden this model," which runs adversarial training and gives you a before/after comparison plus a new, hardened model file to download.

Scope:

• ✅ In scope: white-box FGSM/PGD/C&W attacks, independently selectable defenses, clear before/after accuracy comparisons
• ❌ Out of scope: physical-world attacks (e.g. printed adversarial patches), formal mathematical robustness guarantees, direct production-pipeline integration

Who it's for: ML safety/security researchers, red teams simulating attacks, and security engineers who want to stress-test a model before shipping it.

---

Tech stack

Concern	Technology
Backend	Flask
Auth	Google login (via AWS Cognito) + GitHub OAuth
Database	PostgreSQL + SQLAlchemy
Background jobs	Celery + Redis
Attack engine	IBM Adversarial Robustness Toolbox (ART)
File storage	AWS S3 (server-side encrypted)
Frontend	Jinja2 templates + plain CSS/JS (no framework)
Report generation	ReportLab (downloadable PDF)

Server-rendered templates were chosen over a React/SPA setup because the app is fundamentally form-heavy — upload → configure → view results — and didn't need the overhead of a full single-page app.

Cost: ₹0 total — AWS, ART, and Redis (via Upstash) all run on free tiers.

---

Program flow — how a job actually moves through the system

Walking through this end-to-end:

1. Upload & validate — You upload a model and a CSV dataset through the browser. The model is validated by actually trying to load it with the right library (TorchScript loader, ONNX runtime, or pickle for sklearn); if that fails, the upload is rejected immediately. Both files are stored as raw encrypted bytes in S3, with only metadata (name, framework, S3 key, size) saved to Postgres.
2. Configure & submit — On the Benchmark page you pick a model, a dataset, which attacks to run, and an epsilon (perturbation budget). Submitting creates a queued row in evaluation_jobs and pushes just the job_id onto a Celery/Redis queue — the web request returns instantly, before any actual attack work starts.
3. Celery worker executes — A separate always-running worker process picks up the job, pulls the model and dataset back from S3, preprocesses the data (capped at 500 rows, MinMax-scaled), and wraps the model in an ART classifier interface. For each selected attack, it generates adversarial examples, computes clean vs. robust accuracy, tracks which samples flipped and by how much, and runs ART's own robustness metrics on a small subsample.
4. Persist results — Adversarial examples and the "safe values" (never-flipped samples) are uploaded to S3; per-attack summary metrics and per-sample results are written to Postgres. The job is marked done.
5. Report delivery — The frontend polls the job status every few seconds. Once done, it fetches the full result set as JSON straight from Postgres (no S3 access needed for this) and renders the score, per-attack table, ART metrics, and flipped-sample list. Downloads (safe values, PDF report) are served on-demand via presigned S3 URLs or an in-memory-generated PDF.
6. Optional hardening — Clicking "Harden this model" kicks off a second Celery task that re-runs the pipeline, applies ART's adversarial training to actually update the model's weights, re-evaluates the same attacks against the new model, and uploads a genuinely new, hardened model file to S3.

---

System architecture

At a component level: the Flask app sits between the browser (Jinja2 + vanilla JS) and everything else, handling OAuth (Google/GitHub), routing, and job submission. PostgreSQL holds all structured data — users, model/dataset metadata, job configs, and results. Redis serves double duty as the Celery broker and session store. S3 is the only place raw model/dataset/artifact bytes ever live, always encrypted. The Celery worker is where the actual ML work happens, using IBM's ART to wrap PyTorch/ONNX/sklearn models behind one common attack interface.

The system was deliberately kept modular — for example, all cloud storage access goes through dedicated "worker files," so swapping AWS for Azure or GCP later would mean adding new worker files rather than rewriting the whole pipeline.

---

What it looks like

Landing page

Upload flow — where you upload your model and dataset

Results page — robustness score, per-attack breakdown, flipped samples

PDF report — the same results, exported as a downloadable document

---

How to read the results

• Overall Robustness Score (0–100): a single summary of how much accuracy the model lost across all attacks run, averaged together. Useful as a glance, but a model can score fine overall while being fragile against one specific attack — the per-attack table matters more.
• Clean accuracy vs. Robust accuracy: clean accuracy is the baseline (correct on untouched data); robust accuracy is correctness after the attack. The gap between them is the actual damage done.
• Risk score (per attack): roughly proportional to the accuracy drop that attack caused — useful for comparing attacks against each other on the same model.
• Flipped samples: how many individual predictions changed, in either direction. This can disagree with the accuracy drop — if similar numbers flip each way, they cancel out in the net accuracy number even though a lot of real instability happened underneath.
• Perturbation size (per flipped sample): how much a sample had to change before the model broke. Smaller = worse, since it means less effort was needed to fool it.
• Most perturbed features: which input features the attack leaned on most to cause a flip — a signal for what part of the model's decision boundary is weakest.
• Safe values: the subset of data the model handled correctly no matter what — useful as a starting point for hardening or as a sanity check that the model's core competence is intact.

---

Challenges faced

• ONNX doesn't expose gradients the way PyTorch does, but ART's attacks need gradients to work. Solved this with a finite-differences wrapper (nudging each feature slightly and observing the output change) packaged inside a torch.autograd.Function, so ART could treat the ONNX model like a normal differentiable PyTorch model. It works, but is noticeably slower than attacking a native gradient-based model.
• C&W is expensive since it searches for the minimum perturbation instead of taking a fixed step — had to cap the dataset at 500 samples to keep runtimes reasonable on free-tier infra.
• Flip count vs. accuracy drop was initially confusing: a model can have a large number of flipped predictions while the net accuracy drop looks small, if roughly equal numbers flip right→wrong and wrong→right and cancel out. This wasn't a bug — just two metrics measuring different things — and it's now explained directly on the results page.
• UX feedback from reviewers pushed for a clearer flow (upload → pick attacks → results) and a history page to view past runs; both are implemented now.
• Security of uploaded data: since users upload their own proprietary models, all files are stored with S3 server-side encryption.

---

Limitations

• White-box only — no black-box, query-based, or physical-world attacks
• Tabular data only — no image, sequence, or graph-structured model support
• Sample-capped for speed (attacks: 500 rows; ART metrics like CLEVER: 5–20 points) — results are estimates, not exhaustive guarantees
• No certified/verified robustness — all results are empirical, based on the specific attacks actually run
• Hardening only works for TorchScript models (ONNX and sklearn can't currently be hardened)
• No poisoning, extraction, or inference attack coverage — only evasion (input-time) attacks are handled

---

Conclusion & future scope

This project came out of a genuine feeling that it's wrong for security-critical ML models to be deployed without anyone actually trying to break them first. AO8 makes that testing possible without needing to hand-write attack code every time. Right now it covers the core white-box attacks and produces a usable, readable report end-to-end.

Next steps: more defense techniques, a path toward production-style continuous testing, black-box attacks, and eventually attacks on multi-modal networks.

---

References

• Goodfellow et al., Explaining and Harnessing Adversarial Examples (FGSM): https://arxiv.org/abs/1412.6572
• Madry et al., Towards Deep Learning Models Resistant to Adversarial Attacks (PGD): https://arxiv.org/abs/1706.06083
• IBM Adversarial Robustness Toolbox (ART, open source): https://github.com/Trusted-AI/adversarial-robustness-toolbox