LLMPentest

Chinese version: README_CN.md

Project Structure

• Agent/: our proposed agent scaffolding. It contains the LLM agent, local tools, MCP integrations, and a retained batch entrypoint.
• CVEs/: the target server side. Each CVE-* directory is a runnable vulnerable environment.
• AgentServer/: the base image definition used to run the agent scaffolding.

Note

For safety concerns, we do not provide all the CVEs and configs here, and this respiritory is mainly used to show the structure of the program and how it works.

Quick Start

1. Build the agent scaffolding base image

Run from the repository root:

docker build -f AgentServer/Dockerfile -t agentserver AgentServer

The current AgentServer/Dockerfile installs the latest Metasploit Framework through the official msfinstall script. Our experiments used Metasploit 6.4.102-dev-; for strict reproduction, replace the Metasploit install step in the Dockerfile with the corresponding 6.4.102-dev- version.

This is the base image used later by the single-model test flow.

2. Install host-side Python dependencies

log_monitor.py, the host-side helpers around main.py, and the retained batch_main.py all rely on these Python packages.

pip install -r Agent/requirements.txt

On Windows, make sure both docker and bash are available in PATH.

3. Fill the agent environment config

Before starting log_monitor.py and batch_main.py, fill in Agent/configs/env.yaml:

target_server:
  server_ip: "10.1.2.143"
  server_port: 9000
  attack_ip: "10.1.2.143"

logging:
  server_url: "http://10.1.2.143:8823/log"
  batch_name: "ljq_new_v1_0108"

Field meanings:

• target_server.server_ip: server IP passed into the CVE target server.
• target_server.server_port: server port passed into the CVE target server.
• target_server.attack_ip: attacker IP passed into the CVE target server. In many setups this can be the same as server_ip.
• logging.server_url: log reporting endpoint used by the Agent. It should point to the /log endpoint exposed by log_monitor.py.
• logging.batch_name: log directory name. In the showcase version, you can treat it as a run label.

4. Create the experiment networks

cd CVEs
bash create_pennet.sh

This creates pennet01 through pennet50. The current Agent/batch_main.py uses pennet11 through pennet30.

5. Pre-pull the CVE images

cd CVEs
bash pull_all_images.sh

You can skip this step, since most CVE-* start.sh scripts will try to pull images on demand, but the first run will be slower.

To only list the images:

bash pull_all_images.sh --list

5.1 Start and stop one fixed configuration

This is the recommended workflow for the showcase version.

Each CVE-* directory can now keep fixed compose files such as:

• docker-compose.cve-2025-3248-1a1-config1.yml
• docker-compose.cve-2025-3248-1a1-config2.yml
• docker-compose.cve-2025-3248-3a1-config1.yml
• docker-compose.cve-2025-3248-3a1-config5.yml

The recommended workflow is to start by configuration name instead of passing ten positional parameters:

cd CVEs/CVE-2025-3248
bash start.sh 1A1config1

You can also pass the compose filename or the full container name:

bash start.sh docker-compose.cve-2025-3248-1a1-config1.yml
bash start.sh cve-2025-3248-1a1-config1

Stopping follows the same rule:

cd CVEs/CVE-2025-3248
bash stop.sh 1A1config1

Notes:

• start.sh and stop.sh now prefer matching fixed compose files.
• stop.sh no longer deletes the compose file itself; it only runs docker compose down.
• If you need to override runtime values, you can pass extra arguments after the config name:
  • container_name
  • ipv4_address
  • server_ip
  • server_port
  • attack_ip
  • network_name
  • monitor_log

Example:

bash start.sh 1A1config1 mycontainer 172.16.0.91 10.1.2.143 9000 10.1.2.143 pennet01 ./monitor.txt

If you only want to validate one showcase case, these two steps are usually enough:

cd CVEs/CVE-2025-3248
bash start.sh 1A1config1

Then stop it with:

bash stop.sh 1A1config1

6. Run one model against one CVE configuration

Recommended order:

1. Start one fixed CVE configuration.
2. Start the log receiver.
3. Run one model configuration against that CVE once.

6.1 Start the log monitor

Start it from Agent/, because it relies on relative paths:

cd Agent
python log_monitor.py

By default it listens on 0.0.0.0:8823 and writes logs into ../traj/.

6.2 Prepare a model config

Model config files live under Agent/configs/. Environment parameters are filled in Agent/configs/env.yaml, and the model config template is Agent/configs/model.yaml.

Each config file must contain at least:

api_base_url: "https://your-api-base/v1"
model_name: "your-model-name"
api_key: "sk-xxxx"
conda_base: "/opt/conda/envs/pentest"
toolkit:
  - done

Field meanings:

• api_base_url: base URL of your model service.
• model_name: model name passed into ChatOpenAI.
• api_key: API key for the model service.
• conda_base: Conda path passed into the toolkit. This should match the Conda path inside the scaffolding container.
• toolkit: enabled local tool bundles. The repo currently includes done, while MCP tools are registered automatically by Agent/main.py.

6.3 Run one model + one CVE configuration

Before running this step, make sure:

• the agentserver image has been built;
• the target CVE-* environment is already running;
• log_monitor.py is running;
• you have prepared a model config file such as Agent/configs/qwen3_32B.yaml.

Then run from Agent/:

cd Agent
python main.py <victim_ip> <local_ip> <config_path> <model_name> <cve_number> <cve_config> <service_config>

Example:

cd Agent
python main.py 172.16.0.95 172.16.1.95 configs/qwen3_32B.yaml Qwen3-32B CVE-2025-3248 1A1 supervisord_1.conf

Argument meanings:

• victim_ip: IP of the vulnerable target container.
• local_ip: callback/local IP used by the agent scaffolding side.
• config_path: model config path relative to Agent/.
• model_name: model label recorded in logs.
• cve_number: for example CVE-2025-3248.
• cve_config: for example 1A1 or 3A1.
• service_config: for example supervisord_1.conf.

If your goal is simply to understand how the repository works, this is the main path worth trying first.

7. Optional: Batch entrypoint (batch_main.py)

This section mainly serves the retained batch experiment entrypoint. For most showcase users, it is not the first thing to configure; if you only want to understand the project or run a single case, you can skip it at first.

Location: Agent/batch_main.py

1. DOCKER_NETWORK

Location: Agent/batch_main.py

Purpose: defines the reusable experiment network pool. Each entry contains:

• available: whether the network is currently free.
• target: fixed IP for the vulnerable target container.
• starter_ip: fixed IP for the agent scaffolding container.
• index: scheduler-side index marker.

Notes:

• Every network name listed here must already exist, so run CVEs/create_pennet.sh first.
• MAX_WORKERS should not exceed the number of usable networks.

2. ROOTDIC

Location: Agent/batch_main.py

Purpose: absolute path to the repository root. batch_main.py uses it to:

• locate CVEs/<CVE-ID>/start.sh;
• copy the local Agent/ directory into the scaffolding container.

You must change it to the actual repository path on your machine.

3. MAX_WORKERS

Location: Agent/batch_main.py

Purpose: number of concurrent workers.

For the showcase version, there is usually no reason to set this very high.

4. MODEL_CONFIGS

Location: Agent/batch_main.py

Purpose: mapping from model display name to config file path.

• key: model name shown in logs.
• value: config file path relative to the Agent/ working directory.

5. SERVICE_CONFIGS

Location: Agent/batch_main.py

Purpose: list of supervisord_*.conf files used to control target-side service layouts.

These files live under 1A1/ or 3A1/ inside each CVE-* directory. Each item is combined with task_des to create different experiment conditions.

6. TEST_TIMES

Location: Agent/batch_main.py

Purpose: repeat count for each task_des entry.

7. task_des

Location: Agent/batch_main.py

Purpose: defines which CVEs and which target environments to run.

Each item contains:

• cve_id: target vulnerability directory name.
• cve_config: experiment config directory, currently mainly 1A1 or 3A1.
• test_times: repeat count for that entry, typically reusing TEST_TIMES.

Logs and Outputs

• log_monitor.py listens on 8823 by default.
• Logs are typically written under traj/ at the repository root.
• utils.py includes CVE_NUMBER, MODEL, CVE_CONFIG, and SERVICE_CONFIG in the log payload.
• The batch log directory name comes from logging.batch_name in Agent/configs/env.yaml.

Cleanup

Remove the pennet networks:

cd CVEs
bash remove_pennet.sh

Remove experiment containers created within the last day:

cd Agent
bash docker_remove.sh

License

This project is licensed under the MIT License. See LICENSE for details.