Question about compute requirements and recommended reproduction scale

[dymm9977]

Hi GenLIP authors,

Thank you for releasing this great work. I am interested in reproducing the two-stage GenLIP pretraining setup.

I was wondering if you could share a rough estimate of the compute requirements for reproducing the reported two-stage pretraining, such as the
GPU type/count, GPU memory requirements, and approximate wall-clock time for Stage 1 and Stage 2.

Even an approximate range would be very helpful for planning experiments. Thanks again!

[YanFangCS]

Hi, thanks for your interest in our work.

For the reported two-stage GenLIP pretraining, we used a multi-node training setup on our internal cluster. The exact wall-clock time may vary depending on GPU type, storage / data-loading speed, network bandwidth, and effective batch size, but we are happy to share a rough estimate. Below, we use GenLIP-So16 as an example:

1. GPU type and count

For our reported runs, we used 4 nodes, each equipped with 8 GPUs, i.e., 32 GPUs in total. Each GPU has 80GB memory. Due to internal disclosure constraints, we are unable to provide the exact GPU model here, but GPUs with comparable memory and compute capacity should be suitable for reproduction.

2. GPU memory requirement

The released configuration is designed for 80GB GPUs. In practice, the memory requirement is affected by the per-GPU batch size, the packed sequence length, precision setting, and gradient checkpointing.

If you have similar 80GB GPUs, you may try increasing the per-GPU batch size to 32, which uses around 70GB memory per GPU and better utilizes the available memory. By default, we use a per-GPU batch size of 16, which consumes around 40GB memory per GPU.

Users with fewer GPUs can reduce the per-GPU batch size and use gradient accumulation to keep a similar effective global batch size, but the training time will increase accordingly.

3. Wall-clock time

In our environment, the approximate wall-clock time is:

• Stage 1: around 12 days.
• Stage 2: less than 1 day.

These numbers are only approximate, since the actual speed can be strongly affected by the data-loading pipeline and cluster environment.

4. Suggested reproduction setup

For faithful reproduction, we recommend keeping the global batch size and training schedule close to the released configuration. For debugging or smaller-scale reproduction, users may start with fewer GPUs and a smaller data subset to verify the pipeline first.

Before launching the full training, you may first run a smaller experiment that stops after seeing 1B training samples to verify the training pipeline and configuration, which takes around 40 hours in our environment.

We hope this information is helpful for planning your experiments. Please feel free to let us know if you encounter any other issues.