Download P3 tiles onto the Canada Alliance cluster (Nibi)

[cailmdaley]

Goal

Stage the P3 patch tiles (image + weight) on the new Canada Alliance cluster as a manual bulk download, reusing ShapePipe's on-the-fly retrieval mechanism rather than running it inside the pipeline.

How ShapePipe does it (what we're yoinking)

The get_images module (get_images.py ~L323–357) downloads tiles from CADC VOSpace via vcp (the vos python package, wrapped by cs_util.canfar.vosHandler), with configurable retries (default 3).

• Remote paths: vos:cfis/tiles_DR6/CFIS.XXX.YYY.r.fits + vos:cfis/tiles_DR6/CFIS.XXX.YYY.r.weight.fits.fz
• Auth: CADC proxy certificate at $HOME/.ssl/cadcproxy.pem, passed as --certfile=... (generate with cadc-get-cert)
• Tile list: auxdir/CFIS/tiles_202106/tiles_P3.txt in the shapepipe repo — 1268 tiles, one XXX.YYY ID per line. (Newer list versions exist under auxdir/CFIS/tiles_202510/ and tiles_202604/ — confirm which vintage P3 definition we want before launching.)

Plan

Essentially a certificate + a loop:

for tile_id in $(cat tiles_P3.txt); do
  for f in "CFIS.${tile_id}.r.fits" "CFIS.${tile_id}.r.weight.fits.fz"; do
    vcp --certfile=$HOME/.ssl/cadcproxy.pem "vos:cfis/tiles_DR6/$f" "$DEST/$f"
  done
done

with retry-on-failure and a completeness check at the end (2536 files expected: 1268 × {image, weight}).

Tasks

• vos package installed on the Alliance cluster (pip or container)
• Valid CADC proxy cert on the cluster
• Confirm tile-list vintage (202106 vs 202510/202604) and DR (tiles_DR6?)
• Download script with retries + logging (adapt the loop above, or drive get_images_runner standalone via example/cfis/config_tile_Git_vos.ini as template)
• Run + verify completeness (file count, sizes, fpack integrity spot-check)

~1268 tiles × O(0.5 GB each) → order 1–2 TB; pick the destination filesystem accordingly.

[martinkilbinger]

Ana Ennis has downloaded tiles to nibi; tile images are in $HOME/projects/def-mjhudson/anaennis/data/{raw_outside_expected_hold,processed_tiles} . There are 7300 tiles.

[cailmdaley]

great! so we can close this?

[martinkilbinger]

great! so we can close this?

don't know whether P3 tiles are included in the 7300 from Ana.

[martinkilbinger]

@cailmdaley can you make the unions_wl folder a+rwx?

[cailmdaley]

sure, i will look into what Ana has, and perhaps symlink her files into unions_wl so we can keep using the latter as our data directory.

[cailmdaley]

I audited what Ana has on Nibi (/project/6001537/anaennis/data), cross-matched against auxdir/CFIS/tiles_202106/tiles_P3.txt (byte-identical to the 202510 vintage, so no ambiguity there):

What's there — 8482 unique DR6 tiles (processed_tiles: 7307, raw_outside_expected_hold: 1175, disjoint). Each tile directory holds CFIS.XXX.YYY.r.fits (all 8482 present) plus u.fits, and WHIGS g / WISHES z coadds where they exist at CADC. Source is vos:cfis/tiles_DR6/ per the download logs.

P3 coverage — 671 / 1268 P3 tiles present; 597 missing.

Weight maps — none; Ana downloaded images only.

The gap is now downloading: all 1268 P3 *.r.weight.fits.fz + the 597 missing P3 r-images (~0.4 TB) are being fetched from vos:cfis/tiles_DR6/ directly into unions_wl/tiles/ (script + logs in the repo dir). Once complete + verified, P3 will be fully staged and this issue can close.

unions_wl is now structured and opened up (a+rwx as requested):

unions-wl/
  README.md       # layout, provenance, coverage, caveats
  tiles/          # flat ShapePipe-style layout: CFIS.XXX.YYY.r.fits (symlinks → Ana's files)
                  # + CFIS.XXX.YYY.r.weight.fits.fz (real files, arriving)
  tile-lists/     # tiles_P3.txt, tiles_P3_present.txt (671), tiles_P3_missing.txt (597), tiles_present_all.txt
  scripts/, logs/ # download machinery

Caveat: Ana's files are group-readable (def-mjhudson) only, so the image symlinks resolve only for group members.

— Claude (Fable) on behalf of Cail

[cailmdaley]

@AstroMike thank you for giving us access to this machine! The project quota is at 25/27 TB, and I'm estimating that all the data needed for a full shapepipe run would be ~10 TB. So we'll probably have to do the shapepipe downloading on-the-fly unless we could increase the quota or free up 10 TB. There is 10 TB free on the /nearline tape-based storage if anything is archivable.

[martinkilbinger]

argh I have also started to download the tiles...

[AstroMike]

I think Ana has finished her run . I will ask her to see if she can move non-r band tiles to near line Mike On Jul 11, 2026, at 14:13, Cail McLean Daley ***@***.***> wrote:  [https://avatars.githubusercontent.com/u/13065208?s=20&v=4]cailmdaley left a comment (CosmoStat/shapepipe#801)<#801 (comment)> @AstroMike<https://github.com/AstroMike> thank you for giving us access to this machine! The project quota is at 25/27 TB, and I'm estimating that all the data needed for a full shapepipe run would be 10-15 TB. So we'll probably have to do the shapepipe downloading on-the-fly unless we could increase the quota or free up 10+ TB. There is also 10 TB free on the /nearline tape-based storage if anything is archivable. — Reply to this email directly, view it on GitHub<#801?email_source=notifications&email_token=ABKCQ3VUUEX64MJJF4QDHAL5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4945644325>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKCQ3XPAACR6TDVX2AH2C35EIVNRAVCNFSNUABFKJSXA33TNF2G64TZHMZDMMJUHA3DONBQHNEXG43VMU5TIOBUG44DANBUHAY2C5QC>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/ABKCQ3SES7MUEM444AKFVVT5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/ABKCQ3RJPUQWTF5K6BTRI4D5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[AstroMike]

For now I would try to just download a subset to see if the scripts run Mike On Jul 11, 2026, at 16:09, Mike Hudson ***@***.***> wrote:  I think Ana has finished her run . I will ask her to see if she can move non-r band tiles to near line Mike On Jul 11, 2026, at 14:13, Cail McLean Daley ***@***.***> wrote:  [https://avatars.githubusercontent.com/u/13065208?s=20&v=4]cailmdaley left a comment (CosmoStat/shapepipe#801)<#801 (comment)> @AstroMike<https://github.com/AstroMike> thank you for giving us access to this machine! The project quota is at 25/27 TB, and I'm estimating that all the data needed for a full shapepipe run would be 10-15 TB. So we'll probably have to do the shapepipe downloading on-the-fly unless we could increase the quota or free up 10+ TB. There is also 10 TB free on the /nearline tape-based storage if anything is archivable. — Reply to this email directly, view it on GitHub<#801?email_source=notifications&email_token=ABKCQ3VUUEX64MJJF4QDHAL5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4945644325>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABKCQ3XPAACR6TDVX2AH2C35EIVNRAVCNFSNUABFKJSXA33TNF2G64TZHMZDMMJUHA3DONBQHNEXG43VMU5TIOBUG44DANBUHAY2C5QC>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/ABKCQ3SES7MUEM444AKFVVT5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/ABKCQ3RJPUQWTF5K6BTRI4D5EIVNRA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIOJUGU3DINBTGI22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>