Commit 2026-09-15 00:29 2bca5089

feat(cache): put with native Azure and S3 backends (#43378) This PR reworks the upload side of the cache tool. It prepares CI to upload to more than one storage location. The defaults stay the same: uploads continue to go to the Azure account, and the put-staged invocation in CI works unchanged. Major changes: Upload commands.:

• put-staged is moved to run before the expensive hash-memo build.
• All upload commands share one write path: runPut resolves the destination and dispatches to the selected backend. put-unpacked and the commit commands are removed; nothing in the tool reads a commit file. Backends:
• The new --backend option selects the storage backend on every upload command: azure (the default) or s3. Each backend reads only its own credentials and resolves its own destination
• azure signs with the OIDC bearer token and writes to the chosen --container on the Azure account.
• s3 signs each request with SigV4: a credential pair, an optional session token, and the region MATHLIB_CACHE_S3_REGION. It writes the same container layout ({base}/{container}/{key}) under the bucket endpoint MATHLIB_CACHE_PUT_BASE_URL, of the form https://endpoint/bucket Transfer tools:
• selected by the backend: azure uploads with curl (the built-in tool, as before); s3 uploads with a system rclone when one works on PATH, and with curl otherwise
• rclone receives the S3 credentials through its environment Clear upload boundary.:
• The new Cache/Upload/ modules hold everything that moves staged bytes.
• Upload/Azure.lean and Upload/S3.lean each implement a complete backend: credential resolution, destination, tool choice, and the transfer entry point.
• Upload/Curl.lean and Upload/Rclone.lean have the transfer mechanics
• Upload/Dest.lean the destination contract
• Upload/Defs.lean the backend selection and the destination arbitration Documentation. The README and cache --help document the public surface: reads, staging, and the external-cache interface. The upload commands, options, and variables are documented in the new Cache/CI.md. Behavior changes beyond the new upload path:
• put-staged without --staging-dir exits with code 1. Before, it printed a message and exited 0, so a misconfigured CI job reported success and uploaded nothing.
• An upload without --container (and without MATHLIB_CACHE_PUT_URL) errors and names the known containers. Before, it defaulted to the bare mathlib4 container, so a misconfigured job wrote to the lowest-trust namespace.
• A scope that is not a hex commit SHA causes an error. Before, a path-shaped MATHLIB_CACHE_REPO_SCOPE could reach URL paths and a local temp-file name.
• MATHLIB_CACHE_SAS is retired: the tool does not read it. CI mints OIDC bearer tokens and the broker mints S3 pairs; nothing sets a SAS token.