Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions clients/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,54 @@ existing_parts = resumed.list_parts()
key = resumed.complete(new_parts + existing_parts)
```

### Many API (batch operations)

The Many API lets you enqueue multiple operations that the client executes using
Objectstore's batch endpoint, minimizing network overhead. Operations too large
to batch (oversized inserts) automatically fall back to individual requests, and
batch and individual requests run concurrently.

`send()` returns an `OperationResults` list. Results are **not** guaranteed to be
in the order they were enqueued, since operations execute concurrently. Each
result is a `GetResult`, `PutResult`, `DeleteResult`, `HeadResult`, or (for errors
that can't be attributed to a specific operation) an `ErrorResult`.

```python
from objectstore_client import Client, Usecase
from objectstore_client.many import GetResult, PutResult

client = Client("http://localhost:8888")
session = client.session(Usecase("attachments"), org=42, project=1337)

results = (
session.many()
.put(b"file1 contents", key="file1")
.put(b"file2 contents", key="file2")
.get("file3")
.delete("file4")
.send()
)

for result in results:
if result.error is not None:
... # handle per-operation error
elif isinstance(result, GetResult):
payload = result.response.payload if result.response else None # None => 404
elif isinstance(result, PutResult):
stored_key = result.key
```

If you don't need to inspect individual results and just want to raise if any
operation failed, use `raise_for_failures()`, which raises an `ExceptionGroup` of
all per-operation errors:

```python
session.many().put(b"file1", key="file1").put(b"file2", key="file2").send().raise_for_failures()
```

Concurrency is configurable via `.max_individual_concurrency(n)` (default 5) and
`.max_batch_concurrency(n)` (default 3).

### Authentication

If your Objectstore instance enforces authorization, you must configure authentication
Expand Down
8 changes: 8 additions & 0 deletions clients/python/docs/objectstore_client.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,14 @@ objectstore\_client.errors module
:show-inheritance:
:undoc-members:

objectstore\_client.many module
-------------------------------

.. automodule:: objectstore_client.many
:members:
:show-inheritance:
:undoc-members:

objectstore\_client.metadata module
-----------------------------------

Expand Down
22 changes: 21 additions & 1 deletion clients/python/src/objectstore_client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from collections.abc import Mapping, Sequence
from dataclasses import asdict, dataclass
from io import BytesIO
from typing import IO, Any, Literal, NamedTuple, cast
from typing import IO, TYPE_CHECKING, Any, Literal, NamedTuple, cast
from urllib.parse import urlparse

import sentry_sdk
Expand Down Expand Up @@ -32,6 +32,9 @@
from objectstore_client.multipart import MultipartUpload
from objectstore_client.scope import Scope

if TYPE_CHECKING:
from objectstore_client.many import ManyBuilder


class GetResponse(NamedTuple):
metadata: Metadata
Expand Down Expand Up @@ -275,6 +278,23 @@ def _make_url(self, key: str | None, full: bool = False) -> str:
return f"http://{self._pool.host}:{self._pool.port}{path}"
return path

def _make_batch_url(self) -> str:
relative_path = f"/v1/objects:batch/{self._usecase.name}/{self._scope}/"
return self._base_path.rstrip("/") + relative_path

def many(self) -> ManyBuilder:
"""
Creates a [ManyBuilder] associated with this session.

A [ManyBuilder] enqueues multiple operations, which the client sends as
batch requests via a dedicated endpoint (minimizing network overhead),
falling back to individual requests for operations too large to batch.
"""
# Imported lazily to avoid a circular import at module load time.
from objectstore_client.many import ManyBuilder

return ManyBuilder(self)

def _make_multipart_url(
self,
action: str | None,
Expand Down
Loading
Loading