[SYNPY-1869] Add SearchIndex entity + search-management APIs; rename SchemaOrganization to Organization

[BryanFauble]

Problem:

Two related gaps in the OOP model layer:

1. SearchIndex / search-management surface was incomplete. The SearchManagementController in Synapse-Repository-Services exposes a family of resources — SearchIndex (an OpenSearch index built from a defining SQL query), plus the configuration resources that control how indexes are built: TextAnalyzer, ColumnAnalyzerOverride, SynonymSet, SearchConfiguration, and SearchConfigBinding — as well as index query and autocomplete endpoints. The Python client only had a partial, not-fully-correct starting point for these.

2. SchemaOrganization naming was inconsistent. The model that holds JSON-schema resources was named SchemaOrganization, but conceptually it's just an "Organization" that holds resources. The name should read consistently.

Solution:

Search management (new):

• Added the SearchIndex entity model (models/search_index.py) following the async-first @async_to_sync pattern — store/get/delete/autocomplete, columns derived read-only from defining_sql, registered concrete type org.sagebionetworks.repo.model.search.table.SearchIndex.
• Added the search-management resource models (models/search_management.py): TextAnalyzer, ColumnAnalyzerOverride, SynonymSet, SearchConfiguration, SearchConfigBinding, plus query/autocomplete request+hit types.
• Added the REST API layer (api/search_services.py) and wired it into api/__init__.py.
• Registered the new entity in concrete_types.py, entity_factory.py, mixins/asynchronous_job.py, and marked SearchIndex read-only-schema in mixins/table_components.py.

Rename (hard, no back-compat alias):

• SchemaOrganization → Organization (class + SchemaOrganizationProtocol → OrganizationProtocol), module helper list_json_schema_organizations → list_organizations.
• Renamed files via git mv to preserve history: models/schema_organization.py → models/organization.py, the unit/integration test modules, and docs/reference/experimental/{async,sync}/schema_organization.md → organization.md; updated mkdocs.yml nav.
• Updated all references across models/__init__.py, operations/{store,delete}_operations.py, mixins, curator extension, and the deprecated services/json_schema.py docstring pointers.
• Note: the legacy public JsonSchemaOrganization class in services/json_schema.py is intentionally left unchanged — it is a separate deprecated API, not the model being renamed.

Testing:

• New async unit tests for the search-management models and Organization; existing store/delete/factory routing unit tests updated for the rename.
• Focused unit run green: 142 passed across json_schema services, organization, search-management, and store/delete operations.
• Integration tests added for SearchIndex (create/retrieve, match-all query, autocomplete, delete) — kept minimal to limit CI/Synapse-server load; index-creation tests skip gracefully on a 403 where creation is restricted.
• ruff and black clean on all changed files (the 2 remaining F841 flags in the renamed org test module and the json_schema tutorial are pre-existing debt, left untouched per surgical-change scope).

[Copilot]

Pull request overview

Adds first-class OOP support for Synapse search-management resources and the new SearchIndex entity, and completes the rename of the JSON-schema “SchemaOrganization” model to Organization across the SDK surface (models, operations routing, docs, and tests).

Changes:

• Introduces SearchIndex entity model plus org-scoped search-management models (analyzers/config/binding) and REST API wrappers.
• Wires SearchIndex into entity factory/concrete types/async-job routing and container sync/walk surfaces.
• Renames SchemaOrganization → Organization across models, operations, docs, and test suites.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/synapseutils/unit_test_synapseutils_sync.py	Extends include-types coverage to include searchindex in sync/walk tests.
tests/unit/synapseclient/operations/unit_test_store_operations.py	Updates store routing tests for Organization rename.
tests/unit/synapseclient/operations/unit_test_delete_operations.py	Updates delete routing tests for Organization rename.
tests/unit/synapseclient/models/async/unit_test_search_management_async.py	Adds unit tests for search-management dataclasses and query/autocomplete flows.
tests/unit/synapseclient/models/async/unit_test_project_async.py	Adds coverage for project sync populating searchindexes.
tests/unit/synapseclient/models/async/unit_test_organization_async.py	Renames/updates unit tests for Organization and JSONSchema behaviors.
tests/unit/synapseclient/extensions/test_schema_management.py	Updates extension tests to patch organization.JSONSchema.
tests/integration/synapseclient/test_command_line_client.py	Updates CLI integration fixture to use Organization.
tests/integration/synapseclient/operations/async/test_factory_operations_store_async.py	Updates integration store/delete coverage for renamed Organization.
tests/integration/synapseclient/models/async/test_search_management_async.py	Adds integration coverage for org-scoped search-management resources and binding lifecycle.
tests/integration/synapseclient/models/async/test_search_index_async.py	Adds integration coverage for SearchIndex create/query/autocomplete/delete flows.
tests/integration/synapseclient/models/async/test_organization_async.py	Updates integration tests to Organization and list_organizations.
tests/integration/synapseclient/extensions/curator/test_schema_management.py	Updates curator integration tests to use Organization.
synapseclient/services/json_schema.py	Updates deprecated-service docstrings to point to models.Organization.
synapseclient/operations/store_operations.py	Updates operations routing/allowed-types messaging for Organization.
synapseclient/operations/delete_operations.py	Updates delete operations routing/docs for Organization.
synapseclient/models/search_management.py	New: search-management models (analyzers, configs, bindings, query DTOs) and dispatch helpers.
synapseclient/models/search_index.py	New: SearchIndex entity model with autocomplete support and table-like mixins.
synapseclient/models/protocols/search_management_protocol.py	New: sync protocol for SearchConfigBinding (async-to-sync generated methods).
synapseclient/models/protocols/search_index_protocol.py	New: sync protocol for SearchIndex (async-to-sync generated methods).
synapseclient/models/project.py	Adds searchindexes collection to Project.
synapseclient/models/organization.py	Renamed model file and class: SchemaOrganization → Organization; helper list_organizations.
synapseclient/models/mixins/table_components.py	Marks SearchIndex as read-only schema in table mixin logic.
synapseclient/models/mixins/storable_container.py	Extends container sync/walk to recognize and hydrate SearchIndex children.
synapseclient/models/mixins/asynchronous_job.py	Adds async-job route for SEARCH_INDEX_QUERY.
synapseclient/models/mixins/access_control.py	Extends entity-type mapping to include searchindex.
synapseclient/models/folder.py	Adds searchindexes collection to Folder.
synapseclient/models/init.py	Exports Organization, SearchIndex, and search-management model surface.
synapseclient/extensions/curator/schema_management.py	Updates curator extension imports for renamed Organization module.
synapseclient/core/constants/concrete_types.py	Adds concrete types for SearchIndex and SearchIndexQuery.
synapseclient/api/search_services.py	New: REST API functions for search-management resources and autocomplete.
synapseclient/api/entity_services.py	Adds searchindex to include-types example list.
synapseclient/api/entity_factory.py	Registers SearchIndex in concrete-type → model dispatch.
synapseclient/api/init.py	Exports search services in API surface.
mkdocs.yml	Updates docs nav to Organization reference pages.
docs/tutorials/python/tutorial_scripts/json_schema.py	Updates tutorial script imports/usage to Organization.
docs/reference/json_schema.md	Updates deprecation notice to refer to Organization.
docs/reference/experimental/sync/organization.md	Renamed reference page to Organization and updates mkdocstrings target.
docs/reference/experimental/async/organization.md	Renamed reference page to Organization and updates mkdocstrings target.

Comments suppressed due to low confidence (4)

synapseclient/models/search_management.py:1018

• SearchConfigBinding.get_async() has the same potential issue as store_async(): if object_id on the instance is un-prefixed (as returned by the service), this will build an invalid /entity/{entityId}/... URI. Normalize to a Synapse ID when calling the API.

        if not self.object_id:
            raise ValueError("SearchConfigBinding must have an object_id set.")
        result = await get_search_config_binding(
            self.object_id, synapse_client=synapse_client
        )
        return self.fill_from_dict(result)

synapseclient/models/search_management.py:1047

• SearchConfigBinding.delete_async() can also be called on an instance whose object_id came from fill_from_dict() (and may not include the syn prefix). Normalize to a Synapse ID when calling the clear endpoint.

        if not self.object_id:
            raise ValueError("SearchConfigBinding must have an object_id set.")
        await clear_search_config_binding(self.object_id, synapse_client=synapse_client)

synapseclient/models/organization.py:355

• Grammar in docstring: use “an Organization” (not “a Organization”).

This issue also appears on line 374 of the same file.
synapseclient/models/organization.py:374

• Grammar in docstring: use “an Organization” (not “a Organization”).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[andrewelamb]

The rename of SchemaOrganization -> Organization is a breaking change. SchemaOrganization should remain as-is for now, but be deprecated and point to the new Organization class.

[andrewelamb]

These tests fail:

    async def test_store_search_index_routes_to_table_handler(self):
        """Test that SearchIndex routes through the table entity handler.

        SearchIndex is a table-like entity (registered in entity_factory,
        concrete_types, and sync_from_synapse) and should be storable through the
        functional store() API just like its peers MaterializedView and
        VirtualTable. This currently fails because SearchIndex was not added to the
        isinstance dispatch in store_async, so it falls through to the else branch
        and raises "Unsupported entity type: SearchIndex".
        """
        # GIVEN a mock SearchIndex entity
        from synapseclient.models import SearchIndex

        mock_si = SearchIndex(name="Test SI", parent_id="syn123")
        mock_si.store_async = AsyncMock(return_value=mock_si)

        # WHEN I call store_async
        result = await store_async(entity=mock_si, synapse_client=MagicMock())

        # THEN store_async is called
        mock_si.store_async.assert_awaited_once()
        assert result is mock_si

    async def test_delete_search_index_entity(self):
        """Test that a SearchIndex entity is deleted via delete_async.

        SearchIndex is a table-like entity and should be deletable through the
        functional delete() API just like its peers MaterializedView and
        VirtualTable. This currently fails because SearchIndex was not added to the
        isinstance dispatch in delete_async, so it falls through to the else branch
        and raises "Unsupported entity type: SearchIndex".
        """
        # GIVEN a mock SearchIndex entity
        from synapseclient.models import SearchIndex

        mock_si = SearchIndex(id="syn123456")
        mock_si.delete_async = AsyncMock(return_value=None)

        # WHEN I call delete_async
        result = await delete_async(entity=mock_si, synapse_client=MagicMock())

        # THEN delete_async is called
        mock_si.delete_async.assert_awaited_once()
        assert result is None

E           ValueError: Unsupported entity type: SearchIndex. Supported types are: str (Synapse ID), CurationTask, Dataset, DatasetCollection, EntityView, Evaluation, File, Folder, Grid, JSONSchema, MaterializedView, Project, RecordSet, Organization, SubmissionView, Table, Team, VirtualTable.

synapseclient/operations/delete_operations.py:599: ValueError
========================================================================================== short test summary info ===========================================================================================
FAILED tests/unit/synapseclient/operations/unit_test_delete_operations.py::TestDeleteTableLikeEntityRoute::test_delete_search_index_entity - ValueError: Unsupported entity type: SearchIndex. Supported types are: str (Synapse ID), CurationTask, Dataset, DatasetCollection, EntityView, Evaluation, File, Folder, Grid, JSONSchema, MaterializedVi...

[linglp]

@andrewelamb those tests failed because searchIndex was not added to operations/ layer. I could create an additional ticket for this: https://sagebionetworks.jira.com/browse/SYNPY-1896

[andrewelamb]

@andrewelamb those tests failed because searchIndex was not added to operations/ layer. I could create an additional ticket for this: https://sagebionetworks.jira.com/browse/SYNPY-1896

Shouldn't they be part of this ticket?

[linglp]

@andrewelamb those tests failed because searchIndex was not added to operations/ layer. I could create an additional ticket for this: https://sagebionetworks.jira.com/browse/SYNPY-1896

Shouldn't they be part of this ticket?

This ticket already has a lot going on, including deprecating SchemaOrganization. If I could just get these changes cross the finish line and worry about other improvements later, that would greatly help.