feat: expose variation ID, SDK status, and harden runtime safety

.github/workflows/code-quality.yaml

@@ -28,7 +28,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install flake8 pytest pytest-mock
+          pip install flake8 pytest pytest-mock build
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
       - name: Lint with flake8
         run: |
@@ -37,3 +37,12 @@ jobs:
       - name: Test with pytest
         run: |
           pytest
+      - name: Audit concurrency, memory, and thread cleanup
+        if: matrix.python-version == '3.12'
+        run: python scripts/resource_audit.py --evaluations 20000 --workers 8
+      - name: Build production distributions
+        if: matrix.python-version == '3.12'
+        run: python -m build
+      - name: Verify wheel contains production packages only
+        if: matrix.python-version == '3.12'
+        run: python -c "import glob, zipfile; names = zipfile.ZipFile(glob.glob('dist/*.whl')[0]).namelist(); assert not any(name.startswith(('tests/', 'featbit_openfeature/')) for name in names)"

MANIFEST.in

@@ -1,4 +1,4 @@
 include requirements.txt
 include README.md
 include dev-requirements.txt
-include release/package.json
+include release/package.json

README.md

@@ -18,7 +18,7 @@ If you want to use your own data source, see [Offline Mode](#offline-mode).
 ## Get Started
 
 ### Installation
-install the sdk in using pip, this version of the SDK is compatible with Python 3.6 through 3.11.
+Install the SDK using pip. This version is compatible with Python 3.6 through 3.12.
 
 ```shell
 pip install fb-python-sdk
@@ -66,6 +66,29 @@ client.stop()
 
 - [Python Demo](https://github.com/featbit/featbit-samples/blob/main/samples/dino-game/demo-python/demo_python.py)
 
+### SDK reliability and live verification
+
+The repository includes a repeatable [reliability review](docs/REVIEW.md)
+covering unit tests, thread safety, memory retention, background-thread
+shutdown, public runtime exception isolation, and redundant code. Run the
+local resource audit with:
+
+```shell
+python scripts/resource_audit.py --evaluations 80000 --workers 8
+```
+
+To verify WebSocket synchronization, remote evaluation, status changes, event
+delivery, and clean shutdown against FeatBit Cloud or a self-hosted evaluation
+service, set `FEATBIT_ENV_SECRET` and run:
+
+```shell
+python scripts/live_integration_check.py
+```
+
+The live script never prints or persists the environment secret. See the
+review document for all supported environment variables and verification
+criteria.
+
 ### FBClient
 
 Applications **SHOULD instantiate a single FBClient instance** for the lifetime of the application. In the case where an application
@@ -112,6 +135,16 @@ if client.update_status_provider.wait_for_OKState():
 
 It's possible to set a timeout in seconds for the `wait_for_OKState` method. If the timeout is reached, the method will return `False` and the client will still be in an uninitialized state. If you do not specify a timeout, the method will wait indefinitely.
 
+You can also observe later connection interruptions and recoveries:
+
+```python
+def on_status_change(state):
+    print(state.state_type, state.error_track)
+
+client.update_status_provider.add_listener(on_status_change)
+# Remove it during application shutdown:
+client.update_status_provider.remove_listener(on_status_change)
+```
 
 > To check if the client is ready is optional. Even if the client is not ready, you can still evaluate feature flags, but the default value will be returned if SDK is not yet initialized.
 
@@ -136,6 +169,7 @@ if client.initialize:
     flag_value = client.variation(flag_key, user, default_value)
     # evaluate the flag value and get the detail
     detail = client.variation_detail(flag_key, user, default=None)
+    print(detail.variation, detail.variation_id, detail.reason)
 ```
 
 If you would like to get variations of all feature flags in a special environment, you can use `fbclient.client.FBClient.get_all_latest_flag_variations`, SDK will return `fbclient.common_types.AllFlagStates`, that explain the details of all feature flags. `fbclient.common_types.AllFlagStates.get()` returns the detail of a given feature flag key.

docs/REVIEW.md

@@ -0,0 +1,108 @@
+# Python Server SDK reliability review
+
+This review covers the production risks requested for the Python Server SDK:
+unit behavior, WebSocket synchronization, event processing, evaluation,
+logging/error isolation, thread safety, shutdown, memory retention, redundant
+code, and live FeatBit service interoperability.
+
+## Review result
+
+The review found and fixed the following concrete issues:
+
+- `RepeatableTask` stored an `Event` in `Thread._stop`, which breaks Python's
+  own `Thread.join()` cleanup path. It now uses a separate stop event and joins
+  deterministically.
+- WebSocket reconnect backoff used an uninterruptible sleep. Client shutdown
+  now interrupts the wait, closes the socket defensively, stops the ping task,
+  and joins the streaming thread.
+- The notice broadcaster mutated its listener registry concurrently. Listener
+  operations now use a lock and dispatch from an immutable snapshot; a queue
+  sentinel makes shutdown immediate and repeatable.
+- The event processor did not retain or join its dispatcher. It now owns the
+  dispatcher lifecycle, joins the periodic flush task, and always completes a
+  synchronous message even if message handling fails.
+- `FBClient.stop()` could expose an extension exception and skip all later
+  cleanup. Shutdown is now idempotent, producer-first, and isolates each
+  component failure.
+- Custom event processor errors could escape from `identify`, `track_*`, or
+  `flush`. Event delivery is now best-effort and cannot fail the application
+  request.
+- Invalid offline bootstrap JSON and unsupported runtime fallback objects could
+  raise from evaluation-related calls. They now return `False` or the supplied
+  fallback, with a diagnostic log message.
+- `Config` and the HTTP helper used mutable default objects. Each client now
+  gets independent HTTP/WebSocket options and headers.
+- A duplicate `FBUser.from_dict()` call in `track_metric` was removed.
+
+Constructor/configuration validation still raises for invalid required
+configuration. This is intentional fail-fast behavior before an SDK client is
+usable; runtime evaluation, event, status, and shutdown paths are isolated.
+
+## Automated evidence
+
+Run from the repository root:
+
+```shell
+python -m pytest
+python -m flake8 .
+python scripts/resource_audit.py --evaluations 80000 --workers 8
+python -m build
+```
+
+Result on Python 3.11.8 (2026-07-29):
+
+- 66 unit/integration tests passed.
+- Flake8 passed.
+- 80,000 evaluations across 8 workers completed with 0 concurrent errors.
+- Throughput was 11,425 evaluations/second on the final audit run.
+- No FeatBit SDK worker threads remained after shutdown.
+- 24,389 traced bytes remained after garbage collection, below the 2 MiB
+  regression threshold.
+- A clean wheel built from the source distribution contained 27 files, no
+  `tests/` package, and no stale `featbit_openfeature/` package.
+
+The tests specifically cover concurrent evaluation, status-listener ordering,
+listener registration/removal under contention, event-processor failure
+isolation, repeated client lifecycle, periodic task joining, and interrupting a
+blocked WebSocket lifecycle.
+
+## Live FeatBit service verification
+
+Use a real environment Server Key without storing it in the repository:
+
+```shell
+export FEATBIT_ENV_SECRET='<server-key>'
+export FEATBIT_FLAG_KEY='python-app-release'
+python scripts/live_integration_check.py
+```
+
+Optional variables are `FEATBIT_STREAMING_URL`, `FEATBIT_EVENT_URL`, and
+`FEATBIT_START_WAIT_SECONDS`. The script verifies:
+
+1. authenticated WebSocket connection and initial data synchronization;
+2. SDK status reaches `OK`;
+3. remote flag evaluation returns value, reason, and variation ID;
+4. feature-flag, identify, and custom metric events enter the delivery path;
+5. explicit flush and final synchronous drain complete;
+6. status changes to `OFF` and no SDK threads remain after close.
+
+The secret is read only from the process environment and is never printed or
+written to disk.
+
+A recorded FeatBit Cloud run on 2026-07-24 reached the WebSocket connected
+state, processed the initial data-sync payload, evaluated the remote
+`python-app-release` flag, exercised the gray-release application under
+concurrent HTTP traffic, and later demonstrated automatic reconnection after a
+remote-host disconnect. The checked-in script turns that one-off validation
+into a repeatable, secret-safe release check.
+
+## Remaining operational considerations
+
+- The SDK is designed as a long-lived singleton, not one client per request.
+- Event delivery is intentionally best-effort so analytics failure cannot
+  affect application availability.
+- A custom `DataStorage`, `EventProcessor`, or `UpdateProcessor` remains
+  responsible for its own internal correctness; the client isolates its
+  runtime and shutdown exceptions.
+- Memory thresholds are regression guards, not universal capacity limits;
+  production sizing should use the application's real flag and segment data.

fbclient/__init__.py

@@ -24,9 +24,7 @@ def get() -> FBClient:
     If you need to create multiple client instances with different environments, instead of this
     singleton approach you can call directly the :class:`fbclient.client.FBClient` constructor.
     """
-    global __config
     global __client
-    global __lock
 
     try:
         __lock.read_lock()
@@ -58,7 +56,6 @@ def set_config(config: Config):
     """
     global __config
     global __client
-    global __lock
 
     try:
         __lock.write_lock()

fbclient/client.py

@@ -68,6 +68,8 @@ def __init__(self, config: Config, start_wait: float = 15.):
             raise ValueError("Config is not valid")
 
         self._config = config
+        self._stop_lock = threading.Lock()
+        self._closed = False
         if self._config.is_offline:
             log.info("FB Python SDK: SDK is in offline mode")
         else:
@@ -79,7 +81,7 @@ def __init__(self, config: Config, start_wait: float = 15.):
         # init components
         # event processor
         self._event_processor = self._build_event_processor(config)
-        self._event_handler = lambda event: self._event_processor.send_event(event)
+        self._event_handler = self._send_event_safely
         # data storage
         self._data_storage = config.data_storage
         # evaluator
@@ -126,6 +128,15 @@ def _build_update_processor(self, config: Config, broadcaster: NoticeBroadcater,
 
         return Streaming(config, broadcaster, update_status_provider, update_processor_event)
 
+    def _send_event_safely(self, event):
+        try:
+            if not self._closed:
+                self._event_processor.send_event(event)
+        except Exception:
+            # Event delivery must never affect the application request that
+            # produced the event, including when a custom processor is used.
+            log.exception('FB Python SDK: event processor failed')
+
     @property
     def initialize(self) -> bool:
         """Returns true if the client has successfully connected to feature flag center.
@@ -154,11 +165,23 @@ def stop(self):
 
         Do not attempt to use the client after calling this method.
         """
-        log.info("FB Python SDK: Python SDK client is closing...")
-        self._data_storage.stop()
-        self._update_processor.stop()
-        self._event_processor.stop()
-        self._broadcaster.stop()
+        with self._stop_lock:
+            if self._closed:
+                return
+            self._closed = True
+            log.info("FB Python SDK: Python SDK client is closing...")
+            # Stop producers before consumers, and isolate every component so
+            # one extension failure cannot prevent the remaining resources
+            # from being released or escape into application shutdown code.
+            for name, component in (
+                    ('update processor', self._update_processor),
+                    ('event processor', self._event_processor),
+                    ('notice broadcaster', self._broadcaster),
+                    ('data storage', self._data_storage)):
+                try:
+                    component.stop()
+                except Exception:
+                    log.exception('FB Python SDK: %s failed to stop' % name)
 
     def __enter__(self):
         return self
@@ -174,9 +197,13 @@ def is_offline(self) -> bool:
     def _get_flag_internal(self, key: str) -> Optional[dict]:
         return self._data_storage.get(FEATURE_FLAGS, key)
 
-    def __handle_default_value(self, key: str, default: Any) -> Tuple[Optional[str], Optional[str]]:
+    def __handle_default_value(self, key: str, default: Any) -> Tuple[Optional[str], Any]:
         default_value = self._config.get_default_value(key, default)
-        default_value_type = simple_type_inference(default_value)
+        try:
+            default_value_type = simple_type_inference(default_value)
+        except Exception:
+            log.warning('FB Python SDK: unsupported default value; returning it unchanged on evaluation failure')
+            return None, default_value
         if default_value is None:
             return None, None
         elif default_value_type == 'boolean':
@@ -235,7 +262,7 @@ def variation(self, key: str, user: dict, default: Any = None) -> Any:
         :param default: the default value of the flag, to be used if the return value is not available
         :return: one of the flag's values in any type in any type of string, bool, float, json
         or the default value if flag evaluation fails
-        :raises: ValueError if the default is not a string, boolean, numeric, or json type
+        Unsupported defaults are returned unchanged if evaluation cannot produce a flag value.
         """
         er = self._evaluate_internal(key, user, default)
         return cast_variation_by_flag_type(er.flag_type, er.value)
@@ -252,7 +279,7 @@ def variation_detail(self, key: str, user: dict, default: Any = None) -> EvalDet
         :param user: the attributes of the user
         :param default: the default value of the flag, to be used if the return value is not available
         :return: an :class:`fbclient.common_types.EvalDetail` object
-        :raises: ValueError if the default is not a string, boolean, numeric, or json type
+        Unsupported defaults are returned unchanged if evaluation cannot produce a flag value.
         """
         return self._evaluate_internal(key, user, default).to_evail_detail
 
@@ -318,7 +345,11 @@ def flush(self):
         schedules the next event delivery to be as soon as possible; however, the delivery still
         happens asynchronously on a thread, so this method will return immediately.
         """
-        self._event_processor.flush()
+        try:
+            if not self._closed:
+                self._event_processor.flush()
+        except Exception:
+            log.exception('FB Python SDK: event processor flush failed')
 
     def identify(self, user: dict):
         """register an end user in the feature flag center
@@ -352,7 +383,6 @@ def track_metric(self, user: dict, event_name: str, metric_value: float = 1.0):
             log.warning('FB Python SDK: user invalid')
             return
 
-        fb_user = FBUser.from_dict(user)
         metric_event = MetricEvent(fb_user).add(Metric(event_name, metric_value))
         self._event_handler(metric_event)
 
@@ -385,10 +415,13 @@ def initialize_from_external_json(self, json_str: str) -> bool:
         :param json_str: feature flags, segments...etc in the json format
         :return: True if the initialization is well done
         """
-        if self._config.is_offline:
-            all_data = json.loads(json_str)
-            if valide_all_data(all_data):
-                version, data = _data_to_dict(all_data['data'])
-                return self._update_status_provider.init(data, version)
+        try:
+            if self._config.is_offline:
+                all_data = json.loads(json_str)
+                if valide_all_data(all_data):
+                    version, data = _data_to_dict(all_data['data'])
+                    return self._update_status_provider.init(data, version)
+        except Exception:
+            log.exception('FB Python SDK: invalid external bootstrap data')
 
         return False