Skip to content

Make camera streaming non-blocking to keep MQTT alive#101

Merged
b-Tomas merged 2 commits into
mainfrom
b-Tomas/fix-camera-stream-mqtt-block
Jun 2, 2026
Merged

Make camera streaming non-blocking to keep MQTT alive#101
b-Tomas merged 2 commits into
mainfrom
b-Tomas/fix-camera-stream-mqtt-block

Conversation

@b-Tomas

@b-Tomas b-Tomas commented Jun 2, 2026

Copy link
Copy Markdown
Member

Your checklist for this pull request

🚨Please review the guidelines for contributing to this repository.

  • Make sure you are requesting to pull a topic/feature/bugfix branch (right side). Don't request your main!
  • Make sure you are making a pull request against the main branch (left side). Also you should start your branch off our main.
  • Check your code additions will fail neither code linting checks nor unit test.

Description

When a camera's RTSP IP is unreachable, stopping the stream dropped the edge-sdk MQTT session and the robot went briefly offline. load_module/unload_module cameras are dispatched synchronously on paho's single MQTT network thread, and CameraStreamer.stop() joined the camera threads — which are stuck in OpenCV's ~30s grab timeout on the bad IP — so the join blocked the MQTT thread past the keepalive.

This replaces the spawn-per-start / join-per-stop CameraStreamer with a single long-lived daemon worker driven by two Events:

  • start() / stop() / shutdown() only toggle Events, so they never block the caller — in particular the MQTT thread.
  • A single worker serializes camera.open() / close(), removing the start/stop race the old blocking join was guarding against.
  • The session body is wrapped so a failed open() / grab() can't kill the worker; it logs, backs off, and waits for the next start().

Supporting changes:

  • OpenCVCamera: capture thread is now a daemon, and close() null-guards the join.
  • RobotSession: register_camera shuts down a replaced streamer; disconnect signals shutdown on all streamers then bounded-joins.

Tests: updated test_robot_session_register_camera for the new lifecycle and added concurrency tests. tox passes.

Note: the camera's own worker thread can still sit in OpenCV's ~30s open on an unreachable IP, but that no longer affects MQTT. Bounding that timeout is left out of this PR.

Demo

Before — stopping the stream blocked the MQTT thread: the session disconnected, in-flight messages failed, and the robot reconnected only after OpenCV timed out:

[ WARN:0@35.701] global cap_ffmpeg_impl.hpp:453 _opencv_ffmpeg_interrupt_callback Stream timeout triggered after 30036.403362 ms
12:15:47 INFO [OpenCVCamera] Waiting for the capture thread to finish (video.py:85)
12:15:47 WARNING [RobotSession] Unexpected disconnection: reason_code '128': Unspecified error. (robot.py:665)
12:15:47 WARNING [RobotSession] There was a problem sending message 170: The client is not currently connected. (robot.py:1236)
12:15:47 WARNING [RobotSession] There was a problem sending message 171: The client is not currently connected. (robot.py:1236)
12:15:47 WARNING [RobotSession] There was a problem sending message 172: The client is not currently connected. (robot.py:1236)
12:15:47 WARNING [RobotSession] There was a problem sending message 173: The client is not currently connected. (robot.py:1236)
12:15:47 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
12:15:48 INFO [RobotSession] Connected to MQTT (robot.py:590)

After — the streaming/capture threads still tear down and OpenCV still errors on the unreachable stream, but the MQTT thread is never blocked: no disconnection, no failed sends, and Robot data published keeps flowing once per second:

[ERROR:0@12.143] global cap_ffmpeg_impl.hpp:1867 retrieveFrame Picture does not contain data
13:58:32 INFO [CameraStreamer] Waiting for the streaming thread to finish (video.py:140)
13:58:32 INFO [OpenCVCamera] Waiting for the capture thread to finish (video.py:85)
13:58:32 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:33 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:34 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:35 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:36 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:37 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
[ERROR:22@18.821] global cap_ffmpeg_impl.hpp:1867 retrieveFrame Picture does not contain data
13:58:38 INFO [inorbit_connector.connector] Robot data published (connector.py:95)
13:58:39 INFO [inorbit_connector.connector] Robot data published (connector.py:95)

Stopping a camera stream ran on the paho MQTT network thread (via the
load/unload module handlers) and joined the camera threads. When the
configured RTSP IP is unreachable, those threads are stuck in OpenCV's
~30s grab timeout, so the join blocked the MQTT thread past the keepalive
and the edge-sdk session dropped (robot briefly offline).

Replace the spawn-per-start / join-per-stop CameraStreamer with a single
long-lived daemon worker driven by two Events:

- start()/stop()/shutdown() only toggle Events -> never block the caller,
  in particular the MQTT thread.
- One worker serializes camera.open()/close(), removing the start/stop
  race that the old blocking join was guarding against.
- The session body is wrapped so a failed open()/grab() can't kill the
  worker; it logs, backs off, and waits for the next start().

OpenCVCamera: capture thread is now a daemon and close() null-guards the
join (the worker's `finally: close()` may run before a thread exists).

RobotSession: register_camera shuts down a replaced streamer; disconnect
signals shutdown on all streamers then bounded-joins (daemon threads
guarantee process exit).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@b-Tomas b-Tomas changed the title Make camera streaming non-blocking to keep MQTT alive (PLATFORM-3136) Make camera streaming non-blocking to keep MQTT alive Jun 2, 2026
@b-Tomas b-Tomas requested a review from leandropineda June 2, 2026 17:05
@b-Tomas b-Tomas self-assigned this Jun 2, 2026

@leandropineda leandropineda left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just nits

Comment thread inorbit_edge/robot.py Outdated
Comment thread inorbit_edge/robot.py
@b-Tomas b-Tomas merged commit ad227eb into main Jun 2, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants