Deadlock under concurrent multithreaded use when DEBUG logging is enabled via setup_logging()

[sdebruyn]

Describe the bug

When driver logging is enabled via mssql_python.setup_logging(output="file") (DEBUG level) and multiple threads concurrently open connections and execute statements (for example a ThreadPoolExecutor with 4 workers), the process permanently deadlocks. It sits at 0% CPU — fully frozen, not merely slow.

The same code with setup_logging not called does not deadlock. It also does not deadlock with a single worker thread. The trigger is therefore the combination of DEBUG logging + concurrency (workers > 1).

Environment

• mssql-python version: 1.10.0
• Python version: 3.12.11
• Operating system: macOS (arm64)
• Authentication: ActiveDirectoryDefault (Entra ID) against a Microsoft Fabric Warehouse. The deadlock is in the logging path and is very likely authentication-independent.

Observed thread stacks

Captured with Python's faulthandler. The dumps were identical across 7 dumps over roughly 7 minutes, which confirms a permanent deadlock rather than slowness.

Thread A (opening a connection):

File ".../mssql_python/connection.py", line 571 in setautocommit
File ".../mssql_python/connection.py", line 410 in __init__
File ".../mssql_python/db_connection.py", line 55 in connect
-> (calls into Python logging) ...
File ".../logging/handlers.py", line 75 in emit
File ".../logging/__init__.py", line 1144 in flush   # blocked here

Thread B (executing a statement, a COPY INTO):

File ".../mssql_python/cursor.py", line 1545 in execute   # blocked in native execute

Server-side check

While the client is frozen, the database server shows no query running for this session. This confirms the block is entirely client-side and is not waiting on the server.

Hypothesized mechanism

mssql-python emits DEBUG log records from within native code paths (for example setautocommit during connect, and cursor.execute) that hold the GIL / internal locks. Under concurrency this creates a lock-acquisition cycle with Python's logging Handler locks across threads:

• Thread A holds the logging handler lock and is blocked in flush.
• Thread B is blocked in native execute and cannot proceed.

This is a classic logging-from-native-under-the-GIL deadlock. It only reproduces with workers > 1.

Steps to reproduce (minimal sketch)

1. Call mssql_python.setup_logging(output="file") to enable DEBUG logging.
2. From N >= 2 threads, concurrently open connections (autocommit=True) and run statements against the same server.

The process deadlocks within seconds to a couple of minutes.

Expected behavior

Enabling DEBUG logging should not affect correctness. Concurrent, multithreaded workloads should run to completion whether or not setup_logging is enabled.

Workaround

Do not enable setup_logging for concurrent / multithreaded workloads.

Impact

This makes the DEBUG logging feature unusable for any multithreaded application (for example parallel data-loading), which is precisely the scenario where enabling DEBUG logging is most valuable for diagnosis.

Possibly related issues

The following existing issues appear related but are distinct from this report:

• GIL is held for the entire duration of cursor.execute(), starving other threads / asyncio event loop #540 — GIL held for the entire duration of cursor.execute() (describes the native-call-holds-GIL behaviour that is part of the hypothesized mechanism here, but is about starvation rather than a permanent deadlock and does not involve logging).
• Running mssql-python on more than 2 threads can lead to hang/deadlocks #321 — multithreaded hang/deadlock with 3+ threads (does not involve logging; closed as non-reproducible).
• hang in connection.cpp at static SqlHandlePtr getEnvHandle() #421 — hang with setup_logging enabled (single-threaded, diagnosed as a driver-load hang on Linux rather than a concurrency logging-lock deadlock).

[github-actions]

Hi @sdebruyn, thank you for opening this issue!

Our team will review it shortly. We aim to triage all new issues within 24-48 hours and get back to you.

If you have additional information to share, please feel free to update the issue.

Thank you for your patience!

[bewithgaurav]

@sdebruyn - thanks for reporting the issue and providing detailed logs

confirmed that this is a real deadlock - reproduced on 1.10.0 (macos arm64) against a local sql server:

• setup_logging(output="file") + a ThreadPoolExecutor with 4 workers opening connections and executing hangs permanently at 0% cpu
• same workload with logging off finishes in ~3s, and a single worker never hangs

so the trigger is exactly what you called out, DEBUG logging + workers > 1.

your read is right, it's a logging-from-native-under-the-GIL lock cycle. native stacks pin it to two spots:

• logger_bridge.cpp, LoggerBridge::log takes a C++ std::mutex and then acquires the GIL. every python thread does the reverse (holds the GIL, then enters native code that logs), so with 2+ threads you get an AB-BA cycle. that mutex isn't needed, python's logging is already thread-safe and the GIL serializes the call.
• connection_pool.cpp, acquireConnection logs "Creating new connection pool" while still holding _manager_mutex, which pulls that mutex into the same cycle. the comment right below it already moved pool->acquire() out of the lock for this exact reason, the log line just got left inside.

from a sample of the hung run: one worker parked inside LoggerBridge::log holding both mutexes plus the GIL and blocked on a python logging lock, another blocked on std::mutex::lock() in acquireConnection, a third blocked in take_gil.

fix is under development, no api change needed. workaround for now is what you found, skip setup_logging for the multithreaded paths. I'll follow up here when the fix is up, thanks again for your support on this