Updating authenticators from latest in Tiled by davidpcls · Pull Request #81 · bluesky/bluesky-httpserver

davidpcls · 2026-01-29T20:29:31Z

Description

These changes bring in the authenticators from Tiled 0.2.12 into bluesky-httpserver along with the required changes to the app and supporting files to support them. This allows us to use the same authentication setup as from Tiled deployments (at least at the current moment). The main other change is to remove the "mode" flag and instead use the class type to determine if it is an internal or external authenticator.

Part of this work is also the updating of the bluesky-queueserver-api, for which I have created a PR. Both are required in order for the OIDC workflows to work.

Like Tiled, the authentication flow forces a login prompt on each attempt to login. I have tried to minimize any differences between similar functions/files in Tiled and bluesky-httpserver.

Motivation and Context

This solves the problem of having different authentication schemes to maintain between Tiled and HTTP server, which came from the same code around 3 years ago. Tiled has been updated but HTTP server was not. This addresses that.

Summary of Changes for Release Notes

Updated authenticators based off Tiled v0.2.12.
Made minimal changes to app.py and authentication.py to support the changes. Some new endpoints needed to be created to do this, but fairly minimal. The majority of the work was getting the unit test runners stable.
Added local parallel unit test runners to aid development.
Updated the github runners to be more stable and reliable by making the port used dynamic.

Fixed

Added

Authenticators.py from Tiled v0.2.12
protocols.py from Tiled v0.2.12
new endpoints for authorization workflows in app.py
Pending sessions in the database
Documentation and examples on using OIDC based off Tiled

Changed

Unit tests to support above changes
Unit test structure to be a little more stable
The LDAP container as the previous one was no longer supported

Removed

How Has This Been Tested?

Testing was done against MS Entra. I tested that these workflows work for both internal (localhost) and external servers. Using this the login workflow changes to just simply being:

from bluesky_queueserver_api.http import REManagerAPI
from bluesky_queueserver_api import BPlan
RM = REManagerAPI(http_server_uri="http://localhost:60610", http_auth_provider="entra/authorize")
RM.login()

While this was tested with MS Entra, the design is based on Tiled's OIDC and testing used the OIDC flow with MS Entra, so it should theoretically work with all other authenticators using OIDC.

TODO:

Migrate in changes from tiled 0.2.10 to 0.2.12
Update unit tests
Update documentation for authenticators

danielballan · 2026-02-03T15:34:55Z

Wow, thanks @davidpcls!

dmgav · 2026-02-03T16:50:50Z

Thank you. I am going to try to fix the unit tests first, I may push a few commits.

prjemian · 2026-02-03T17:40:01Z

If either of you intend to commit more work to this PR, put it in Draft mode. Change the mode back when you are ready for final review.

davidpcls · 2026-02-03T17:42:15Z

@prjemian , thanks I've converted it.

@dmgav sure that sounds good.

dmgav · 2026-02-05T16:53:03Z

I fixed the unit tests. I also rebased the branch to main, so I had to force push the changes.

davidpcls · 2026-02-17T16:00:15Z

Thanks for working on that @dmgav! I just got things setup so I can test with this against MS Entra, so I'll go about verifying there are no other changes required for this to work properly. Took a little longer than expected to get the Entra stuff worked out.

This is working okay, although it doens't really work smoothly for the API based login and the http command based login isn't great, as it requires the user to copy and past token around. Compared to ldap which just logs the user in. So still some work to do here to smooth out the user experience.

This solves the problem that what was implemented was actually authenticating the application and not the user like expected. It worked but it required that the user input a code. This solves that problem so that when you click the login link, if you are already logged in with you SSO provider you'll just automatically log in to the HTTP Server. Likewise if you use the bluesky queueserver api, when you call RM.Login you'll automatically be logged in, no user interaction required.

These should correct some of the problems in the last CI workflow. I moved the LDAP and docker image into the continuous_integration folder so it matches tiled.

This addresses documentation problems, the levels were incorrect as I did not understand what the next level should have been in the docs. I've also updated the usage documentation a little to be more useful.

These allow for running the unit tests in a containerized system just like how they are done in the ci pipeline, but locally and in a way that can maximize processor usage and minimize runtime.

Testing with ms entra

davidpcls · 2026-03-13T18:05:31Z

I've added in these changes, I'm going to work on cleaning up failing unit tests now and then I will update the pull request description to match your style and provide information on testing. After that I will remove the WIP status.

davidpcls · 2026-03-13T18:06:49Z

Please also see the related PR for the API: bluesky/bluesky-queueserver-api#62

This is a set of test changes intended to improve the reliability of unit testing, as the current unit tests are randomly failing due to test design. Primarily this appears to be centered around LDAP. So this work was to: * Fix for ldap errors * Hardening unit tests so they fail less frequency * Try to handle console output more reliably

davidpcls · 2026-03-20T19:59:02Z

Sorry, got excited about the unit tests passing. I need to do one last integration test first, which is why I converted it back into a WIP

davidpcls · 2026-03-20T20:41:16Z

@dmgav , I've now tested this on remote machines and with httpserver running locally and both work. Let me know if you want an example. I'm not adding a whole lot here for evidence just because I'm not entirely sure what could be leaking security-sensitive information out.

danielballan · 2026-05-22T16:52:46Z

@davidpcls With Tiled v0.2.10 (released this morning) we addressed several issues in OIDC authentication. I think it would make sense to incorporate those fixes in this PR, so that QS will be fully caught up. Do you have capacity to take that on?

See the diff in tiled.authenticators and tiled.server.authentication between v0.2.9 and v0.2.10.

davidpcls · 2026-05-25T16:19:39Z

@danielballan, I'll take a look and see if it's something I can do. The unit tests are usually the hard part here as they take a long time to run, if anyone wants to tackle improving that it would help immensely.

checkmarx-gh-ast-us-povs · 2026-05-25T19:24:40Z

Checkmarx One – Scan Summary & Details – a0210fce-3a46-44d9-b2ce-56dfc68132b2

New Issues (6)

Checkmarx found the following issues in this Pull Request

#	Issue	Source File / Package	Checkmarx Insight
1	Passwords And Secrets - Generic Password	/ldap-docker-compose.yml: 10	details Query to find passwords and secrets in infrastructure code. `ID: 0uORs8s6P6h1jOJEMOa3uIr6F5k%3D`
2	Container Capabilities Unrestricted	/ldap-docker-compose.yml: 2	details Some capabilities are not needed in certain (or any) containers. Make sure that you only add capabilities that your container needs. Drop unnec... `ID: j%2Fx7zqyqeGV5OT6YextuGFOrQx0%3D`
3	Container Traffic Not Bound To Host Interface	/ldap-docker-compose.yml: 4	details Incoming container traffic should be bound to a specific host interface `ID: 4cqgOfD5LWmOrO%2FoHwOJyK90ZSc%3D`
4	Healthcheck Not Set	/ldap-docker-compose.yml: 2	details Check containers periodically to see if they are running properly. `ID: SbI1wM7bsqAma%2BzrlWGZKGM%2F%2BH0%3D`
5	Privileged Ports Mapped In Container	/ldap-docker-compose.yml: 4	details Privileged ports (1 to 1023) should not be mapped. Also you should drop net_bind_service linux capability from the container unless you absolu... `ID: hpFU1vm8AZBeQTJYYs4wkm%2Foy2E%3D`
6	Security Opt Not Set	/ldap-docker-compose.yml: 2	details Attribute 'security_opt' should be defined. `ID: 9gd0q%2Ft8iSnz4NBVRq7X6%2BeSzQc%3D`

Fixed Issues (9)

Great job! The following issues were fixed in this Pull Request

Severity	Issue	Source File / Package
	~~Passwords And Secrets - Generic Password~~	/ldap-docker-compose.yml: 11
	~~Container Capabilities Unrestricted~~	/ldap-docker-compose.yml: 4
	~~Container Traffic Not Bound To Host Interface~~	/ldap-docker-compose.yml: 6
	~~Healthcheck Not Set~~	/ldap-docker-compose.yml: 4
	~~Memory Not Limited~~	/ldap-docker-compose.yml: 4
	~~Pids Limit Not Set~~	/ldap-docker-compose.yml: 4
	~~Security Opt Not Set~~	/ldap-docker-compose.yml: 4
	~~Use_Of_Hardcoded_Password~~	bluesky_httpserver/authentication.py: 59
	~~Cpus Not Limited~~	/ldap-docker-compose.yml: 4

Communicate with Checkmarx by submitting a PR comment with @Checkmarx followed by one of the supported commands. Learn about the supported commands here.

davidpcls · 2026-05-28T20:27:01Z

@danielballan, I think this is going to take a little bit, as I won't be able to work on this for a couple weeks and I've not made huge progress so far. If you need to get this going I'd suggest someone else take over integrating in the latest from Tiled. If not I'll work on it later.

davidpcls · 2026-06-23T18:02:38Z

I'm looking at this right now, trying to move to a common auth package instead so we can limit the amount of churn here. Should also make unit tests easier to write.

davidpcls · 2026-07-14T21:33:46Z

I am still working on this. I had stared on moving to a common auth package but that ballooned the changes so much that it didn't make sense to me. Instead I'm just completing this task and making anything I do match Tiled as much as possible. Later tasks I think should involve making Bluesky-HTTPServer and Tiled look more similar. At that point it'll make much, much more sense to have a common auth.

You can see the updates to this PR in my own code base here: https://github.com/davidpcls/bluesky-httpserver/pull/9/changes

Still a few things to clean up but I think it's looking much better. Also resolved that bug where the unit tests just fail randomly because of bad ports.

dmgav · 2026-07-17T17:28:26Z

+    volumes:
+      - 'openldap_data:/var/lib/ldap'
+
+volumes:
+  openldap_data:
+    driver: local


Suggested change

volumes:

- 'openldap_data:/var/lib/ldap'

volumes:

openldap_data:

driver: local

volumes:

- 'openldap_data:/var/lib/ldap'

- 'openldap_config:/etc/ldap/slapd.d'

volumes:

openldap_data:

driver: local

openldap_config:

driver: local

The container failed to start properly without this.

How were you running this? I'm running it locally with the local docker script:

./scripts/run_ci_docker_parallel.sh --python-versions 3.11 --workers 20 --chunks 100

Which, as mentioned, is a mess but useful I think until we can speed up the tests other ways.

It is still good to have the ability to manually start the container and run individual tests by hand. The issue that I observe is that if I try to restart the LDAP container by hand, it fails to start because /var/lib/ldap volume still exists but /etc/ldap/slapd.d volume is missing. I have to delete the volumes explicitly by running

docker compose -f continuous_integration/docker-configs/ldap-docker-compose.yml down -v

Adding openldap_config volume seems to fix the issue. It also works if openldap_data is removed from the configuration, but I am not sure if this is the right solution.

I start the LDAP container by running

bash continuous_integration/scripts/start_LDAP.sh

dmgav · 2026-07-20T15:45:08Z

@davidpcls I am trying to authentication manually using Google provider and authentication fails. Did you test it with Google? Which provider did you use?

Traceback (most recent call last):
  File "/home/dgavrilov/Projects/bluesky-httpserver/bluesky_httpserver/authenticators.py", line 234, in authenticate
    verified_body = self.decode_token(id_token)
  File "/home/dgavrilov/Projects/bluesky-httpserver/bluesky_httpserver/authenticators.py", line 199, in decode_token
    return jwt.decode(
           ~~~~~~~~~~^
        token,
        ^^^^^^
    ...<3 lines>...
        issuer=self.issuer,
        ^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/dgavrilov/tmp/test-httpserver-pixi/.pixi/envs/default/lib/python3.13/site-packages/jose/jwt.py", line 174, in decode
    _validate_claims(
    ~~~~~~~~~~~~~~~~^
        claims,
        ^^^^^^^
    ...<5 lines>...
        options=defaults,
        ^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/dgavrilov/tmp/test-httpserver-pixi/.pixi/envs/default/lib/python3.13/site-packages/jose/jwt.py", line 512, in _validate_claims
    _validate_at_hash(claims, access_token, algorithm)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dgavrilov/tmp/test-httpserver-pixi/.pixi/envs/default/lib/python3.13/site-packages/jose/jwt.py", line 463, in _validate_at_hash
    raise JWTClaimsError(msg)
jose.exceptions.JWTClaimsError: No access_token provided to compare against at_hash claim.
INFO:     127.0.0.1:46298 - "GET /api/auth/provider/google/code?iss=https%3A%2F%2Faccounts.google.com&code=...........&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=2&prompt=none HTTP/1.1" 401 Unauthorized

davidpcls · 2026-07-20T16:10:43Z

@dmgav are you looking at the changes from https://github.com/davidpcls/bluesky-httpserver/pull/9/changes? That's where the current work is and I'm hoping to merge that in soon to this branch, then clean this branch up as I realize it wasn't as ready for review as I thought it was.

I haven't tested this with Google auth yet, I've been focusing solely on entra.

davidpcls · 2026-07-20T16:23:28Z

@danielballan This specifically is the change I'm proposing to both Tiled and bluesky-httpserver's entra authenticator for entra support:

https://github.com/davidpcls/bluesky-httpserver/pull/9/changes#diff-1c0840fe6b2402f90b8eb187783dbdb058d69dc5798d2875f102d582caf2b2e2R314

Adding in an optional parameter to check against graphAPI when requested. By default it would do just what you originally wrote and should be backwards compatible.

From what I've read and my testing here at the CLS, many claims are optional, configurable, and can vary. For us none could be used as an immutable human readable username. This is partially because of our infrastructure choices and partially because MS Entra itself (and OIDC auth in general) has no concept of an enforced user profile in the auth flow, it only authenticates that an identity is known to a identity provider and provides a unique id for that user: the OID. With entra you then make a MS GraphAPI request to get profile information you require, like the system-wide username you use.

* Updating authenticators * Migrating changes from 0.2.12 tiled in * Adding new port tests * Removing lazy import * Removing my dumb changes * Cleaning up the shutdown changes * Adding auto-upgrade ability * Fixing typo * Code review adjustments * Removing timezone changes to reduce scope * Removing unnecessary formatting changes This just makes the diff easier to understand * Some more cleanup * Fixing linting issue reported by isort * Cleaning up pre-commit check * Fixed up unit tests * Fixing linting issue * Making tests more robust * Pre-commit checks * Fixing up test * Refactoring unit tests * Fixed up unit tests after refactoring Moved things around to look more like tiled * Fixing pre-commit errors * Cleanup of the get_current_principal func It was just too large, so moving it towards tiled style as much as possible. Still not perfect but at least better. Tests need updating. * Updating unit test * Working version with MS graph API * Cleanup from pre-commit * Fixes and cleanup to the authenticators * Pre-commit fixes * Cleaning up issues with function renaming in tests * Cleanup test errors

davidpcls

Comments to assist others reviewing the code

davidpcls · 2026-07-21T19:25:27Z

        # Reject any of those old sessions and force reauthentication.
        return None

    session = db.query(Session).filter(Session.uuid == uuid_module.UUID(hex=session_id)).first()


This does not request eager loading like Tiled. The data structure of Tiled does eager loading of the pending session automatically and since we're still on the sync version of the database this is fine.

davidpcls · 2026-07-21T19:26:16Z

+    On successful lookup the matching ``Identity.latest_login`` is updated to
+    now.
+    """
+    identity = db.query(Identity).filter(Identity.id == id).filter(Identity.provider == identity_provider).first()


This does not request eager loading like Tiled. The data structure of Tiled does eager loading of the pending session automatically and since we're still on the sync version of the database this is fine.

davidpcls · 2026-07-21T19:27:26Z

+    """
+    hashed_device_code = hashlib.sha256(device_code).digest()
+    pending_session = (
+        db.query(PendingSession).filter(PendingSession.hashed_device_code == hashed_device_code).first()


This does not request eager loading like Tiled. The data structure of Tiled does eager loading of the pending session automatically and since we're still on the sync version of the database this is fine.

davidpcls · 2026-07-21T20:04:45Z

@@ -0,0 +1,37 @@
+from abc import ABC


File copied from Tiled

davidpcls · 2026-07-21T20:08:36Z

@@ -0,0 +1,80 @@
+#!/usr/bin/env bash


I'll fully admit, i did not write this myself. AI did it. It works reliably for me and brings test times down for local development to around 10 minutes.

davidpcls · 2026-07-21T20:08:47Z

I'll fully admit, i did not write this myself. AI did it. It works reliably for me and brings test times down for local development to around 10 minutes.

This looks like a good idea to run local tests in multiple independent processes. It should also be possible to use pytest-split instead of shards to balance load between processes (as in https://github.com/bluesky/bluesky-queueserver/blob/0a084cfb722308f384847c24e6a43a8f572f302e/.github/workflows/testing.yml#L50). This way the tests are split on equal groups based on execution time. It seems to work well in the bluesky-queueserver repository for running CI tests. But this is for future discussion.

davidpcls · 2026-07-21T21:53:01Z

After these changes, still seems to be working for me. Neato.

danielballan requested a review from dmgav February 3, 2026 15:34

danielballan requested a review from sligara7 February 3, 2026 15:35

davidpcls marked this pull request as draft February 3, 2026 17:41

davidpcls and others added 2 commits February 4, 2026 10:33

Updating authenticators from latest in Tiled

c474afd

TST: fix unit tests

f54ef36

dmgav force-pushed the auth_updates_from_tiled branch from 579d82a to f54ef36 Compare February 5, 2026 16:21

STY: reformat with black

be73eda

davidpcls added 10 commits February 17, 2026 15:04

Removing some unnecessary code.

d90ad0c

Fixes from running black

96cd9db

Adding documentation on how to use OIDC

967fcba

Fixes for unit tests, moving start LDAP

5906b28

These should correct some of the problems in the last CI workflow. I moved the LDAP and docker image into the continuous_integration folder so it matches tiled.

fixing pre-commit issues

28483f9

fixing documentation issues

a4551e3

This addresses documentation problems, the levels were incorrect as I did not understand what the next level should have been in the docs. I've also updated the usage documentation a little to be more useful.

Adding in helper scripts for testing

8fa89ab

These allow for running the unit tests in a containerized system just like how they are done in the ci pipeline, but locally and in a way that can maximize processor usage and minimize runtime.

Merge pull request #2 from davidpcls/testing_with_ms_entra

318ab48

Testing with ms entra

davidpcls marked this pull request as ready for review March 20, 2026 19:54

davidpcls changed the title ~~WIP - Updating authenticators from latest in Tiled~~ Updating authenticators from latest in Tiled Mar 20, 2026

davidpcls marked this pull request as draft March 20, 2026 19:58

davidpcls marked this pull request as ready for review March 20, 2026 20:39

Merge branch 'main' into auth_updates_from_tiled

2a12569

davidpcls mentioned this pull request Jul 3, 2026

Update bluesky-httpserver from changes in Tiled #86

Open

davidpcls marked this pull request as draft July 3, 2026 17:20

dmgav reviewed Jul 17, 2026

View reviewed changes

davidpcls added 3 commits July 20, 2026 15:38

Merging database versions together

3ad4fb0

Making orm match more closely that of tiled

3795bee

davidpcls commented Jul 21, 2026

View reviewed changes

davidpcls added 3 commits July 21, 2026 15:27

Adding documentation for entra to docs

46ff3a7

Fixing pre-commit issues

aecc357

Fixing title

073ba2b

Adding back default state type as per unit test

90e0479

davidpcls marked this pull request as ready for review July 22, 2026 15:47

davidpcls requested a review from dmgav July 22, 2026 15:48

davidpcls mentioned this pull request Jul 22, 2026

DISCUSSION - Scope of work Canadian-Light-Source/bluesky-authentication#5

Open

Uh oh!

Conversation

davidpcls commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Summary of Changes for Release Notes

Fixed

Added

Changed

Removed

How Has This Been Tested?

TODO:

Uh oh!

danielballan commented Feb 3, 2026

Uh oh!

dmgav commented Feb 3, 2026

Uh oh!

prjemian commented Feb 3, 2026

Uh oh!

davidpcls commented Feb 3, 2026

Uh oh!

dmgav commented Feb 5, 2026

Uh oh!

davidpcls commented Feb 17, 2026

Uh oh!

davidpcls commented Mar 13, 2026

Uh oh!

davidpcls commented Mar 13, 2026

Uh oh!

davidpcls commented Mar 20, 2026

Uh oh!

davidpcls commented Mar 20, 2026

Uh oh!

danielballan commented May 22, 2026

Uh oh!

davidpcls commented May 25, 2026

Uh oh!

checkmarx-gh-ast-us-povs Bot commented May 25, 2026

Uh oh!

davidpcls commented May 28, 2026

Uh oh!

davidpcls commented Jun 23, 2026

Uh oh!

davidpcls commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmgav Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmgav commented Jul 20, 2026

Uh oh!

davidpcls commented Jul 20, 2026

Uh oh!

davidpcls commented Jul 20, 2026

Uh oh!

davidpcls left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidpcls commented Jul 21, 2026

Uh oh!

davidpcls commented Jan 29, 2026 •

edited

Loading

davidpcls commented Jul 14, 2026 •

edited

Loading

dmgav Jul 23, 2026 •

edited

Loading