Xpress direct API performance

[CharlieFModo]

Hi guys - I recently benchmarks the xpress direct API in linopy 0.8.0 against the implementation I floated in #596. That PR was closed because it prompted a separate refactor of the matrix accessors. In my tests the API is around 8x slower on a problem with the stats below. I'd guess this is coming from the matrix accessors rather than the direct API.

For the direct APIs, in general I would expect them to be faster than the polars IO provided the solar has a performant, low level API for writing problems. It should be faster because there isn't the added write-read overhead of passing an mps file into the solver. Xpress definitely has a performant API!

I'd be happy to have a go at improving the performance again if you have some context on why I'm seeing such a discrepancy.

Reading Problem linopy
Problem Statistics
     5697205 (      0 spare) rows
     4107168 (      0 spare) structural columns
    17704428 (      0 spare) non-zero elements
MIP Entity Statistics
     1472256 entities        0 sets        0 set members

[coroa]

The path taken for the matrix accessor rewrite in #630 is slightly different from #596 , by building per constraint block CSR matrices and then stacking them together into a full model CSR matrix which is passed to the solver using the direct io.

In our tests with the Highs and Gurobi solver this is faster than the polars io, indeed. We did not test xpress, i think.

Two possible culprits for why this implementation might be slower than what you did:

1. If you have a large number of very small constraint objects, i guess the stacking might become more dominant and then putting everything directly into one matrix would win. (the benefit of converting per constraint object, is that you can persist the matrix as a CSRConstraint, which is smaller than the xarray)
2. Your PR is building a CSC matrix directly (which is not vertically, but only horizontally stackable, so one could not build it per constraint object), this CSC matrix is then passed into xpress. In the current master the all constraint object CSR matrix is converted to CSC using scipy's internals before being passed off, maybe that overhead is not as small as thought?

[FBumann]

@CharlieFModo If you could provide the size of your linopy.Model, this would help a lot.
I found some things much slover than expected when I started using linopy. The reason was me using many small 1D variables/constraints instead of properly using dimensions.