Detached concurrent jobs

[monnerat]

Hi Yuki,

Here is the code I plan to implement concurrent tasks.
It is highly based on promises and io loop scheduling and mainly consists of:

• A fully asynchronous interface to subprocesses,
• A curl interface,
• A detached job queue with parallel execution.

I tested it here successfully, but this is not ready to be merged, because of a bug in Net::Curl awaiting a pending fix to be applied (sparky/perl-Net-Curl#87). I've investigated the way to local patching in setup_module, but unfortunately cpanm is not able to do it. Any comment about your ideas on this is welcomed.

There is another fix pending allowing Net::Curl::Promiser to use a custom IO loop (FGasper/p5-Net-Curl-Promiser#4). This is optional yet, but will be needed to use curl synchronously.

I make this PR a draft for the above reasons, but you can test it providing you fix Net::Curl manually.

Regards,
Patrick

[yuki-kimoto]

I'd like to take some time to think this over, as I want to make sure I fully grasp the technical complexities involved.

[monnerat]

I can document how it works if you need it.

[yuki-kimoto]

I'd like to go through this step-by-step to make sure I fully understand each part.

[yuki-kimoto]

The reason for using IPC::Open3 is to monitor file descriptors that involve both reading and writing, isn't it?

[monnerat]

Just a few hints:

• The real "magic" is the io loop and its reactor: the main io loop is run by Mojolicious and its reactor calls callbacks and it continues to run when an HTTP request is finished.
• You can use an alternate io loop (that you have to run yourself or via run_blocking_jobs) if you want to reuse the Jobs support synchronously.
• Curl and command interfaces handle the stdio parameters in a polymorphic way, so you may redirect any io to files, scalars or callbacks.
• When a job is started, we do not wait for it to be complete, but count on its associated IO loop to perform the needed callbacks, finally resolving or rejecting the job as a promise (which may run promise-associated callbacks indeed). Therefore the real wait is performed by the io loop.
• If you can avoid it, you better not dive into Mojo::Promise code: it is VERY hard to understand !

[monnerat]

The reason for using IPC::Open3 is to monitor file descriptors that involve both reading and writing, isn't it?

Yes it is.
File descriptors are mapped by IPC::Open3 to file handles, which are turned to IO streams by the new code. IO streams may be integrated in an IO loop reactor and are capable of generating events. All the asynchronous IO is performed this way.

Because the command subroutine is a general interface, you can use it for any IO combination case. However I did not change code snippets that only read.

[yuki-kimoto]

You mean it depends on libcurl, and I need to run an install command such as 'apt install libcurl4-openssl-dev' to install it?

[monnerat]

You mean it depends on libcurl, and I need to run an install command such as 'apt install libcurl4-openssl-dev' to install it?

Yes, this is needed to compile Net::Curl.
I don't know how debian-based distributions handle development packages, but you should already have some similar requirements, like for libxcrypt or libpng.

[yuki-kimoto]

Once the bugs in the dependent libraries are fixed, I am okay with merging the changes.

SMTP/SMTPS : I understand that we can use libcurl's multi interface through Mojo::IOLoop and Net::Curl::Promisser for non-blocking operation.

sendmail command: we use subprocess and IPC::Open3 for non-blocking operation .

[monnerat]

Once the bugs in the dependent libraries are fixed, I am okay with merging the changes.

Thanks for your trust. I'm glad you like this new code.

SMTP/SMTPS : I understand that we can use libcurl's multi interface through Mojo::IOLoop and Net::Curl::Promisser for non-blocking operation.

Yes, that's it.

sendmail command: we use subprocess and IPC::Open3 for non-blocking operation .

Right too, if you mean system subprocess and not Mojo::IOLoop::Subprocess which does not fit our requirements.

It is possible however to use these new interfaces synchronously too and for something else than mail: see templates/smart-http/info-refs.html.ep for a subprocess example. I also already have here a modified templates/user-settings.html.ep that uses curl instead of Mojo::UserAgent for gravatar access via https: it works perfectly with patched packages but I did not included it in PR since it requires curl synchronous mode availability.

You may have noticed that templates/smart-http/service.html.ep has also been changed to use the new interface: this is a typical case of fully asynchronous use of the Jobs::command subroutine and illustrates very well the versatility of the latter. In addition, it does not block possible concurrent detached jobs because it does not wait for IO by itself anymore.

[monnerat]

BTW: Did you ever consider a way to patch external modules in your setup procedure? What are your thinking about it?

[yuki-kimoto]

BTW: Did you ever consider a way to patch external modules in your setup procedure? What are your thinking about it?

If you mean patching the .xs or .c files included in distributions, I honestly can't think of an easy way to do that.

As for .pm files, it is possible to apply monkey patches from GitPrep code at runtime for temporary fixes.

If you have a specific approach in mind, I'm certainly open to including it in the setup procedure.

[monnerat]

If you mean patching the .xs or .c files included in distributions, I honestly can't think of an easy way to do that.

Yes, this is what I mean.
Unfortunately, cpanm does not allow a patch phase before compilation.
I investigated the way to split cpanm builds, but for this I'll need to know the dependency tree. Unfortunately again, cpanm people removed the options to get it :-(
After all brainstorming I did about it. I don't see any alternate reasonable solution than moving away from cpanm, maybe for cpan that seems to feature a patch phase. The drawback is an heavier configuration.
And I don't feel comfortable about adding a patch feature to cpanm...

[yuki-kimoto]

The only feasible workaround I can think of is to let cpanm run the installation first(using -n), and then manually re-install only the modules that require patching.

However, if even using -n causes the compilation to fail, this approach wouldn't be applicable.

[monnerat]

Using your suggestion for our current case, the Net::Curl bug blocks Net::Curl::Promiser tests,, so the initial install must be performed without tests.
The uestion then is: should we care about "post-testing"?

[monnerat]

Another solution would be to download all packages and get the topological order (using deprecated cpanm options I'm afraid), untar packages and apply patches, then building from local dirs in the correct order.
This would require a perl script: bash is not powerful enough forthis.

[yuki-kimoto]

Could you provide more details?

I’m wondering, can we really determine all the necessary distributions and the correct installation order by using deprecated cpanm options? What are the deprecated cpanm options you're referring to?

[yuki-kimoto]

I have an idea: what if we manually pre-install only the distributions that require patching?

Specifically, I could extract the tar.gz, move into the directory, run cpanm --installdeps . to fetch the dependencies, apply the patch, and then proceed with perl Makefile.PL && make && make test (or perl Build.PL ...), and make install

[monnerat]

I tried:
perl -Iextlib/lib/perl5 cpanm -L extlib --scandeps --save-dists ./ducon --format dists Module::CoreList --installdeps . LWP::Protocol::https ExtUtils::MakeMaker
This downloads tarballs and the output finishes with the list:

BINGOS/Module-CoreList-5.20260601.tar.gz
RSAVAGE/Config-Tiny-2.30.tgz
HMBRAND/DBI-1.651.tgz
ISHIGAKI/DBD-SQLite-1.78.tar.gz
KIMOTO/Object-Simple-3.19.tar.gz
SYP/Net-Curl-0.58.tar.gz
SRI/Mojolicious-9.48.tar.gz
KIMOTO/Mojolicious-Plugin-AutoRoute-0.23.tar.gz
LEONT/Module-Build-0.4234.tar.gz
ETHER/Test-Warnings-0.038.tar.gz
HAARG/Test-Needs-0.002010.tar.gz
REHSACK/MIME-Base32-1.303.tar.gz
ETHER/Try-Tiny-0.32.tar.gz
RJBS/Test-Fatal-0.018.tar.gz
OALDERS/URI-5.35.tar.gz
GAAS/Font-AFM-1.20.tar.gz
...

Options --scandeps and --format are deprecated.

[monnerat]

I have an idea: what if we manually pre-install only the distributions that require patching?

They may require other packages. And you have to give them in the correct order.

In addition, I would prefer a more automated solution.

[yuki-kimoto]

They may require other packages. And you have to give them in the correct order.

`cpanm --installdeps .' resolves that?

# Install all dependent modules for Foo
cd Foo
cpanm --installdeps .

[monnerat]

`cpanm --installdeps .' resolves that?

Yes, but not between patched packages, if you have more than one. Thus it would be a dirty trick rather than an orthogonal process.

That said, the commit for Net::Curl::Promiser has now been released (0.21) so we have a single needed patch pending left.

[yuki-kimoto]

Yes, but not between patched packages, if you have more than one. Thus it would be a dirty trick rather than an orthogonal process.

That’s true, but aren't patches really meant to be temporary, emergency fixes? Plus, if you need to apply patches to two different distributions, couldn't you just determine the order of dependencies and handle them accordingly?

Don't you feel like writing your own installation script to handle CPAN dependency resolution is just too heavy of a task?

[monnerat]

aren't patches really meant to be temporary, emergency fixes?

They should But I'm afraid some PRs stay a long time before the maintainer considers and merges them. And I feel the one that blocks our changes falls in this category.

couldn't you just determine the order of dependencies and handle them accordingly?

You mean manually ? This is highly error-prone, as an unsuspected third package may require the patched one without you even notice it.

Don't you feel like writing your own installation script to handle CPAN dependency resolution is just too heavy of a task?

I dont want to rewrite cpanm! I just want to use it from a script that will probably too sophisticated to be implemented as a bash script

[yuki-kimoto]

Rewriting the setup script in Perl is OK.

[yuki-kimoto]

Feel free to create a separate pull request for the dependency resolution and installation logic you have in mind.