Use Travis CI to download Rubygems data for listed gems

[ashmaroli]

The following Rake task takes the directory listing (_data/themes.yml and _data/plugins.yml clubbed into a single _data/entries.yml) and downloads each gem into their respective directory

# frozen_string_literal: true

require "rubygems"
require "rake"
require "open-uri"
require "safe_yaml"

task :default do
  entries = SafeYAML.load_file("_data/entries.yml")
  entries.each do |type, list|
    FileUtils.mkdir_p("_data/#{type}")
    list.each do |item|
      begin
        data = open("https://rubygems.org/api/v1/gems/#{item}.yaml").read
        File.open("_data/#{type}/#{item}.yml", "wb") { |file| file.puts data }
      rescue OpenURI::HTTPError
        puts "#{item} does not seem to exist at rubygems.org"
        next
      end
    end
  end
end

The data files can then be used to define the front end..
We can then set up Travis to download these files at a set interval..
Thoughts?

[mattr-]

How would _data/entries.yml get created?

Also, in order for this to work, Travis would have to commit these back to GitHub, would it not? I'm not sure that's something they would support doing. I'm also not sure if I've understood your idea correctly. Let me know if I've got it wrong.

[ashmaroli]

How would _data/entries.yml get created?

If this idea gets approved, I'll open a PR that clubs existing _data/themes.yml and _data/plugins.yml files into _data/entries.yml and add the rake task to this repo as well. In future, any user desiring to get their plugin listed need to open a PR adding their gem / plugin-name / theme-name to _data/entries.yml

Travis would have to commit these back to GitHub, would it not?

Yes, Travis CI can be used to do this.. Its documented. My plan is therefore:

• set up Travis to commit back to this repo. (I've already run trials on my fork). Perhaps @parkr can lend a hand at adding a secure access token for this repo..)
• instruct Travis to:
  • download gem data from Rubygems.org API if available
  • use this data to build the production-ready site
  • push the generated site back to gh-pages branch on this repo (default)
  • set up cron jobs to have this automated at set intervals
  • update Readme document here to ease contribution and development

[mattr-]

I'm onboard with this. 👍

Out of curiousity, what drove the idea behind creating a single file vs. keeping the two files and just smashing them together as part of the rake task?

[DirtyF]

👍 Interesting, right now we have manually added documents, this would allow us to display more info like latest version, author, downloads, link to changelog, etc. 😄

Downloads could be use as a sort filter.

{
    "authors": "Mark Otto",
    "bug_tracker_uri": null,
    "changelog_uri": null,
        "runtime": [
            {
                "name": "jekyll",
                "requirements": "~> 3.6"
            },
            {
                "name": "jekyll-feed",
                "requirements": "~> 0.9"
            },
            {
                "name": "jekyll-gist",
                "requirements": "~> 1.5"
            },
            {
                "name": "jekyll-paginate",
                "requirements": "~> 1.1"
            },
            {
                "name": "jekyll-sitemap",
                "requirements": "~> 1.1"
            }
        ]
    },
    "documentation_uri": "http://www.rubydoc.info/gems/jekyll-theme-poole/2.1.0",
    "downloads": 857,
    "gem_uri": "https://rubygems.org/gems/jekyll-theme-poole-2.1.0.gem",
    "homepage_uri": "https://github.com/poole/poole",
    "info": "The Jekyll Butler, designed and developed by @mdo to provide a clear and concise foundational setup for any Jekyll site.",
    "licenses": [
        "MIT"
    ],
    "metadata": {
        "plugin_type": "theme"
    },
    "name": "jekyll-theme-poole",
    "project_uri": "https://rubygems.org/gems/jekyll-theme-poole",
    "version": "2.1.0",
 
}

[ashmaroli]

what drove the idea behind creating a single file vs. keeping the two files and just smashing them together as part of the rake task?

Easier to code to load a single YAML file and proceed.., as against reading two files and merging data or clobbering the two files into one and then loading that.... 😄

[ashmaroli]

allow us to display more info

yes.. though we still need the manual data in sync with showcasing themes and plugins that have not been gemmified..

[pathawks]

though we still need the manual data in sync with showcasing themes and plugins that have not been gemmified..

I think this directory should only include plugins and themes that are available as Gems

[jekyllbot]

This issue has been automatically marked as stale because it has not been commented on for at least two months.

The resources of the Jekyll team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, please consider whether it can be accomplished in another way. If it cannot, please elaborate on why it is core to this project and why you feel more than 80% of users would find this beneficial.

This issue will automatically be closed in two months if no further activity occurs. Thank you for all your contributions.

[pathawks]

No! This is not going to die due to inactivity!

What do we need to do to carry this across the finish line?

[jekyllbot]

This issue has been automatically marked as stale because it has not been commented on for at least two months.

The resources of the Jekyll team are limited, and so we are asking for your help.

If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.

If this is a feature request, please consider whether it can be accomplished in another way. If it cannot, please elaborate on why it is core to this project and why you feel more than 80% of users would find this beneficial.

This issue will automatically be closed in two months if no further activity occurs. Thank you for all your contributions.

[pathawks]

What do we need to do to carry this across the finish line?