Add the Resyntax Grimoire

base.rkt

@@ -50,7 +50,7 @@
          resyntax/default-recommendations/private/definition-context
          resyntax/private/analyzer
          resyntax/private/logger
-         resyntax/private/source
+         resyntax/grimoire/source
          resyntax/private/syntax-neighbors
          resyntax/private/syntax-replacement
          syntax/parse
@@ -116,7 +116,7 @@
 (define (refactoring-rule-refactor rule syntax source)
 
   ;; Before refactoring the input syntax, we create a new scope and add it. Combined with the code in
-  ;; resyntax/private/source which marks the original path of every syntax object before expansion,
+  ;; resyntax/grimoire/source which marks the original path of every syntax object before expansion,
   ;; this allows us to tell when two neighboring subforms within the output syntax object are
   ;; originally from the input and were originally next to each other in the input. This allows
   ;; Resyntax to preserve any formatting and comments between those two subform when rendering the

cli.rkt

@@ -27,7 +27,7 @@
          resyntax/private/file-group
          resyntax/private/github
          resyntax/private/refactoring-result
-         resyntax/private/source
+         resyntax/grimoire/source
          resyntax/private/string-indent
          resyntax/private/syntax-replacement)
 
@@ -75,7 +75,7 @@
 
    ("--directory"
     dirpath
-    "A directory to anaylze, including subdirectories."
+    "A directory to analyze, including subdirectories."
     (vector-builder-add targets (directory-file-group dirpath)))
 
    ("--package"

cli.scrbl

@@ -0,0 +1,94 @@
+#lang scribble/manual
+
+
+@(require (for-label racket/base
+                     resyntax/base)
+          scribble/bnf)
+
+
+@title[#:tag "cli"]{The Resyntax Command-Line Interface}
+
+
+Resyntax provides a command-line @exec{resyntax} tool for analyzing and refactoring code. The tool has
+two commands: @exec{resyntax analyze} for analyzing code without changing it, and @exec{resyntax fix}
+for fixing code by applying Resyntax's suggestions.
+
+Note that at present, Resyntax is limited in what files it can fix. Resyntax only analyzes files with
+the @exec{.rkt} extension where @tt{#lang racket/base} is the first line in the file.
+
+
+@section[#:tag "install"]{Installation}
+
+Use the Racket package manager to install Resyntax in the installation scope:
+
+@verbatim{
+ % raco pkg install --installation resyntax
+}
+
+The @exec{--installation} flag (shorthand for @exec{--scope installation}) installs packages for
+all users of a Racket installation, ensuring @exec{resyntax} is in your @envvar{PATH}.
+
+e.g.
+@verbatim{
+ % resyntax analyze --file example.rkt
+ resyntax: --- analyzing code ---
+ resyntax: --- displaying results ---
+ %
+}
+
+
+@section{Running @exec{resyntax analyze}}
+
+
+The @exec{resyntax analyze} command accepts flags for specifying what modules to analyze. After
+analysis, suggestions are printed in the console. Any of the following flags can be specified any
+number of times:
+
+
+@itemlist[
+
+ @item{@exec{--file} @nonterm{file-path} --- A file to analyze.}
+
+ @item{@exec{--directory} @nonterm{directory-path} --- A directory to analyze, including
+  subdirectories.}
+
+ @item{@exec{--package} @nonterm{package-name} --- An installed package to analyze.}
+
+ @item{@exec{--local-git-repository} @nonterm{repository-path} @nonterm{base-ref} --- A local Git
+  repository to analyze the changed files of. Only files which have changed relative to
+  @nonterm{base-ref} are analyzed. Base references must be given in the form
+  @exec{remotename/branchname}, for example @exec{origin/main} or @exec{upstream/my-feature-branch}.}
+
+ @item{@exec{--refactoring-suite} @nonterm{module-path} @nonterm{suite-name} --- A
+  @tech{refactoring suite} to use instead of Resyntax's default recommendations. Custom refactoring
+  suites can be created with @racket[define-refactoring-suite].}]
+
+
+@section{Running @exec{resyntax fix}}
+
+
+The @exec{resyntax fix} command accepts the same flags as @exec{resyntax analyze} for specifying what
+modules to fix. After analysis, fixes are applied and a summary is printed.
+
+
+@itemlist[
+
+ @item{@exec{--file} @nonterm{file-path} --- A file to fix.}
+
+ @item{@exec{--directory} @nonterm{directory-path} --- A directory to fix, including
+  subdirectories.}
+
+ @item{@exec{--package} @nonterm{package-name} --- An installed package to fix.}
+
+ @item{@exec{--local-git-repository} @nonterm{repository-path} @nonterm{base-ref} --- A local Git
+  repository to fix the changed files of. Only files which have changed relative to @nonterm{base-ref}
+  are fixed. Base references must be given in the form @exec{remotename/branchname}, for example
+  @exec{origin/main} or @exec{upstream/my-feature-branch}.}
+
+ @item{@exec{--refactoring-suite} @nonterm{module-path} @nonterm{suite-name} --- A
+  @tech{refactoring suite} to use instead of Resyntax's default recommendations. Custom refactoring
+  suites can be created with @racket[define-refactoring-suite].}]
+
+
+If two suggestions try to fix the same code, one of them will be rejected. At present, the best way to
+handle overlapping fixes is to run Resyntax multiple times until no fixes are rejected.

grimoire.scrbl

@@ -0,0 +1,202 @@
+#lang scribble/manual
+
+
+@(require (for-label racket/base
+                     racket/contract/base
+                     racket/path
+                     rebellion/base/immutable-string
+                     rebellion/collection/range-set
+                     resyntax/grimoire/source
+                     syntax/modread))
+
+
+@title[#:tag "grimoire"]{The Resyntax Grimoire}
+
+Resyntax's implementation is complex. This document serves as a reference manual for many of the
+internal libraries and abstractions contained within Resyntax. @bold{The APIs documented here are
+ unstable and not meant for public consumption at this time.} This grimoire is intended for those
+seeking to understand how Resyntax operates under the hood. Danger awaits those who come to rely
+programmatically on anything found here.
+
+
+@section{Source Code}
+@defmodule[resyntax/grimoire/source]
+
+In Resyntax, @deftech{source code} refers to @racket[source?] values, which come in three types:
+
+@itemlist[
+ @item{@emph{Source files}, constructed with @racket[file-source], which don't contain the code
+  directly but refer to it by a local filesystem path.}
+
+ @item{@emph{Source strings}, constructed with @racket[string-source], which contain the source
+  code directly as a string and don't exist anywhere on the local filesystem. (These are useful for
+ testing Resyntax, and other scenarios where Resyntax needs to operate on code that doesn't exist on
+ disk.)}
+
+ @item{@emph{Modified sources}, constructed by passing another (unmodified) source to
+  @racket[modified-source] along with a string representing what to replace the source's contents
+  with. A modified source contains both its new updated contents and a reference to the original
+  source.}]
+
+Resyntax's basic architecture is to recursively take sources of any kind as input, produce
+@racket[modified-source?] values as output, then re-analyze the modified sources again until no
+further modifications are desired. Then, Resyntax decides whether to commit those final modifications
+to the filesystem (as in @tt{resyntax fix}) or merely display them to users (as in
+@tt{resyntax analyze}). This recursive loop approach allows Resyntax to "look ahead" and produce a
+stack of dependent changes to commit in series without actually mutating the files on disk.
+
+
+@subsection{Basic Source Operations}
+
+
+@defproc[(source? [v any/c]) boolean?]{
+ A predicate that recognizes @tech{source code} values of any kind --- file, string, or modified.}
+
+
+@defproc[(unmodified-source? [v any/c]) boolean?]{
+ A predicate that recognizes @tech{source code} values that are @emph{not} modified sources, i.e.
+ either file sources or string sources.}
+
+
+@defproc[(file-source? [v any/c]) boolean?]{
+ A predicate that recognizes (unmodified) source files.}
+
+
+@defproc[(file-source [path path-string?]) file-source?]{
+ Constructs a source file that refers to the code stored on disk at @racket[path]. The path is
+ normalized with @racket[simple-form-path] upon construction.}
+
+
+@defproc[(file-source-path [code file-source?]) path?]{
+ Returns the filesystem path that @racket[code] refers to.}
+
+
+@defproc[(string-source? [v any/c]) boolean?]{
+ A predicate that recognizes source strings.}
+
+
+@defproc[(string-source [contents string?]) string-source?]{
+ Constructs a source string containing @racket[contents] directly.}
+
+
+@defproc[(string-source-contents [code string-source?]) immutable-string?]{
+ Returns the source code text contained within @racket[code]. Note that this is a distinct operation
+ from @racket[source->string] --- the latter returns the current source code text of @emph{any} source
+ code, which may involve reading files from disk. This operation, in contrast, cannot perform I/O
+ because it only operates on an unmodified @racket[string-source?].}
+
+
+@defproc[(modified-source? [v any/c]) boolean?]{
+ A predicate that recognizes modified sources. A modified source is always a wrapper around an
+ unmodified source plus a string containing the full replacement text that the modified source should
+ contain instead of what the original source contains.}
+
+
+@defproc[(modified-source [original unmodified-source?] [new-contents string?]) modified-source?]{
+ Constructs a modified source that replaces the contents of @racket[original] with
+ @racket[new-contents]. This represents a whole-file replacement --- the @emph{complete} contents of
+ @racket[original] are @emph{entirely} swapped out with @racket[new-contents]. Modified sources cannot
+ represent partial edits on their own.}
+
+
+@defproc[(modified-source-contents [code modified-source?]) immutable-string?]{
+ Returns the new, updated contents of @racket[code], ignoring whatever contents the original source
+ of @racket[code] had.}
+
+
+@defproc[(modified-source-original [code modified-source?]) unmodified-source?]{
+ Returns the original, unmodified source that @racket[code] was constructed from.}
+
+
+@defproc[(source-name [code source?]) (or/c path? symbol?)]{
+ Returns a name identifying @racket[code], suitable for use as a syntax object's source location
+ name. For file-based sources this is the source file's path, and for string-based sources this is the
+ symbol @racket['string]. Modified sources always have the same name as their original unmodified
+ sources.}
+
+
+@defproc[(source-path [code source?]) (or/c path? #false)]{
+ Returns the filesystem path of @racket[code], or @racket[#false] if @racket[code] is not
+ file-based. Modified sources are file-based if their original source is file-based.}
+
+
+@defproc[(source-directory [code source?]) (or/c path? #false)]{
+ Returns the directory containing @racket[code], or @racket[#false] if @racket[code] is not
+ file-based. Modified sources are file-based if their original source is file-based.}
+
+
+@defproc[(source-original [code source?]) unmodified-source?]{
+ Returns the original, unmodified source underlying @racket[code]. If @racket[code] is already
+ unmodified, it is returned as-is.}
+
+
+@defproc[(source->string [code source?]) immutable-string?]{
+ Returns the full text of @racket[code], reading it from the filesystem if necessary. For
+ @racket[modified-source?] values, this returns the new, updated text rather than the original
+ unmodified text.}
+
+
+@defproc[(with-input-from-source [code source?] [proc (-> any)]) any]{
+ Calls @racket[proc] with @racket[current-input-port] set to a freshly opened input port reading
+ the contents of @racket[code]. For unmodified file sources, this opens a file port. For modified
+ sources and string sources, this opens a string port without interacting with the filesystem.}
+
+
+@subsection{Parsing, Expanding, and Compiling Sources}
+
+
+The following operations allow treating @tech{source code} values as inputs to Racket's compiler.
+For file sources, this is roughly the same as reading the file into a syntax object using
+@racket[with-module-reading-parameterization] and expanding that syntax object using @racket[expand].
+However, string sources and modified sources behave slightly differently, especially with regard to
+source location information on derived syntax objects:
+
+@itemlist[
+ @item{A string source behaves as if it were an @emph{anonymous file} with no well-defined location on
+  the filesystem. Relative file path imports will not work correctly. Source location information will
+  still be present with line and column numbers, but will claim to be located in a source named
+  @racket['string] instead of a file.}
+
+ @item{A modified source behaves the same as its wrapped unmodified source, except as if its contents
+  were completely replaced. Source location information will be present, @bold{but will not correspond
+   to positions within the original source} as they will instead refer to positions in the modified
+  contents. If the original source was a file source, accidentally using source locations from the
+  modified source to make edits to the original file will produce malformed changes.}]
+
+Note that while a file source is being read and expanded, the current directory is parameterized to
+the file source's parent directory. This ensures that relative file path imports in source files can
+still be resolved regardless of what the current directory is before the source is read or expanded.
+This applies to both modified and unmodified file sources.
+
+
+@defproc[(source-read-language [code source?]) (or/c module-path? #false)]{
+ Detects the @hash-lang[] language of @racket[code] and returns the module path of that language.
+ Returns @racket[#false] if @racket[code] does not begin with a @hash-lang[] line.}
+
+
+@defproc[(source-read-syntax [code source?]) syntax?]{
+ Reads @racket[code] as a syntax object, using the module reading parameterization to allow the
+ source's @hash-lang[] to control the reader.}
+
+
+@defproc[(source-expand [code source?]) syntax?]{
+ Reads @racket[code] and fully expands it, as in @racket[expand].}
+
+
+@defproc[(source-can-expand? [code source?]) boolean?]{
+ Attempts to fully expand @racket[code], then returns @racket[#true] if expansion finished
+ without raising an error and returns @racket[#false] otherwise.}
+
+
+@defproc[(source-text-of [code source?] [stx syntax?]) immutable-string?]{
+ Returns the source text within @racket[code] that produced @racket[stx], based on the source location
+ information attached to @racket[stx]. Raises a contract violation if @racket[stx] does not have
+ source location information.}
+
+
+@defproc[(source-comment-locations [code source?]) immutable-range-set?]{
+ Returns a range set containing the positions of all comments in @racket[code]. This is implemented
+ by looking up the lexer of the @hash-lang[] that @racket[code] is written in, using the
+ @racketmodname[syntax-color/module-lexer] API. @bold{Warning: the positions are zero-based}, unlike
+ the one-based positions returned from @racket[syntax-position]. Additionally, positions are in terms
+ of @emph{characters} and not @emph{bytes}.}