1. 13 Dec, 2020 1 commit
    • KOBAYASHI Shuji's avatar
      Improve source scanning for presym · 456878ba
      KOBAYASHI Shuji authored
      The accuracy is greatly improved by using the C preprocessor to scan C
      sources for presym. C preprocessor can perfectly interpret all comments and
      preprocessor directives, so it can detect all symbols defined, for example
      `mrbgems/mruby-socket/src/const.cstub`.
      
      Also, as described later, this change will greatly improve the accuracy of
      presym detection from Ruby sources.
      
      ## Result
      
      The number of lines in the `presym` file for all gems is as follows:
      
        ```console
        Previous:   999 (false positive = 89, undetected = 297)
        New:       1207
        ```
      
      ## Build process
      
      The new build process (with presym) is as follows:
      
      1. Build `mrbc` without presym (more on building without presym later).
      2. Compile Ruby sources to C struct format with the `mrbc` created in
         step 1, and create` mrblib.c` and `gem_init.c`. Note that the symbols
         in the created files are output as `MRB_SYM` family macros or
         `mrb_intern_lit` instead of IDs (details will be described later).
      3. C preprocessor processes C sources including the created files of
         step 2 and outputs them as `.i` files. In these files, for example,
         `MRB_IVSYM(foo)` is converted to `<@! "@" "foo" !@>` and
         `mrb_define_module(mrb, "Foo")` is converted to `<@! "Foo" !@>`.
      4. Scan the files created in step 3 and create `presym` and` presym.inc`
         files.
      
      The files created in step 2 should output all static symbols defined in Ruby
      sources, including local variables, so we can detect all presyms by just
      scanning C sources without scanning Ruby sources directly.
      
      Further, by this process, the files to be scanned becomes the same as the
      files to be compiled, so that there is no excess or deficiency.
      
      ## Related changes
      
      The following changes have been made in relation to realizing this feature.
      
      ### Allow build without presym
      
      It enables build without presym to achieve the "Build process: 1". This
      incorporates #5202, see its issue for details.
      
      Note that when presym is enabled, even adding a local variable to a Ruby
      source may change contents of presym and require recompilation of almost
      all C sources. This is inconvenient, especially during trial and error in
      development, but this feature is also useful because it does not cause
      this problem if presym is disabled.
      
      ### Automatically create build target for `mrbc` without presym
      
      The `mrbc` used in the "Build process: 1" will be built by automatically
      creating a build target for it. The build name is `SOURCE_BUILD_NAME/mrbc`.
      
      ### Constantize output of C struct format by `mrbc`
      
      To realizing the "Build process: 2", as mentioned above, symbol IDs are not
      output directly in C struct format output by `mrbc`. As a result, the output
      becomes constant regardless of the state of presym at the time of `mrbc`
      build, and it is possible to detect symbols of Ruby sources in the same way
      as other C sources.
      
      Note that `mrb_intern_lit` is used for symbols that do not become presym,
      but in this state, the corresponding element in the symbol array cannot be
      statically initialized, so it is initialized at run time (therefore, in this
      case, the `const` qualifier is not added to the symbol array).
      
      ### Specify arbitrary `mrbc` file
      
      To realizing the "Build process: 2", enabled to specify `mrbc` created by
      another build target or pre-built` mrbc`. Use `MRuby::Build#mrbcfile =` to
      specify it explicitly. You can omit the "Build process: 1" by specifying
      pre-built `mrbc`, and you can always use an optimized build to compile Ruby
      sources faster. I think changes that affect the output of `mrbc` are rare,
      so in many cases it helps to improve efficiency.
      
      With presym, the build will be a little slower due to more build steps, but
      this feature will improve it a bit.
      
      ### Create presym files for each build target
      
      This feature was proposed at #5194 and merged once, but was reverted in
      5c205e6e due to problems especially with cross-compilation. It has been
      introduced again because this change solves the problem.
      
      The presym files will be created below.
      
      * `build/NAME/presym`
      * `build/NAME/include/mruby/presym.inc`
      
      ### Other changes
      
      * Because presym detection accuracy is greatly improved as mentioned above,
        `MRuby::Gem::Specification#cdump?` is set to true by default, and
        `disable_cdump` is added instead of `enable_cdump`. Also, support for gem
        specific presym files has been discontinued (https://github.com/mruby/mruby/issues/5151#issuecomment-730967232).
      * Previously, `mrbc` was automatically created for the `host` build, but it
        will not be created if the build target for `mrbc` mentioned above is
        automatically created. At this time, `mrbc` file of the `mrbc` build is
        copied to` bin/`.
      * Two types of `.d` files will be created, `.o.d` and `.i.d`. oThis is
        because if `.i` depends on `presym.inc`, the dependency will circulate, so
        the `.d` file cannot be shared.
      * Changed file created with `enable_cxx_exception` to `X-cxx.cxx` from
        `X.cxx` to use the mruby standard Rake rule.
      
      ### Note
      
      Almost all C sources will need to be recompiled if there are any changes to
      `persym.inc` (if not recompiled properly, it will often result in run-time
      error). If `gcc` toolchain is used, dependencies are resolved by the `.d`
      file, so it become automatically recompile target, but if not (e.g. MSVC),
      it is necessary to manually make it recompile target.
      
      Also, even if `gcc` toolchain is used, it may not become recompile target if
      external gems does not use the mruby standard Rake rule. In particular, if
      the standard rule is overwritten, such as
      https://github.com/mruby/mruby/pull/5112/files, `.d` file will not be read,
      so be careful.
      456878ba
  2. 12 Dec, 2020 1 commit
  3. 09 Dec, 2020 1 commit
  4. 08 Dec, 2020 1 commit
  5. 07 Dec, 2020 1 commit
  6. 06 Dec, 2020 2 commits
  7. 05 Dec, 2020 9 commits
  8. 04 Dec, 2020 4 commits
  9. 02 Dec, 2020 10 commits
  10. 01 Dec, 2020 3 commits
  11. 30 Nov, 2020 6 commits
  12. 29 Nov, 2020 1 commit