Is your feature request related to a problem? Please describe. In order to be able to detect feature usage in structured configuration files (yaml, json, xml...) it would be very useful to be able to search for occurrences of certain entries regardless of the order of declaration in the file.

An example would be:

If we have a config.yaml file and we have the following configuration:

app:
  myfeature:
     prop1: x
     prop2: x
     propN: x
     enabled: true

If I want to detect with a rule if the 'myfeature' functionality is enabled I would not have a reliable solution at the moment. Detecting if app.my-feature.enabled = true with patterns, modiffiers or conditions does not cover all cases, e.g. if the declaration order is changed or the configuration is put in one line. I.e. a pattern such as:

"pattern": "app:\s+myfeature:\s+enabled: *true".

It would not cover the above example since the order of the enabled property can be changed as it is a structured data type.

Describe the solution you'd like Ability to be able to perform patterns on structured data types.

Describe alternatives you've considered

• Implement new pattern types to support operations on structured data types (yaml, json...).
• Provide an extension mechanism to be able to do file processing with parsing tools.

Rewrite AnalyzeCommand to process files in parallel. Made the stats in Metadata threadsafe. Compile regexes for performance. Fix #205 Update multiextractor to work in parallel when possible. * Update multiextractor to catch exceptions. *

• I also contributed better implementations of these upstream to multiextractor in OSSGadget which we can incorporate once that is published as a nuget. microsoft/OSSGadget#47

Describe the bug We have experienced sometimes, apparently random, that applications inspector analysis stucks the machine (we think maybe it is a memory problem) and never ends. We always pass to the cli the argument --process-timeout, but in this cases the execution never ends. What do you think it's going on? Thanks for the help.

Expected behavior The execution of an analysis always respects the process-timeout param time.

Additional context It happens when execute the application inspector inside a github or a bamboo workflow.

Unit tests Module • 189 passing tests added covering both NuGet and CLI entry points under separate namespaces/classes • Improved overall code quality and reporting of invalid args/combinations

Overall Code Structure • Refactoring+rework for technical debt getting to 1.0 eliminating unused code, renaming, moving to isolate to right project • *Object results-based support for NuGet callers rather than json/text strings • *New support for First/Best match options for results (best match elevates higher confidence value matches) • Separation of CLI vs Operation argument options • All file output format and writters moved to CLI project • Cleaner isolation of functionality and readability • Text and Json output support added for all commands (previously limited to text for all but analyze) • Better use of inheritance to organize common arguments and override • Better use of polymorphism for checking arguments

Resolves #191

Describe the bug D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba
Microsoft Application Inspector 1.1.5+ed45aa547b Analyze command running Decompressing files... Problem while decompressing D:\emailba\WebContent\WEB-INF\lib\aws-java-sdk-core-1.11.543.jar: Unzip and try running on uncompressed files. A runtime error has occured. Please see log file for more information.

To Reproduce D:\ApplicationInspector_1.1.5>ApplicationInspector.CLI.exe analyze -s D:\emailba\

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows
• Version 10

Describe the bug I am running the App Inspector Analysis on a large repository with multiple languages, frameworks and components. The process runs fine till 50% but then it freezes there forever. And also, the appinspector.log.txt file is empty, hence have no idea what is going wrong. I tried with some moderate or small sized repository, and it works fine.

To Reproduce Steps to reproduce the behavior:

1. Clone a significantly large repository
2. Run ApplicationInspector.CLI.exe analyze -s <Repo Local Path>

Expected behavior Firstly, the analysis should be complete. If not, then some error should be posted on command line where I am running the same, or logs should be written.

Desktop (please complete the following information):

• OS: Windows 10
• Browser: Edge Chromium

Is your feature request related to a problem? Please describe. With some rules, we can get a large number of matches, but we only need to have two or three examples of that matches, and in big projects, the appinspector generates a very large .json output with all the matches.

Describe the solution you'd like Add a cli parameter like "--max-matches" that admits a number to configure a maximum number of matches reported for each tag.

Additional context I see in documentation here that talks about 'match-depth' and 'allow-duplicates' arguments, with a similar idea, but I have seen they were deleted in this release

Describe the bug Since you updated to .NET 5, the generated report files (JSON & HTML) became very very large.

To Reproduce Steps to reproduce the behavior:

1. Retrieve https://github.com/OrchardCMS/OrchardCore as a test application (git clone)
2. Analyze the repo dotnet ApplicationInspector.CLI.dll analyze -s "/folder/to/OrchardCore" -v Info -x high -c high,medium -k lib,.vs,.git,.idea
3. You will obtain one 611MB-large HTML file and one 564MB large JSON file

Expected behavior With AppInspector 1.2.89 (version I keep using) the same files had a size of around 3 MB.

Desktop

• OS: macOS BigSur 11.3.1 (20E241)
• Version: App Inspector 1.3.17 dotnet_core distribution (leveraging dotnet 5.0.203)

Description Following the documentation here, I wanted to see detail of Classes and Functions in the reports, so I created this tagcounters.json file in the preferences directory:

[
    {
      "tag": "Metric.Code.Class.Defined",
      "displayName": "Classes",
      "includeAsMatch": true
    },
    {
      "tag": "Metric.Code.Function.Defined",
      "displayName": "Functions",
      "includeAsMatch": true
    },
    {
      "tag": "Metric.Code.Exception.Caught",
      "displayName": "Exceptions",
      "includeAsMatch": true
    },
    {
      "tag": "Metric.Code.HTMLForm.Defined",
      "displayName": "Forms",
      "includeAsMatch": true
    },
    {
      "tag": "Dependency.SourceInclude",
      "displayName": "Dependencies",
      "includeAsMatch": false
    },
    {
      "tag": "Miscellaneous.CodeHygiene.Comment.Todo",
      "displayName": "Comments",
      "includeAsMatch": false
    },
    {
      "tag": "Miscellaneous.CodeHygiene.Comment.Fix",
      "displayName": "Fix-comments",
      "includeAsMatch": true
    },
    {
      "tag": "Miscellaneous.CodeHygiene.Comment.Suspicious",
      "displayName": "Suspicious-comments",
      "includeAsMatch": true
    },
    {
      "tag": "Metric.Code.Logging.Call",
      "displayName": "Logging Calls",
      "includeAsMatch": false
    },
    {
      "tag": "Metric.Code.URL",
      "displayName": "URL's",
      "includeAsMatch": false
    }
  ]

and then added this to tagreportgroups.json:

      {
        "title": "Code Metrics",
        "dataRef": "CodeMetrics",
        "patterns": [
          {
            "searchPattern": "Metric.Code.Logging.Call",
            "displayName": "Code Logging",
            "detectedIcon": "fab fa-dev"
          },
          {
            "searchPattern": "Metric.Code.Class.Defined",
            "displayName": "Class Defined",
            "detectedIcon": "fab fa-dev"
          },
          {
            "searchPattern": "Metric.Code.Exception.Caught",
            "displayName": "Exception Caught",
            "detectedIcon": "fab fa-dev"
          },
          {
            "searchPattern": "Metric.Code.URL",
            "displayName": "Code URL",
            "detectedIcon": "fab fa-dev"
          },
          {
            "searchPattern": "Metric.Code.Function.Defined",
            "displayName": "Function Defined",
            "detectedIcon": "fab fa-dev"
          }                                       
        ]
      } 

Running an analysis results in the custom Code Metrics section being added to the report, but no results for Class Defined or Function Defined are shown.

Furthermore on the Summary - Tag Counters screen, it displays:

• Metric.Code.Function.Defined: Count: 119 Include as Match: false
• Metric.Code.Class.Defined - Count: 18 Include as Match: false

So it appears to be picking up the customisations in tagreportgroups.json, but not in tagcounters.json.

Screenshots

Desktop

• macOS Catalina
• Google Chrome v85
• Application Inspector v1.2.63

Describe the bug Icons are not being grayed out when they are not present.

To Reproduce Get patch from #265 first to ensure resources are loaded.

Expected behavior Icons for properties that don't apply should be gray not blue.

Screenshots

Describe the bug After installing the Microsoft.CST.ApplicationInspector.Command pkg and running the containing app, the calls fail due to the default rules folder missing from where the dll is copied to i.e. the containing app \bin\debug or \bin\release folders causing an exception. A default or custom rules path is expected. The caller wanting to use the default ruleset would have to copy them manually from the nuget install location locally into their local app bin folder

To Reproduce Steps to reproduce the behavior:

1. Download the pkg from Nuget via Visual Studio into an app
2. Create a AnalyzeCommandOption object
3. Create an AnalyzeCommand object
4. Call GetResults method on the command object. See the exception from step 3.

Expected behavior User does not have to manually copy over the Rules folder into their bin folders for use.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

Is your feature request related to a problem? Please describe. A large portion of the execution time for AppInspector is allocating memory for the FullContent field of TextContainer. We should test the performance uplift we can get by using StreamRegex.

Difficult portions here will be properly implementing the conditions based behavior from OAT and optimizing number of passes through the stream with multiple rules.

Is your feature request related to a problem? Please describe. App inspector deserializes using Newtonsoft.Json, accordingly it ignores System.Text.Json attributes. If you were to extend an AI rule with more attributes and then call AddString, even if you the T is specified the deserialization may not be correct if your json properties dont match the attribute names or if they have special handling.

Any downstream users who implement extensions to AI (like DevSkim) are then required to include newtonsoft for the annotations to use the AddString methods.

Describe the solution you'd like Ideally AppInspector could just use System.Text.Json for serialization in the rules path, using the standard attributes.

Provide an option flag on the RuleProcessor to use non backtracking regex.

https://devblogs.microsoft.com/dotnet/regular-expression-improvements-in-dotnet-7/#backtracking-and-regexoptions-nonbacktracking

Rule self-tests only confirm that the rule as a whole matched a sample. It may be the case that a rule looks for multiple different things in the same category and you'd like to have self-tests to audit that you have samples for every pattern in every rule. The same perhaps for conditions.

Other potential enhancements:

1. Mechanism to indicate where the expected match in the must-match should be

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Download the OWASP Benchmark by git clone https://github.com/OWASP-Benchmark/BenchmarkJava.
2. Change the directory to the benchmark: cd BenchmarkJava.
3. Run ApplicationInspector with appinspector analyze -s ./.
4. See the error.

Expected behavior

Get the result file from the benchmark.

Screenshots

Operating Environment (please complete the following information):

• Application Inspector Version: ApplicationInspector.CLI 1.5.20+3790d0a4cb
• OS: WSL 2 on Windows 10 [version 10.0.19044.1826].
• .NET version: 6.0.302