CutomGPT: Auto-generate Assets Discovery Custom Patterns

The emergence of large language models (LLMs) and generative AI technologies represents a significant advancement in machine learning capabilities, facilitating a wide array of applications from content generation to customer support.

Atlassian is actively integrating this evolving technology into many of its products such as Jira Service Management (JSM), Confluence , and Bitbucket through "Atlassian Intelligence". While immensely beneficial for various tasks like creating and editing Confluence pages or composing Bitbucket commit messages, we believe that there is still potential for more specialized use-cases.

In order to assess the viability of such specialized solutions, we developed a custom Assistant capable of generating Custom Patterns for Jira Assets Discovery.

What is Jira Assets Discovery?

Jira Assets Discovery is a free Jira Marketplace app created by Atlassian for Jira Service Management. It aims to assist organizations in directly managing and tracking their assets within the service management tool. This facilitates automatic discovery, tracking, and management of all hardware, software, and other assets throughout the organization.

This process involves scanning all devices on the company's network using Scan Patterns . A pattern comprises a shell command to obtain system information along with short segments of C# code used to parse the command line output into objects that can be imported into Assets for JSM.

For instance, if one needs information about the last logged in users, a pattern would execute the last comamnd. An example command output might look like this:

mark     pts/0        10.10.0.7   Fri Feb 21 21:23   still logged in
mark     pts/0        10.10.0.7   Tue Feb 18 22:34 - 00:05  (01:31)
lisa     :0           :0          Thu Feb 13 09:19   gone - no logout
reboot   system boot  4.15.0-74-g Fri Jan 24 08:03 - 08:03  (00:00)

Subsequently, it runs the script to parse each entry from the resulting command line output in order to generate instances of Atlassian’s HostInfo object with the LastUser field set such that they can be imported into JSM.

This system offers great flexibility as it allows customization through writing patterns capable of gathering any system information obtainable via shell commands. For example: if an enterprise uses proprietary databases for specific storage needs, one can craft custom patterns to automatically retrieve details about these databases.

How do I create custom patterns if I don't know C#?

While offering an excellent method for crafting tailored solutions according to our customers' requirements, there are some drawbacks. These drawbacks mainly stem from the necessity of knowledge in C#, required shell commands, and familiarity with Assets Discovery's C# library when fulfilling non-standard inventory needs.

Often this results in our customers having difficulty implementing their own requirements or needing assistance with their implementations. Our observation indicates that although our customers often possess scripting experience - usually in languages other than C#, or even if they have expertise in C#, they lack the time required to learn how to work with Atlassian's provided library. Still, they do possess clear understanding regarding their system requirement details (e.g., what kind of information should be known about our systems?).

What's needed is a tool which can translate these requirements into complete patterns while minimizing reliance on extensive knowledge or experience with regard,¨to C#/ programming in general.

Using ChatGPT to write custom patterns

Since the C# code necessary for parsing the shell commands is quite short, we attempted tasking ChatGPT with writing up a suitable implementation based on specified set of requirements.

Sadly, results were quite unsatisfactory, to say the least. In spite of its capability of efficiently producing concise snippets 
of plain C# code, it was evident that GPT wasn't properly trained on documentation concerning Atlassian's C# library.

Referring back to our last user example: out-of-the-box GPT managed to understand that the last command can be used to retreive the data. Nonetheless, it lacked any understanding of the fields a pattern xml should have, with the pattern not containing any C# code at all. Even after telling it that a pattern has to contain C# code for parsing, GPT4 is unable to come up with any meaningful solution.

Introducing: Discovery Custom Pattern Assistant

To address the issue, we needed to provide ChatGPT with the necessary documentation. Fortunately, at that time, OpenAI introduced CustomGPTs, enabling the addition of custom knowledge files and execution of "Actions".

Custom Knowledge

When a user asks a question, ChatGPT first conducts a text similarity search of the provided knowledge base and silently attaches the search results to the user's question. This empowers the chatbot to use information it wasn't specifically trained on in order to answer questions.

As ChatGPT supports multiple knowledge file uploads, we opted to divide our data into three files:

• General documentation about Assets Discovery and Custom Patterns from Atlassian's documentation
• Our comprehensive documentation on the actual C# classes from the library
• A list of compiler errors and their respective solutions for common bugs caused by ChatGPT

After providing this relevant knowledge, there was an evident improvement: previously where AI had no understanding of how to use the library, it was now correctly utilizing classes and creating accurate objects.

While these results were already promising compared to before; following a "trust but verify" approach for AI, we proceeded by adding generated patterns into Assets Discovery. The results were notably improved compared to before but still varied. Most pattern failures were due to (mostly trivial) compiler errors. A compiler is used to translate C# source code into machine-readable code while checking for any code errors during compilation.

This implies that it's an automated process that can yield either a compiled program ready for execution or a set of errors along with their locations in the code. If we grant our custom GPT capability to compile its own code, it can automatically verify its results. In case of failure, it can then adjust its code accordingly using identified errors.

Custom Actions

Actions serve as a means for CustomGPTs to make web requests to a custom backend - granting them flexibility in using custom-built tools such as providing current weather data or conducting complex physics simulations.

In order to automatically check generated code for compiler errors, we implemented our own backend exposing an endpoint that receives a C# snippet and attempts compilation. The response includes either success confirmation or detection of error(s).

Within ChatGPT interface itself, we created an two actions facilitating interaction with our custom backend, one for C# code compilation and one for validation of the XML pattern file containing the C# code as well as some metadata fields.

Upon giving our ChatGPT the ability to both compile and self-check its own code, we evaluated our assistant once more and finally achieved outcomes exceeding our expectations.

Tying it all together

With all components in place now required building successful Custom Pattern bot, the final piece involves implementing system messages within which provides background info alongside general schema explaining pattern creation process works.

Performance evaluation of the virtual agent

In the linked chat session, we took the role of a knowledgeable Jira / JSM user without further background context in C#, or writing customer patterns.

We initiated a conversation, requesting the generation of a custom pattern given a set of requirements:

User: I need a custom pattern to get the last logged in user on MacOS devices

CustonPatternGPT: To create a custom pattern for fetching the last logged-in user on MacOS devices, you can leverage the SSH protocol, as MacOS is Unix-based and supports SSH for remote command execution. The command to get the last logged-in user on a MacOS device is last -1 -t console. This command fetches the most recent login on the device, specifically on the console, which typically indicates a physical login.

Below is a template for creating such a pattern, including all necessary using directives and adhering to the specifications for .NET Framework 4.7 or below, as well as the requirements you've provided:

<?xml version="1.0" encoding="utf-8"?>
<ScanPattern xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Version>1.0</Version>
  <PatternID>MacOS-LastUserLogin</PatternID>
  <OrderNr>0</OrderNr>
  <ProcessType>SSHExecute</ProcessType>
  <PatternType>Custom</PatternType>
  <Command>
    <![CDATA[
    last -1 -t console
    ]]>
  </Command>
  <Processing>
    <![CDATA[
    using System;
    using Insight.Discovery.InfoClasses;
    using Insight.Discovery.ProviderClasses; // Include the ProviderClasses Namespace
    using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // For handling SSH command results
    namespace Insight.Discovery {
      public class PatternExec {
        public void PerformAction(object[] parameters) {
          HostInfo hostInfo = (HostInfo)parameters[2]; // Cast the third parameter to HostInfo
          // Cast the first parameter to SSHExecuteResult and implicitly assign to string
          string sshOutput = ((SSHExecuteResult)parameters[0]);
          // Processing the SSH output to extract the last logged-in user
          if (!string.IsNullOrEmpty(sshOutput)) {
            var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
            if (lines.Length > 0) {
              var lastLoginInfo = lines[0].Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
              if (lastLoginInfo.Length > 0) {
                string lastUser = lastLoginInfo[0];
                // Setting the LastUser property of the HostInfo object
                hostInfo.LastUser = lastUser;
              }
            }
          }
        }
      }
    } 
    ]]>
  </Processing>
</ScanPattern>

This custom pattern XML defines a pattern for executing an SSH command on a MacOS device to retrieve the last logged-in user. The <Command> section contains the actual SSH command to be executed, and the <Processing> section contains C# code to parse the command's output and extract the last logged-in user's username, which is then assigned to the LastUser property of the HostInfo object. This pattern relies on the assumption that the device supports SSH access and that the executing account has the necessary permissions to execute the last command.

Remember to test this pattern in a controlled environment before deploying it in a production setting to ensure it meets your specific requirements and works as expected across different MacOS versions.

The GPT understood it's assignment and started by explaining which shell command can be used to fetch the information from the target machine, as well as giving some background on which C# version and language features to use.

In the next step, we asked our GPT to compile the C# code to check for syntax and compiler errors and then validate the pattern XML file to ensure all necessary config items are present.

User: Cool. Now, can you compile the pattern and then validate the xml to check if there aren't any issues?

CustonPatternGPT: 
Talked to custompatterngpt.hostname.com [compiling for errors]
 Taked to custompatterngpt.hostname.com [validating the pattern]

The pattern's C# code successfully compiled without any issues, indicating the code itself is syntactically correct and should work as intended.

However, when validating the XML pattern, an error occurred: Invalid PatternType: Custom. The validation process indicates that "Custom" is not a recognized PatternType. Valid pattern types include Host, Deviceinfo, Network, CPU, OS, FileSystem, Application, Patch, ApplicationService, OSProductKey, ApplicationProductKey, User, Group, ConnectedPeripherie, SNMPExtendedValues, HostInfoList, and PostProcessing.

To align with the recognized pattern types, you might consider categorizing this custom pattern under a valid type that closely matches the data it's designed to fetch. Since this pattern retrieves information about the last logged user, which is closely related to user information, a suitable PatternType might be User. If a more specific categorization is preferred and none of the listed types fit perfectly, consider using a general type like HostInfoList or PostProcessing, depending on the broader context of how you plan to use this data.

Would you like to proceed with adjusting the PatternType, or do you need further assistance with another aspect of your pattern?

The compilation was successful, however in the XML file it made the mistake to set the pattern type to Custom - which makes sense at first glance as we're writing a custom pattern. However,  Custom - is not a valid pattern type. After checking, it quickly spotted the problem and gave a bunch of options for a valid pattern type that might fit the current pattern.

Playing along with our role of a non-technical user, we just asked to pick the pattern type the GPT sees most fit.

User: Just pick the one you see fit most and please validate the pattern again

CustonPatternGPT:
Talked to custompatterngpt.hostname.com [pattern validation]

The pattern has been successfully adjusted to use the PatternType "User", which is more appropriate for the data it's designed to fetch. The validation process did not return any errors this time, indicating that the XML structure and the pattern type are now correctly defined according to the system's expectations.

<?xml version= "1.0" encoding= "utf-8"?>
<ScanPattern xmlns:xsi= "http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd= "http://www.w3.org/2001/XMLSchema">" <Version>1.0</Version>" <PatternID>MacOS-LastUserLogin</PatternID>
     <OrderNr>0</OrderNr> 
     <ProcessType>SSHExecute</ProcessType> 
     <PatternType>User</PatternType>
     <Command> 
        <![CDATA[ last -1 -t console ]]> 
    </Command> 
    <Processing> 
        <![CDATA[ 
            using System;
            using Insight.Discovery.InfoClasses;
            using Insight.Discovery.ProviderClasses; // Include the ProviderClasses Namespace
            using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // For handling SSH command results
            namespace Insight.Discovery {
            public class PatternExec {
                public void PerformAction(object[] parameters) {
                HostInfo hostInfo = (HostInfo)parameters[2]; // Cast the third parameter to HostInfo
                // Cast the first parameter to SSHExecuteResult and implicitly assign to string
                string sshOutput = ((SSHExecuteResult)parameters[0]);
                // Processing the SSH output to extract the last logged-in user
                if (!string.IsNullOrEmpty(sshOutput)) {
                    var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
                    if (lines.Length > 0) {
                    var lastLoginInfo = lines[0].Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
                    if (lastLoginInfo.Length > 0) {
                        string lastUser = lastLoginInfo[0];
                        // Setting the LastUser property of the HostInfo object
                        hostInfo.LastUser = lastUser;
                    }
                    }
                }
            }
        }
        ]]> 
    </Processing>
</ScanPattern>

You should now be able to use this pattern for discovering the last logged-in user on MacOS devices via SSH commands. If you need further assistance or have any more requirements, feel free to ask!

The pattern chose a pattern type and successfully re-validated the generated pattern. And running the pattern on a MacBook confirmed that the pattern is properly detecting and setting the last logged in user on the resulting HostInfo.

However, looking at the data, we saw that the last user was set to “jiradiscovery, -”the ssh user used to execute the patterns. This is exactly what we asked for: the last logged in user - however obviously not what we wanted:

User: Great! I just tried the pattern and it's getting the last user, however because the pattern itself is executing with ssh, the last user is jiradiscovery. Can you instead get the last user that's not the jiradiscovery SSH user? Once you're done, please re-compile to see if it's still valid

CustonPatternGPT:
Talked to custompatterngpt.hostname.com [pattern compilation]
Talked to custompatterngpt.hostname.com [pattern validation]

The updated pattern, which now filters out the "jiradiscovery" user and targets the last logged-in user that's not associated with the SSH session, has been successfully re-compiled without any issues. This means the code is syntactically correct and aligns with the desired functionality.

Additionally, I've validated the pattern against the expected XML structure and pattern requirements. The process did not return any errors, indicating that the pattern is correctly defined and should work as intended for discovering the last logged-in user on MacOS devices, excluding the "jiradiscovery" SSH user.

After explaining the issue to ChatGPT, it understood the mistake and proposed, compiled and validated a solution. Using the new refined pattern in Discovery confirmed that it's now collecting the data correctly.

As a final step, we asked the Assistant to add logging for debug purpouses and documentation that's geared towards novice programmers. This way, whoever might be tasked with maintaining the pattern in the future will have a much easier time understanding what it does:

User: Perfect, now it's working flawlessly. Finally, can you add some documentation and logging to the pattern, especially with focus on making the code understandable to a maintainer that might not know c# well?

CustonPatternGPT: To enhance the custom pattern with documentation and logging, aimed at making the code more understandable for maintainers with varying levels of C# knowledge, I've incorporated comments throughout the pattern's processing section. These comments explain the purpose of each code block and the logic behind it, along with adding logging statements to help with debugging and understanding the flow of execution:

<?xml version="1.0" encoding="utf-8"?>
<ScanPattern xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Version>1.0</Version>
  <PatternID>MacOS-LastUserLogin</PatternID>
  <OrderNr>0</OrderNr>
  <ProcessType>SSHExecute</ProcessType>
  <PatternType>User</PatternType>
  <Command>
    <![CDATA[
    last -1 -t console
    ]]>
  </Command>
  <Processing>
    <![CDATA[
    using System;
    using Insight.Discovery.InfoClasses;
    using Insight.Discovery.ProviderClasses; // Needed for executing commands on the host
    using Insight.Discovery.InfoClasses.CommandResult.ResultTypes; // To handle the results from SSH commands
    using Insight.Discovery.Tools; // To use logging services
    namespace Insight.Discovery {
      public class PatternExec {
        public void PerformAction(object[] parameters) {
          // Obtain the HostInfo object from the parameters array. This object represents the host we are currently scanning.
          HostInfo hostInfo = (HostInfo)parameters[2];
          // The SSH command's output is obtained from the parameters array and implicitly converted to a string.
          string sshOutput = ((SSHExecuteResult)parameters[0]);
          // Log the obtained SSH output for debugging purposes.
          LogService.Instance.LogDebug($"SSH Output: {sshOutput}");
          // Check if the SSH output is not empty or null.
          if (!string.IsNullOrEmpty(sshOutput)) {
            // Split the output into lines for processing.
            var lines = sshOutput.Split(new char[] { '\n' }, StringSplitOptions.RemoveEmptyEntries);
            // Iterate through each line in the SSH output.
            foreach (var line in lines) {
              // Split each line into parts to analyze the login information.
              var loginInfo = line.Split(new char[] { ' ' }, StringSplitOptions.RemoveEmptyEntries);
              // Check if the line has content and the user is not 'jiradiscovery'.
              if (loginInfo.Length > 0 && loginInfo[0] != "jiradiscovery") {
                string lastUser = loginInfo[0]; // The first part is assumed to be the username.
                // Log the found user for verification.
                LogService.Instance.LogDebug($"Last logged-in user found: {lastUser}");
                // Assign the found username to the LastUser property of the HostInfo object.
                hostInfo.LastUser = lastUser;
                // Stop processing after the first matching user is found.
                break;
              }
            }
          } else {
            // Log a warning if the SSH command did not return any output.
            LogService.Instance.LogWarning("SSH command did not return any output.");
          }
        }
      }
    }
    ]]>
  </Processing>
</ScanPattern>

This updated version of the pattern now includes detailed comments explaining the functionality of each segment of the code, making it easier for someone who may not be familiar with C# to understand the pattern's purpose and how it works. Logging statements have been added to aid in debugging and to provide insight into the pattern's operation during execution.

This documentation and logging strategy is designed to make the code more accessible and maintainable, helping future maintainers to quickly grasp the logic and purpose behind each operation.

User: Thank you, you're awesome!

CustonPatternGPT: You're welcome! I'm glad I could help. If you have any more questions or need further assistance in the future, feel free to reach out. Have a great day!