LLM Based Input Space Partitioning Testing for Library APIs
This page contains the experimental results in the ICSE 2025 submission titled "LLM Based Input Space Partitioning Testing for Library APIs".

https://github.com/LLMISP/LLMISP
Floating Menu Style

Operation Guidance

  1. Prerequisites

    • A java library that exists in maven central repository.
    • A valid OpenAI API Key.
    • Java 11 & Python 3.10 & Maven 9.

  2. Usage

    • Create a file named llm-seed-generator.cfg in the folder llm-seed-generator, refer to the file llm-seed-generator.example.cfg for details, and replace the OPENAI_KEY in it with your valid OpenAI API Key.

    • Add the java library you want to test to the dependencies in the pom.xml file in the folder llm-JQF/fuzz.

    • Run mvn install -DskipTests in the folder llm-JQF.

    • Run bin/jqf-llm -i -o {library_name} {api_name} in the folder llm-JQF.

    • library_name's format: group_id:artifact_id:version
      api_name's format: class_name.method_name(param_type_names)
      EX. bin/jqf-llm -i -o "org.apache.commons:commons-lang3:3.14.0" "org.apache.commons.lang3.StringUtils.indexOfDifference(CharSequence,CharSequence)"

Code Coverage Overall

If you want to get the coverage information for each API, you can download the full coverage table to review it.

Data Display
Library Name LISP EvoSuite 100s EvoSuite 150s EvoSuite 200s LLM-baseline
coverage

Efficiency

If you want to get efficiency for each library, you can download the full efficiency table to review it.

Covered Edge Num Valid Inputs Num Time (s) Edge / Input Time / Input

Unexpected Behaviors

If you want to get Unexpected Behaviors in each library, you can download the full UB table to review it.

Exception Type LISP EvoSuite 200s LLM-baseline
venn

Ablation Study - Coverage

Library Name LISP Ablation 1 Ablation 2
ablation