qhjqhj00/research-skills-pool/acol-interpreter-benchmark-eval

acol-interpreter-benchmark-eval

On the Performance of Bytecode Interpreters in Prolog — Korner et al. (2020) (arXiv:2008.12543, 2020)

What this evaluates

Evaluates the execution speed and dispatching efficiency of different interpreter implementations (AST vs. bytecode variants) for a simple imperative language (ACOL) in Prolog.

Datasets

• ACOL Interpreter Benchmarks — total 6; splits: test (6); repo https://github.com/pkoerner/prolog-interpreters

Metrics

• geometric_mean_runtime (primary) — range: other
• Geometric mean of 10 execution runs per interpreter per benchmark. Also reported as normalized runtime relative to the baseline AST interpreter.

Input / output format

Input: ACOL programs (while-loops, if-statements, assignments, comparisons, additions/subtractions) with pre-initialized environment variables.

Output: Execution time in seconds (mean ± 0.95 confidence interval over 10 runs).

Scoring recipe

runtimes = []
for _ in range(10):
    t = run_program_in_interpreter(program, interpreter)
    runtimes.append(t)
mean_runtime = geometric_mean(runtimes)
normalized_runtime = mean_runtime / baseline_ast_mean_runtime
return mean_runtime, normalized_runtime

Common pitfalls

• Compilation time is explicitly excluded; only interpreter dispatch/runtime is measured.
• Geometric mean is used for aggregation instead of arithmetic mean, as specified by the authors.
• Small handwritten programs may benefit from AST caching, skewing results compared to larger generated programs.

Evidence (verbatim from paper)

Each program was executed with every interpreter ten times. The runtime consists only of the time spent in the interpreter. Compilation time is excluded, as it is not implemented efficiently and, ultimately, not relevant. The mean value is determined by the geometric mean as proposed by [5].

Citation

@misc{korner2020bytecode,
  title={On the Performance of Bytecode Interpreters in Prolog},
  author={Korner et al. (2020)},
  year={2020},
  note={arXiv:2008.12543}
}

• arXiv: 2008.12543