Embedded Lisp Dialect.

Arl Test Suite

This directory contains the test suite for the Arl language implementation. Tests are organized into two main categories: R tests and native tests.

Test Organization

tests/
├── testthat/              # R-based tests using testthat framework
│   ├── test-*.R           # R test files
│   ├── helper-*.R         # R test helpers
│   └── helper-native.arl  # Arl helpers for native tests (skip, etc.)
├── native/                # Native Arl tests (.arl files)
│   └── test-*.arl         # Native test files
└── skip-examples.arl      # Example usage of skip() (not a test)

R Tests (testthat/)

Traditional R tests using the testthat framework. These tests:

• Test the R implementation of the Arl engine
• Test R interop and integration
• Test stdlib functions from the R side
• Use standard testthat assertions (expect_equal, expect_error, etc.)

Example:

test_that("lambda creates functions", {
  engine <- Engine$new()
  env <- new.env()
  result <- engine$eval(
    engine$read("((lambda (x) (* x 2)) 5)")[[1]], env)
  expect_equal(result, 10)
})

Native Tests (native/)

Native Arl tests written in Arl itself. These tests:

• Test Arl language features and semantics from within Arl
• Test stdlib functionality using Arl assertions
• Provide examples of idiomatic Arl code
• Run faster than R tests for pure Arl logic

How Native Tests Work

1. Test files are .arl files in tests/native/ directory
2. Test functions are defined with names starting with test-
3. Test runner (helper-native.R) automatically discovers and runs all test-* functions
4. Assertions use functions from the assert module (assert-equal, assert-true, etc.)

Writing a Native Test

Create a file tests/native/test-feature.arl:

;;; Tests for my feature

(define test-basic-functionality (lambda ()
  ;; Test that 1 + 1 = 2
  (assert-equal 2 (+ 1 1))))

(define test-edge-case (lambda ()
  ;; Test empty list behavior
  (assert-true (null? (list)))))

(define test-error-handling (lambda ()
  ;; Test that calling a non-existent function errors
  (assert-error (lambda ()
    (error "something went wrong")))))

Key points:

• Each test is a function named test-something
• Use assert-equal, assert-true, assert-false, assert-eq, assert-error
• Tests run in a fresh environment with stdlib loaded
• Tests are isolated from each other

Available Assertions

From the assert module (automatically available):

• (assert condition [message]) - Assert condition is truthy
• (assert-equal expected actual) - Assert structural equality (uses equal?)
• (assert-eq expected actual) - Assert identity (uses identical?)
• (assert-true value) - Assert value is truthy
• (assert-false value) - Assert value is falsy
• (assert-error thunk) - Assert thunk throws an error

Skipping Tests

Use skip() to mark a test as skipped (only available in native tests):

(define test-future-feature (lambda ()
  (skip "Not yet implemented")
  ;; Code after skip is never reached
  (assert-equal 1 2)))

(define test-platform-specific (lambda ()
  ;; Skip on Windows
  (define os (r-call "Sys.info" (list)))
  (if (== (r-call "[[" (list os "sysname")) "Windows")
    (skip "Not supported on Windows")
    (assert-equal 1 1))))

(define test-requires-new-r (lambda ()
  ;; Skip if R version too old
  (if (< (r-call "getRversion" (list)) (r-call "package_version" (list "4.5")))
    (skip "Requires R >= 4.5")
    (assert-equal 1 1))))

Note:skip() is defined in testthat/helper-native.arl and calls testthat::skip(), so it integrates with the R test framework. Skipped tests appear in test output but don't fail the suite.

Test Infrastructure

Native tests are run by helper-native.R:

1. Creates a fresh Engine with stdlib loaded
2. Loads helper-native.arl to provide test utilities (like skip)
3. Discovers all .arl files in native/
4. Loads each file and finds all test-* functions
5. Runs each test inside a test_that() block
6. Catches errors and reports failures/skips

Running Tests

# Run all tests
devtools::test()

# Run only R tests
devtools::test(filter = "^(?!native)")

# Run only native tests
devtools::test(filter = "native")

# Run specific test file
testthat::test_file("tests/testthat/test-engine.R")

Test Guidelines

When to Write R Tests vs Native Tests

Use R tests when:

• Testing the R engine implementation
• Testing R interop features
• Testing integration with R packages
• Need to test R-specific edge cases

Use native tests when:

• Testing Arl language semantics
• Testing stdlib functions from user perspective
• Demonstrating idiomatic Arl patterns
• Testing pure Arl logic without R concerns

General Guidelines

• Keep tests focused and independent
• Use descriptive test names that explain what's being tested
• Group related tests in the same file
• Add comments explaining non-obvious test logic
• Test both success and failure cases
• Consider edge cases (empty lists, null, zero, etc.)

Avoiding Test Duplication

The test suite includes both R tests (tests/testthat/) and native tests (tests/native/), each serving different purposes. To avoid unnecessary duplication and maintenance burden, follow these principles:

Native Tests Should

• Demonstrate idiomatic Arl usage patterns - Show how features are meant to be used
• Test language semantics - Focus on binding, scoping, closures, control flow, quoting
• Show 1-3 simple examples per stdlib function - Demonstrate happy path usage
• Serve as executable documentation - Be readable and exemplary
• Be concise and clear - Each test should illustrate a specific pattern

Example of a good native test:

(define test-string-upcase (lambda ()
  (assert-equal (string-upcase "hello") "HELLO")))

Native Tests Should NOT

• Test comprehensive edge cases - Empty strings, NaN, boundary conditions belong in R tests
• Verify implementation details - Caching behavior, optimization, internal state
• Duplicate detailed R test coverage - If R tests comprehensively cover something, native tests should show only 1-2 examples
• Test system integration - Actual file I/O, R interop details
• Test error handling exhaustively - Basic error cases are OK, but comprehensive error testing belongs in R tests

R Tests Should

• Test R implementation correctness - Engine, compiler, optimizer behavior
• Test comprehensive edge cases and error handling - All boundary conditions, error messages
• Verify R interop and module system - Integration with R, module loading
• Test actual file I/O and system integration - Real filesystem operations
• Verify performance characteristics - Memoization caching, optimization effects

Example of a good R test:

test_that("string-upcase handles edge cases", {
  env <- new.env()
  toplevel_env(engine, env)

  expect_equal(env$`string-upcase`("hello"), "HELLO")
  expect_equal(env$`string-upcase`(""), "")
  expect_equal(env$`string-upcase`("ALREADY"), "ALREADY")
  expect_error(env$`string-upcase`(NULL))
})

Acceptable Overlap

• Core language features can be tested in both suites (different perspectives)
• Important stdlib functions can have basic native examples + comprehensive R tests
• The overlap should be intentional, not accidental - consider whether duplication adds value

Before Writing a Test

1. Check if it's already covered - Search both native/ and testthat/ directories
2. Ask: "What am I testing?"
   • Language semantics or usage pattern → Native test
   • Implementation correctness or edge case → R test
3. For stdlib functions:
   • Native: 1-2 clear examples showing typical usage
   • R: Comprehensive edge cases, error handling, integration
4. When in doubt, prefer R tests for comprehensive testing, reserve native tests for demonstrating idiomatic patterns

Debugging Test Failures

For R tests, use standard R debugging:

testthat::test_file("tests/testthat/test-file.R")
# Set breakpoints, use browser(), etc.

For native tests, you can load and run them manually:

engine <- Engine$new()
env <- engine$get_env()
source("tests/testthat/helper-native.R")  # Loads skip() etc.
engine$load_file_in_env("tests/native/test-something.arl")
# Now you can call test functions directly:
env$`test-my-function`()

Contributing Tests

When adding new features:

1. Add R tests for the implementation
2. Add native tests demonstrating usage
3. Ensure all existing tests still pass
4. Consider edge cases and error conditions

When fixing bugs:

1. Add a test that reproduces the bug (should fail initially)
2. Fix the bug
3. Verify the test now passes
4. Ensure no other tests broke.