Testing Testing Strategy - hiraishikentaro/rails-factorybot-jump GitHub Wiki

Testing: Testing Strategy

Testing Philosophy

Rails FactoryBot Jump follows a comprehensive testing strategy that emphasizes reliability, maintainability, and realistic testing environments. The strategy balances unit testing for isolated components with integration testing for VSCode API interactions.

Testing Framework Architecture

Core Testing Stack

Primary Framework: Mocha 10.8.2

JavaScript testing framework with flexible test organization
Support for asynchronous testing
Rich assertion library integration
Custom timeout handling for VSCode operations

VSCode Testing Environment: @vscode/test-electron 2.4.1

Runs tests in actual VSCode Extension Development Host
Full VSCode API access during testing
Realistic testing environment with file system integration
Cross-platform testing support

Mocking Framework: Sinon 17.0.1

Comprehensive mocking and stubbing capabilities
VSCode API mocking for isolated unit tests
File system operation mocking
Call verification and behavior testing

Test Runner Architecture

Dual Test Runner System:

The extension implements separate test runners for different testing needs:

Unit Test Runner (src/test/runUnitTests.ts):

Focused on isolated component testing
Fast execution without VSCode environment overhead
Comprehensive mocking of external dependencies
Suitable for TDD workflows

Integration Test Runner (src/test/runTest.ts):

Full VSCode Extension Development Host environment
Real API interactions and file system operations
End-to-end workflow testing
Realistic user scenario simulation

Source: package.json#L78,L84,L88

Test Environment Configuration

Test Runner Setup (src/test/runTest.ts):

import { runTests } from "@vscode/test-electron";

async function main() {
  const extensionDevelopmentPath = path.resolve(__dirname, "../../");
  const extensionTestsPath = path.resolve(__dirname, "./suite/index");

  await runTests({
    extensionDevelopmentPath,
    extensionTestsPath,
  });
}

Benefits:

Real VSCode environment testing
Actual extension activation and deactivation
True file system operations
Authentic user interaction simulation

Source: src/test/runTest.ts

Testing Levels

1. Unit Testing

Scope: Individual components and functions in isolation

Target Components:

Models: Factory, Location, and Trait data models
Services: CacheManager, FactoryParser, FileSearcher, ConfigurationManager, ErrorNotificationService
Utils: RegexPatterns, PathUtils
Constants: Default configuration values

Test Structure (src/test/unit/):

unit/
├── index.ts                 # Unit test suite configuration
├── models/                  # Model unit tests
│   ├── factory.test.ts
│   ├── location.test.ts
│   └── trait.test.ts
├── services/                # Service unit tests
│   ├── cacheManager.test.ts
│   └── factoryParser.test.ts
└── utils/                   # Utility unit tests
    └── regexPatterns.test.ts

Mocking Strategy:

// Example: Mock VSCode workspace API
const mockWorkspace = {
  findFiles: sinon.stub(),
  getConfiguration: sinon.stub(),
  createFileSystemWatcher: sinon.stub(),
};

// Example: Mock file system operations
const mockFs = {
  readFile: sinon.stub(),
  stat: sinon.stub(),
};

Isolation Benefits:

Fast test execution
Predictable test conditions
Easy debugging of specific logic
Clear failure attribution

2. Integration Testing

Scope: Component interactions with VSCode APIs and file system

Integration Points:

Document link provider registration
File system watcher functionality
Configuration change handling
Command execution workflow

Real Environment Testing:

// Example: Test with real VSCode document
const document = await vscode.workspace.openTextDocument(testFileUri);
const provider = new FactoryLinkProvider();
const links = provider.provideDocumentLinks(document);

Integration Benefits:

Realistic API behavior testing
Cross-component interaction verification
End-to-end workflow validation
Platform-specific behavior testing

3. System Testing

Scope: Complete extension functionality in realistic scenarios

Test Scenarios:

Extension activation and deactivation
Factory file discovery and parsing
Link generation and navigation
Configuration updates and cache refresh

User Workflow Simulation:

// Simulate complete user workflow
await activateExtension();
await openRubyTestFile();
await hoverOverFactoryCall();
await clickFactoryLink();
await verifyNavigationToFactory();

Test Categories

1. Core Functionality Tests

Factory Detection Tests:

suite("Factory Detection", () => {
  test("detects basic factory calls", () => {
    const text = "user = create(:user)";
    const matches = detectFactoryCalls(text);
    assert.strictEqual(matches.length, 1);
    assert.strictEqual(matches[0].factoryName, "user");
  });

  test("detects factory calls with traits", () => {
    const text = "admin = create(:user, :admin, :verified)";
    const matches = detectFactoryCalls(text);
    assert.strictEqual(matches[0].factoryName, "user");
    assert.deepStrictEqual(matches[0].traits, ["admin", "verified"]);
  });
});

Cache Management Tests:

suite("Cache Management", () => {
  test("builds factory cache from files", async () => {
    const provider = new FactoryLinkProvider();
    await provider.initializeFactoryFiles();
    const cache = provider.getFactoryCache();
    assert.ok(cache.has("user"));
    assert.ok(cache.has("post"));
  });

  test("updates cache on file changes", async () => {
    // Test cache invalidation and rebuild
  });
});

2. Configuration Tests

Settings Handling Tests:

suite("Configuration", () => {
  test("uses default factory paths", () => {
    const config = getDefaultConfiguration();
    assert.deepStrictEqual(config.factoryPaths, ["spec/factories/**/*.rb"]);
  });

  test("respects custom factory paths", async () => {
    await updateConfiguration({
      factoryPaths: ["test/factories/**/*.rb", "lib/factories/**/*.rb"],
    });
    const provider = new FactoryLinkProvider();
    const paths = provider.getFactoryPaths();
    assert.strictEqual(paths.length, 2);
  });
});

3. Edge Case Tests

Error Handling Tests:

suite("Error Handling", () => {
  test("handles missing factory files gracefully", async () => {
    // Mock file system to return empty results
    const provider = new FactoryLinkProvider();
    await provider.initializeFactoryFiles();
    // Should not throw errors
  });

  test("handles malformed factory files", async () => {
    // Test with invalid Ruby syntax
  });

  test("handles permission errors", async () => {
    // Mock file system permission errors
  });
});

Performance Tests:

suite("Performance", () => {
  test("handles large numbers of factory files", async () => {
    // Test with 100+ factory files
    const startTime = Date.now();
    await provider.initializeFactoryFiles();
    const duration = Date.now() - startTime;
    assert.ok(
      duration < 5000,
      "Initialization should complete within 5 seconds"
    );
  });

  test("provides links efficiently for large documents", () => {
    // Test with large test files
  });
});

Test Data Management

1. Mock Factory Files

Test Factory Definitions:

# test/fixtures/factories/users.rb
FactoryBot.define do
  factory :user do
    name { "Test User" }
    email { "[email protected]" }

    trait :admin do
      admin { true }
    end

    trait :verified do
      verified { true }
    end
  end
end

Test File Management:

// Create temporary test files
const testFactoryFile = await createTestFile(
  "factories/users.rb",
  factoryContent
);
const testSpecFile = await createTestFile("spec/user_spec.rb", specContent);

// Cleanup after tests
after(() => {
  cleanupTestFiles();
});

2. Mock VSCode Environment

Document Mocking:

const mockDocument = {
  uri: vscode.Uri.file("/test/spec/user_spec.rb"),
  getText: () => "user = create(:user, :admin)",
  lineAt: sinon.stub(),
  positionAt: sinon.stub(),
};

Configuration Mocking:

const mockConfiguration = {
  get: sinon
    .stub()
    .withArgs("factoryPaths")
    .returns(["spec/factories/**/*.rb"]),
};

Continuous Integration Testing

1. Multi-Platform Testing

Platform Matrix:

Ubuntu Latest: Linux compatibility testing
macOS Latest: macOS compatibility testing
Windows Latest: Windows compatibility testing

Platform-Specific Concerns:

Path separator handling (POSIX vs Windows)
File permission behaviors
VSCode API platform differences

2. Node.js Version Testing

Supported Versions:

Node.js 22.x (current testing version)
Node.js 18.x (minimum supported LTS)
Future Node.js versions (compatibility testing)

Version-Specific Testing:

ES2020 feature availability
API compatibility
Performance characteristics

3. Automated Test Execution

Separate GitHub Actions Workflows:

Unit Test Workflow (.github/workflows/unit_test.yml):

jobs:
  unit-test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npm run test:unit

Integration Test Workflow (.github/workflows/integration_test.yml):

jobs:
  integration-test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: xvfb-run -a npm run test:integration
        if: runner.os == 'Linux'
      - run: npm run test:integration
        if: runner.os != 'Linux'

Test Maintenance Strategy

1. Test Organization

Suite Structure:

test/
├── runTest.ts              # Integration test runner
├── runUnitTests.ts         # Unit test runner
├── suite/                  # Integration tests
│   ├── index.ts           # Integration test configuration
│   └── extension.test.ts  # Integration test cases
├── unit/                   # Unit tests
│   ├── index.ts           # Unit test configuration
│   ├── models/            # Model unit tests
│   ├── services/          # Service unit tests
│   ├── utils/             # Utility unit tests
│   └── unitTestsRunner.ts # Unit test execution logic
└── fixtures/               # Test data files

2. Test Quality Metrics

Coverage Goals:

Core functionality: 95%+ coverage
Error handling paths: 80%+ coverage
Configuration scenarios: 90%+ coverage

Quality Indicators:

Unit tests: under 10 seconds execution
Integration tests: under 30 seconds execution
Zero flaky tests
Clear test failure messages
Minimal test interdependencies

Test Execution Commands:

npm test - Run integration tests (default)
npm run test:unit - Run unit tests only
npm run test:integration - Run integration tests only
npm run test:all - Run both unit and integration tests
npm run test:watch - Run tests in watch mode

3. Regression Testing

Change Impact Testing:

New feature tests
Backward compatibility tests
Performance regression tests

Automated Regression Detection:

Baseline performance metrics
API contract verification
User workflow preservation

Testing Best Practices

1. Test Independence

Isolation Principles:

Each test can run independently
No shared mutable state between tests
Clean setup and teardown for each test

2. Descriptive Test Names

Naming Convention:

test("should detect factory calls with multiple traits in parentheses syntax");
test("should handle missing factory files without throwing errors");
test("should update cache when factory file is modified");

3. Behavior-Driven Testing

Test Structure:

test("should navigate to correct factory definition", async () => {
  // Given: A test file with factory call
  const testFile = createTestFile("user = create(:user)");

  // When: User clicks on factory link
  const links = provider.provideDocumentLinks(testFile);
  const factoryLink = links[0];

  // Then: Should navigate to factory definition
  assert.strictEqual(factoryLink.target.path, "/factories/users.rb");
});

This comprehensive testing strategy ensures the extension maintains high quality, reliability, and performance across all supported platforms and use cases.