An Issue We Encountered During Testing With Spring Batch using DBUnit

An Issue We Encountered During Testing With Spring Batch using DBUnit

Introduction

Hello. I am Takehana from the Payment Platform Team, Common Service Development Group^{[1][2][3][4][5][6]} at the Platform Development Division.
This time, I would like to write about an issue that we encountered while testing with Spring Batch + DBUnit.

Environment

Encountered Issues

• We are using DB unit for testing Spring Boot 3 with Spring Batch.
• The Batch process follows the Chunk model, where ItemReader performs DB searches, and ItemWriter updates the DB.
• Given this setup, when running tests with data volumes exceeding the Chunk size, the tests did not complete...

Investigations and Attempts

Observations

• Code

new StepBuilder("step", jobRepository)
        .<InputDto, OutputDto>chunk(
            CHUNK_SIZE, transactionManager)
        .reader(reader)
        .processor(processor)
        .writer(writer)
        .build();

I was testing a batch with the steps mentioned above as follows.

@SpringBatchTest
@SpringBootTest
@TestPropertySource(
    properties = {
      "spring.batch.job.names: Foobar-batch",
      "targetDate: 2023-01-01",
    })
@Transactional(isolation = Isolation.SERIALIZABLE)
@TestExecutionListeners({
  DependencyInjectionTestExecutionListener.class,
  DirtiesContextTestExecutionListener.class,
  TransactionDbUnitTestExecutionListener.class
})
@DbUnitConfiguration(dataSetLoader = XlsDataSetLoader.class)
class FoobarBatchJobTest {

    @Autowired
    private JobLauncherTestUtils jobLauncherTestUtils;

    @BeforeEach
    void setUp() {
    }

    @Test
    @DatabaseSetup("classpath:dbunit/test_data_import.xlsx")
    @ExpectedDatabase(
            value = "classpath:dbunit/data_expected.xlsx",
            assertionMode = DatabaseAssertionMode.NON_STRICT_UNORDERED)
    void launchJob() throws Exception {
        val jobExecution = jobLauncherTestUtils.launchJob();
        assertEquals(ExitStatus.COMPLETED, jobExecution.getExitStatus());
    }
}

When I set the test data to be less than the chunk size, the test passed without any issues.
However, when the test data exceeded the chunk size, the test froze and never completed.
(This occurred even with a chunk size of 1 and a data count of 1)

• Suspecting the issue might be on DB connections, I noted that Spring Batch treats each chunk as a single transaction. If processing in parallel, it would require more DB connections than the number of concurrent executions. So I adjusted the pool size to test this hypothesis.

spring:
  datasource:
    hikari:
      maximum-pool-size: I changed 10 to 100 among other adjustments,

but the issue was still not resolved…

Start debugging

I set up debug logs and ran the application to observe the behavior.
The execution seemed to stop at the log output on line 88 of org.springframework.batch.core.step.item.ChunkOrientedTasklet. So, I set a breakpoint to verify.

I then reached line 408 of org.springframework.batch.core.step.tasklet.TaskletStep.

It appeared that the semaphore couldn’t acquire a lock (= waiting for the lock to be released), causing the execution to halt there.

Delving deeper into Spring Batch

Continuing my investigation, I traced the flow of execution in the step processing. The rough outline of the relevant parts is as follows.

1. Execute doExecute of TaskletStep
2. Create a semaphore
3. Pass the semaphore to ChunkTransactionCallback, which is an implementation of TransactionSynchronization, link it with the transaction execution, and configure it in RepeatTemplate
4. Step processing begins for the chunk
5. The semaphore is locked in doInTransaction of TaskletStep
6. Execute the main step processing
7. The commit is executed by TransactionSynchronizationUtils`
8. The AbstractPlatformTransactionManager ’s triggerAfterCompletion method is called, and the in-process invokeAfterCompletion` is executed.
9. The semaphore is released in the afterCompletion method of the ChunkTransctionCallback by the invokeAfterCompletion.
10. If data remains, return to 4

During this test run, the semaphore of 9 was not released, and it passed through 4 again and ended up freezing at 5.

Why was the semaphore not released...?
During the review mentioned above, at Step semaphore release, I found the following condition in the relevant code.

status.isNewSynchronization() did not become true, so invokeAfterCompletion was not executed.

org.springframework.transaction.support.DefaultTransactionStatus#isNewSynchronization is as follows:

/**
 * Return if a new transaction synchronization has been opened
 * for this transaction.
 */
public boolean isNewSynchronization() {
    return this.newSynchronization;
}

It returns whether a new transaction synchronization has been opened for this transaction.

Considerations

The current situation is that we haven’t fully traced yet why isNewSynchronization doesn’t become true.
However, I thought I might be able to find some clues in the logs from our various trial and error attempts.

If @Transactional is not applied to the test class

2024-03-27T08:57:14.527+0000 [Test worker] TRACE o.s.t.i.TransactionInterceptor - Completing transaction for [org.springframework.batch.core.repository.support.SimpleJobRepository.update] Foobar-batch 19
2024-03-27T08:57:14.527+0000 [Test worker] DEBUG o.s.orm.jpa.JpaTransactionManager - Initiating transaction commit Foobar-batch 19
2024-03-27T08:57:14.527+0000 [Test worker] DEBUG o.s.orm.jpa.JpaTransactionManager - Committing JPA transaction on EntityManager [SessionImpl(1075727694<open>)] Foobar-batch 19
2024-03-27T08:57:14.534+0000 [Test worker] DEBUG o.s.orm.jpa.JpaTransactionManager - Closing JPA EntityManager [SessionImpl(1075727694<open>)] after transaction Foobar-batch 19
2024-03-27T08:57:14.536+0000 [Test worker] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=2 Foobar-batch 19

If @Transactional is applied to the test class

2024-03-27T09:04:04.600+0000 [Test worker] TRACE o.s.t.i.TransactionInterceptor - Completing transaction for [org.springframework.batch.core.repository.support.SimpleJobRepository.update] Foobar-batch 20
2024-03-27T09:04:04.601+0000 [Test worker] DEBUG o.s.b.repeat.support.RepeatTemplate - Repeat operation about to start at count=2 Foobar-batch 20

When @Transactional is applied, "Initiating transaction commit..." from JpaTransactionManager with @Transactionalis not being logged.
The test class uses TransactionalTestExecutionListener and executes within the same transaction using @Transactional.
This ensures that the test data registered with DBUnit is accessible to code under test and is rolled back after the test is completed.
However, I concluded that isNewSynchronization does not become true because existing transactions are being reused (a new transaction is not started) when the same step is executed.

Workaround

• As a brute-force workaround to avoid using TransactionalTestExecutionListener, I performed the cleanup manually after each test, which successfully prevented the freeze.

class FoobarTestExecutionListenerChain extends TestExecutionListenerChain {
  private static final Class<?>[] CHAIN = {
    FoobarTransactionalTestExecutionListener.class, DbUnitTestExecutionListener.class
  };

  @Override
  protected Class<?>[] getChain() {
    return CHAIN;
  }
}

class HogeTransactionalTestExecutionListener implements TestExecutionListener {

    private static final String CREATE_BACKUP_TABLE_SQL =
            "CREATE TEMPORARY TABLE backup_%s AS SELECT * FROM %s";

    private static final String TRUNCATE_TABLE_SQL = "TRUNCATE TABLE %s";

    private static final String BACKUP_INSERT_SQL = "INSERT INTO %s SELECT * FROM backup_%s";

    private static final List<String> TARGET_TABLE_NAMES =
            List.of(
                    "Foobar",
                    "fuga",
                    "dadada");

    /**
     * Create a test working table
     *
     * @param testContext
     * @throws Exception
     */
    @Override
    public void beforeTestMethod(TestContext testContext) throws Exception {
        val dataSource = (DataSource) testContext.getApplicationContext().getBean("dataSource");
        val jdbcTemp = new JdbcTemplate(dataSource);

        // Backup existing data to a temporary table before testing
        TARGET_TABLE_NAMES.forEach(
                tableName ->
                        jdbcTemp.execute(String.format(CREATE_BACKUP_TABLE_SQL, tableName, tableName)));

        // Table initialization
        TARGET_TABLE_NAMES.forEach(
                tableName -> jdbcTemp.execute(String.format(TRUNCATE_TABLE_SQL, tableName)));
    }

    /**
     * Drop the test working table
     *
     * @param testContext
     * @throws Exception
     */
    @Override
    public void afterTestMethod(TestContext testContext) throws Exception {
        val dataSource = (DataSource) testContext.getApplicationContext().getBean("dataSource");
        val jdbcTemp = new JdbcTemplate(dataSource);
        // Restore the table
        TARGET_TABLE_NAMES.forEach(
                tableName -> jdbcTemp.execute(String.format(TRUNCATE_TABLE_SQL, tableName, tableName)));

        TARGET_TABLE_NAMES.forEach(
                tableName -> jdbcTemp.execute(String.format(BACKUP_INSERT_SQL, tableName, tableName)));
    }
}

Remove TransactionDbUnitTestExecutionListener and avoid using TransactionalTestExecutionListener.
(Use DbUnitTestExecutionListener to lode the test data from Excel)
Create a custom TestExecutionListener and move data from the target table to a temporary table during pre-processing, then restore it after the test.
beforeTestMethod is executed before the test method, and afterTestMethod is executed after the test method.
This approach made it possible to run tests while preserving Spring’s transaction management.

Impressions

Despite extensive searches, I couldn’t find satisfactory information, leaving the issue in a state of uncertainty.
However, by looking further into the Spring Boot source code, I made various discoveries and it turned out to be a valuable learning experience through code reading.
(Although I haven’t fully grasped everything yet…)

I was wondering if I was fundamentally misunderstanding how to use Spring and the test libraries,
questioning whether I was implementing them correctly based on the library creators’ assumptions and if there were more suitable classes available.
This has highlighted that I still have much to learn.
I would like to continue to approach exploration and improvement with the same curiosity, asking, “How does this work?”

Thank you for reading this article.
I hope this will be helpful to others facing similar issues.