How do I design instance cleanup and data retention for BAW (BPMDB growth, GDPR)?

Question

Our BPMDB grows by gigabytes per month and legal wants personal data removed 2 years after a case closes. What is the recommended retention design and which tools implement it?

Answer 1

Retention has to be designed per process (how long instances and their data may live) and implemented with the platform's deletion tools plus your own data minimisation:

1. Classify: per process app the retention period after completion (e.g. 90 days for routine tasks, 2 years for regulated cases), which variables contain personal data, and which documents / tracking data exist outside the instance.
2. Minimise from the start: keep personal data in the system of record, in the instance only what the tasks need (claim check), clear sensitive variables in the last step, do not expose personal fields as searchable business data, and keep tracking groups free of identifiers.
3. Delete finished instances on a schedule with the Operations REST API (BAW 20+) or the instance deletion commands / Process Admin instance cleanup of earlier releases: by container, state and end date; this removes the instance, its tasks, execution contexts, documents in the BPM document store and comments.
4. PDW / BAI: tracking data is separate - prune the PDW (pdw/prune) and set retention on BAI indexes; anonymise instead of delete when statistics must survive.
5. Documents in ECM: retention policies of FileNet (record management) apply to case documents; the instance deletion does not touch ECM content.
6. Erasure requests (right to be forgotten): find instances by business data (search API), delete or anonymise the variables (PUT /process/{id}/variables with masked values for finished instances is not possible - delete them), and the user's personal data record in BAW (DELETE /ops/std/bpm/users/{id}/personal_data, BAW 20+).

# nightly retention job (BAW 20+ Operations REST): finished instances of ORD older than 90 days, then PDW prune; each returns a queue id to poll
DELETE /ops/std/bpm/processes?states=finished,terminated&container=ORD&ended_before=2025-03-01T00:00:00Z
POST   /ops/std/bpm/pdw/prune?days=365
# older releases (8.5.x): Process Admin > Instance cleanup, or the BPMDeleteInstances / SQL-based cleanup scripts IBM documents per version
# check growth: table sizes of LSW_BPD_INSTANCE_DATA, LSW_TASK_EXECUTION_CONTEXT, LSW_TASK, PDW tables

Sizing the job: delete in date windows (a week at a time) during low load; deletion is transactional per instance and slow on big execution contexts; run database statistics afterwards; keep completed snapshots installed while their instances exist. Document the retention rules in the process app's documentation - auditors ask for it.

References

• Operations REST API - instance deletion, PDW prune, personal data
• BAW documentation - deleting process instances and housekeeping