How to write a processing specification
A specification is the difference between a job that runs reliably and one that depends on whoever is around. Start with the output: describe the file you need, column by column, and who uses it. Then work backwards. For each output column, name the source column and write the transformation in a sentence. "Order date, from the source column Date, converted to day/month/year." Where you cannot write the sentence, you have found a decision that needs making.
Next, list the checks. What makes a row invalid? A missing reference, a negative quantity, a date in the future? For each, decide whether the row is corrected by rule, held for review, or rejected. Finally, write the job-level details: schedule, retention, recipients, and what volume change should pause a run.
Deciding how long processed files should be kept
Keep files long enough to be useful and no longer. A reasonable approach is to ask three questions. How far back do you compare runs? How long after a run could someone reasonably query a figure? Is there an obligation, set by your own advisers, to keep a record for a fixed period? The longest genuine answer sets the archive period; working copies should not outlive the run. We will implement what you decide, but deciding retention obligations is for you and your advisers, not for us.
A plain glossary
- Batch
- A set of records processed together in one run.
- Validation
- Checking that a value meets a rule, such as being a date.
- Standardisation
- Rewriting valid values into one consistent form.
- Held row
- A record set aside for human review because it failed a check.
- Reconciliation
- Comparing two sources to find agreement and differences.
- Mapping table
- A record of how each source field becomes a target field.
- Run note
- A short report of what one run received, changed, held and delivered.
- Drift
- Gradual or sudden change in volume or layout between runs.