Data ServicesProcessing & related hosting

Resources

Short guides from the bench.

Practical notes for anyone preparing data for processing, whether or not you work with us.

Free to read. No sign-up.

How to write a processing specification

A specification is the difference between a job that runs reliably and one that depends on whoever is around. Start with the output: describe the file you need, column by column, and who uses it. Then work backwards. For each output column, name the source column and write the transformation in a sentence. "Order date, from the source column Date, converted to day/month/year." Where you cannot write the sentence, you have found a decision that needs making.

Next, list the checks. What makes a row invalid? A missing reference, a negative quantity, a date in the future? For each, decide whether the row is corrected by rule, held for review, or rejected. Finally, write the job-level details: schedule, retention, recipients, and what volume change should pause a run.

Preparing an export that processes cleanly

  • Keep one header row with unique, non-blank column names.
  • Use one date format throughout, and say which.
  • Avoid merged cells, subtotals and notes inside the data area.
  • Export as UTF-8 if the option exists, so accented names survive.
  • Keep the file name pattern consistent between runs.
  • Tell us if the source system changes version; layouts often shift quietly.

Deciding how long processed files should be kept

Keep files long enough to be useful and no longer. A reasonable approach is to ask three questions. How far back do you compare runs? How long after a run could someone reasonably query a figure? Is there an obligation, set by your own advisers, to keep a record for a fixed period? The longest genuine answer sets the archive period; working copies should not outlive the run. We will implement what you decide, but deciding retention obligations is for you and your advisers, not for us.

A plain glossary

Batch
A set of records processed together in one run.
Validation
Checking that a value meets a rule, such as being a date.
Standardisation
Rewriting valid values into one consistent form.
Held row
A record set aside for human review because it failed a check.
Reconciliation
Comparing two sources to find agreement and differences.
Mapping table
A record of how each source field becomes a target field.
Run note
A short report of what one run received, changed, held and delivered.
Drift
Gradual or sudden change in volume or layout between runs.