Skip to content

Engineering

Replacing an AS/400 (IBM i) while the plant keeps running

The production schedule is a bigger obstacle than the code. Cut over one process at a time and budget for four failures.

Computing America. Applications, integrations, data analysis and AI/ML engineering.

Describe the cutover
6 min readComputing America
An old midrange server and green-screen terminal in a plant office, the production line running beyond the window.

In short

  • Cut over per process, not per system, and settle how each cutover rolls back before its date. The journal feed is one-way, so a rollback needs a write-back path.
  • A trained operator finishes a green-screen transaction without looking at the display. Count keystrokes per transaction before designing the new interface, so the replacement doesn't add seconds to every one.
  • The nightly batch chain is an undocumented dependency graph, and the dangerous jobs fail quietly. Log a full month of its starts, counts and messages from the history log, job logs and journals before replacing any of it.
  • For files that are journaled and whose receivers you keep, a journal reader gives you a change feed without modifying a line of RPG. Check both before relying on it.
  • The person who can still read the RPG is a project role. If they are only available for questions, the archaeology phase is missing from the estimate.

The only real cutover window may be a holiday shutdown. A failed cutover on a shop floor means a line that isn't making product, so the schedule sets the plan before the code does. The RPG still works and the business rules are all in it, though perhaps only one person can read it.

What's in there

Inventory what the system does, not what it stores. Look for five parts:

  • Interactive programs behind the green screens, holding business rules written down nowhere else.
  • A batch chain whose dependencies live in the order a CL program submits jobs and in one person's memory.
  • Physical files and the logical files over them, which encode assumptions about keys and uniqueness that a modern schema won't reproduce by accident.
  • Device integrations: label printers, scales, scanners, a CNC handoff, a customer EDI feed. They surface late because they aren't in any repository.
  • Spool-file reports that end up as a customer document, an auditor's evidence or a regulatory submission, in a layout someone downstream depends on.

Four things that break

1. The new screen is slower for the person using it

An operator who has run a 5250 (green-screen) session for a decade finishes a transaction without looking at the display, in gloves. A replacement that needs a mouse, a page transition and a confirmation dialog adds seconds to every transaction, and shift volume multiplies them.

Count keystrokes at the terminal for the ten highest-volume transactions and make that count a design requirement: keyboard-first entry, function-key equivalents, no dialog between the operator and the next scan.

2. The batch chain is a dependency graph that was never drawn

Job order matters, and some jobs are safe to rerun and some aren't. The dangerous jobs fail quietly: a job processes zero records because an earlier step didn't deliver a file, and the gap shows up later as an inventory discrepancy.

For a full month before replacing any of it, log every job's start and end times, record counts, and completion and escape messages. Take starts and ends from the history log (QHST, shown with DSPLOG), which carries job-start and job-end messages, and messages from each job's job log. For record counts, first find the files a job writes: run DSPPGMREF (opens in a new tab) on the job's programs to see how each file is used (input, output, update). Where a file is journaled and its receivers are kept, count its DISPLAY_JOURNAL entries (below) per JOB_NAME and JOB_NUMBER, split by JOURNAL_ENTRY_TYPE, so adds, updates and deletes stay apart. For a file that is not journaled, COUNT(*) before and after a job shows only adds minus deletes. A job that updated every row then looks like one that touched none.

Then confirm the history log still holds the month. It is kept as separate log versions, and IBM says they “accumulate on a system” and old ones are deleted periodically (opens in a new tab). List them with DSPOBJD OBJ(QSYS/QHST*) OBJTYPE(*FILE), check that the oldest predates your month, and save the versions you need. The QHSTLOGSIZ system value (opens in a new tab) sets how many records a version holds. That log is the specification, and the baseline you compare the new system against.

3. The data types carry meaning your new schema drops

Look for packed decimals, six-digit dates, and zero and blank used as different sentinel values. A customer number may be zero-padded in one file and not in another.

Treat migration as reconciliation: move the data, run the same queries against both systems, and compare totals per period, per plant and per part.

4. The edges are physical

A label that prints two millimeters off is a customer receiving failure. A scale that returns weight in a fixed-width field with an implied decimal is silently off by a factor of ten. These integrations surface late and can't be tested from a laptop. Budget bench time with the real hardware on the real floor.

Read before you write, and check the journal first

A plant won't accept a big-bang replacement, so take one process at a time and route it to the new system while the old one keeps running: the strangler fig (opens in a new tab) approach. Choose the seam carefully, and start with reads: a reporting surface built on the existing data changes nothing operationally, delivers value in weeks, and teaches the team the data model.

Side by side, the new system must see changes as they happen. Editing the existing programs to publish events is the obvious route and puts production at risk. Journaling is the alternative: a journal records changes to the objects you specify, and a reader over it can turn them into a change feed. Don't assume the journal is there. Check three things first:

  1. 1.Which files are journaled. Journaling is started per physical file, with STRJRNPF (opens in a new tab). Starting it on a file is a change to production, so schedule it.
  2. 2.How receivers are retained. With system-managed receivers and automatic deletion (opens in a new tab), the system deletes detached receivers. Make sure the reader has consumed a receiver before it is deleted, or keep receivers until it has.
  3. 3.Whether the programs use commitment control. A commit writes a C CM entry to the journal, so only changes made under commitment control come with transaction boundaries. For the rest, the feed carries record changes, not committed transactions.

The reader can use QSYS2.DISPLAY_JOURNAL (opens in a new tab), an SQL table function that returns information similar to the DSPJRN command and the QjoRetrieveJournalEntries API. You can call that API directly instead, or use the RCVJRNE command, which passes journal entries to a user exit program continuously. A reader adds load to the IBM i, so measure it off-peak before the plant depends on it.

Fig. 1
  1. Group: Existing IBM i

    RPG programs

    Screens and batch jobs, unchanged

    • To Files · writes
  2. Group: Existing IBM i

    Files

    Journaling is started per physical file

    • To Journal · journaled files only
    • To Comparison · the old files
  3. Group: Existing IBM i

    Journal

    Only the files that are journaled

    • To Journal reader
  4. Journal reader

    Reads journal entries; adds load

    • To New system · feed
  5. New system

    Fed continuously, checked on real traffic

    • To Comparison
  6. Comparison

    Same queries on both, totals per period, plant and part

The new system is fed only from the journal, so a change the journal does not hold never reaches it, and only the comparison against the old files can show the gap.

Writes come later, one process at a time, behind an anti-corruption layer (opens in a new tab) that translates between the old vocabulary and the new so the new system doesn't inherit thirty years of encoding decisions. Give that layer an owner and a test suite, because it may stay for years.

Cutover is a schedule problem

Settle three rules with the plant manager before the first phase is designed, not during the week it happens.

  1. 1.Cut over per process, not per system. Receiving can move while production reporting stays. One date on which everything changes is a date on which everything can fail.
  2. 2.Decide how each cutover rolls back before its date, and keep that path open until the next business day. While the new system only reads, rollback means pointing users back at the old screens. Once writes move, the journal feed is one-way, so run dual writes for that process or build a reverse feed that carries the new system's writes back into the IBM i files. If you can afford neither, the answer to “can we roll this back by first shift tomorrow?” is no, whatever the tests say. Treat the cutover as one-way, and do not call it ready until the plant manager accepts that in writing.
  3. 3.Cut over into a low-volume window with the old system still warm, and keep it warm until a full month-end close has run on the new system.

Don't fix the process while you replace the system, however bad the process looks. When behavior and technology change together, no one can tell whether a discrepancy is a defect or the new intent.

Start the batch log today, and save the history-log versions you still have, because the history log keeps only so much of the month.

Sources

  1. 1.Strangler Fig (opens in a new tab), martinfowler.com,
  2. 2.Anti-Corruption Layer pattern (opens in a new tab), Microsoft Azure Architecture Center

Next step

Tell us what cannot stop.

Which parts of the floor cannot pause for a weekend, and what the old system still owns outright. We will say where the first seam goes and what has to run in parallel while it moves.

Reply
A person replies, not a sequence: within one business day, from someone who would be on the engagement.

Packages

Start with a package

Four packages, each with its price on the page. Pick one and buy it online.

Billing