CNC Calibration: Key Steps for a Machine That Holds Tolerance
A working sequence for calibrating three-axis, four-axis and five-axis machining centers, written for engineers and shop leads who have to sign off on the result. We cover what to measure, in what order, and which numbers tell you to stop and fix hardware instead of pushing compensation. After reading, you should be able to judge whether a calibration report is real or a sheet of paper.
Key takeaways
What Calibration Actually Corrects
A machining center is a stack of small errors. Ballscrew pitch error, thrust-bearing play, rail twist, spindle tilt and thermal growth all push the tool away from the programmed point. Some of those errors are repeatable, and the controller can cancel them with compensation. Others are random or load-dependent, and no parameter will fix them.
That split is the whole job. Calibration is the process of separating repeatable error from unstable error, then writing only the repeatable part back into the control. A machine that holds ±0.005 mm on a warm afternoon in August is not lucky. Its repeatable errors have been measured and cancelled, and the unstable ones have been removed at the source.
Thermal drift is the largest unstable term on most machines. A spindle running at 12,000 rpm for 40 minutes can grow 20–40 μm in Z. The ballscrew grows too, and the effect is not linear along the axis. This is why a cold-machine measurement tells you almost nothing about afternoon accuracy.
Wear is the second term. A ballscrew that has run 20,000 hours will show backlash in specific zones, not uniformly. Mapping the whole travel beats adjusting one end and hoping the rest follows.
- 1RepeatablePitch error, squareness, backlash, spindle tilt. Compensate these.
- 2UnstableThermal growth, vibration, fixture deflection, tool wear. Fix at the source.
Six Readings to Take First
Do not open the compensation table until you have numbers. Pull the way covers and check the rails for scoring, then look at the lubrication lines. A dry linear guide will read as backlash on the ballbar, and you will spend a day chasing a parameter that was never wrong.
Measure ambient temperature and note it. A 5 °C swing between the morning and afternoon shifts moves a 1,000 mm steel ballscrew by roughly 6 μm. Write that number on the sheet, because it changes how you read everything else.
Check the level and the anchor bolts. Machines on soft floors move seasonally. A 0.02 mm/m change in level shows up as a taper on a 500 mm part, and operators usually blame the tool or the fixture.
Run a quick backlash check on each axis at three positions: near the home end, mid-travel and the far end. If all three read the same, you have uniform wear. If one end reads double, stop and inspect the thrust bearing and the nut preload.
Squareness comes next. Sweep a granite square or use a laser with a straightness optic across the full travel. Typical acceptable values are 0.01–0.02 mm/m for a production mill and under 0.01 mm/m for a five-axis machine doing mold work. Anything above 0.03 mm/m is a mechanical problem, not a software one.
- 1Rail conditionScoring or dry contact makes every later number suspect.
- 2Ambient temperatureRecord it. Thermal correction depends on it.
- 3Level and anchor boltsSeasonal floor movement causes taper.
Laser, Ballbar and Where Each One Fits
A laser interferometer measures linear positioning error along one axis at a time. It gives you the pitch error curve, which the controller can then flatten with a compensation table. It is slow and it needs a clear line of sight, but nothing else gives you the same resolution on a 4,000 mm travel.
A ballbar measures circular interpolation. You command a 300 mm circle at a feed rate around 1,000 mm/min and the ballbar records how far the machine deviates from a true circle. That single test catches squareness, backlash, servo mismatch and reversal spikes at once. It is the fastest way to see whether your machine has a mechanical fault or a tuning problem.
For five-axis machines, add the rotary axes. Sweep a test sphere or run the ISO 10791-7 circle-diamond-square pattern. Rotary centerline offset and tilt error are the two numbers that ruin five-axis parts, and they do not show up in a three-axis laser run.
Use a spindle analyzer or a dial indicator on a test bar to check spindle tilt and radial runout. Runout over 5 μm at the gauge line shows up as a finish problem on every part. Tilt shows up as a step when you change tools.
- 1LaserLinear positioning and pitch error per axis.
- 2BallbarCircularity, backlash, servo mismatch, reversal spikes.
- 3Ball and DBBRotary centerline and tilt error on five-axis.
Writing Values Back Into the Control
Backlash compensation is the first value to enter, and the easiest to get wrong. Enter too little and the error stays. Enter too much and the axis overshoots on every reversal, which produces a worse surface than the original problem. Set it from the measured backlash minus 2–3 μm, then re-measure.
Pitch error compensation comes next. The laser writes a table of position error against axis position, and you load that table into the control. Most builders support 100 to 1,000 points per axis. Use the density that matches your travel: a 4,000 mm axis needs more points than a 500 mm one.
Check the servo tuning before you finish. A machine with a loose position loop will show a reversal spike on the ballbar that no compensation table can remove. Adjust the gain, then re-run the ballbar. If the spike is still there, look at the coupling and the thrust bearing.
Firmware matters more than people expect. Older motion control firmware can carry bugs in the look-ahead or the compensation interpolation. Update before you calibrate, not after, or you will be measuring a machine that is about to change behavior.
- 1Backlash firstMeasured value minus 2–3 μm, then re-check.
- 2Pitch table second100–1,000 points per axis depending on travel.
- 3Firmware lastUpdate first, then measure, then compensate.
The Step by Step Sequence
Run in this order. Skipping a step usually costs more time than it saves.
- 1Clean and inspectPull the covers, wipe the rails, check the lube lines and the way wipers. Log any scoring. A dirty machine gives you dirty numbers.
- 2Thermal soakRun the spindle at your normal production speed for 30–60 minutes. Measure ambient and spindle-housing temperature. Do not compensate a cold machine for warm-shift work.
- 3Level and anchorCheck level across the bed with a precision level at 0.02 mm/m. Torque the anchor bolts. Re-check after 24 hours on a new installation.
- 4Backlash per axisMeasure at three positions per axis with a dial indicator or the laser. Typical good values are under 5 μm. Above 15 μm, inspect the nut and thrust bearing.
- 5Linear positioningRun the laser along full travel. Record pitch error at 25–50 mm intervals. Note the maximum deviation, not just the average.
- 6Squareness and circularitySweep the square or run a 300 mm ballbar circle at 1,000 mm/min. Record circularity, squareness and reversal spikes separately.
- 7Enter compensationLoad backlash and pitch tables. Start conservative. Re-measure each axis after entry, because the control and the laser rarely agree on the first pass.
- 8Test cut and CMMCut a certified test piece such as an ISO 10791 circle or a NAS 979 cone. Measure it on a CMM. Compare to the laser result, not to the drawing alone.
Calibration Checks and What They Catch
Values are typical ranges for production machining centers, not guarantees.
| Check | Tool | Typical limit | Catches |
|---|---|---|---|
| Backlash | Dial indicator or laser | < 5 μm | Nut wear, thrust bearing play |
| Linear positioning | Laser interferometer | < 10 μm over 1,000 mm | Ballscrew pitch error, thermal drift |
| Squareness | Granite square or laser | 0.01–0.02 mm/m | Rail twist, leveling, foundation |
| Circularity | Ballbar, 300 mm circle | < 10 μm | Servo mismatch, reversal spikes |
| Spindle runout | Test bar and indicator | < 5 μm at gauge line | Bearing wear, taper damage |
| Rotary centerline | Ball or DBB test | < 10 μm | Five-axis trunnion offset |
| Thermal drift | Spindle soak + probe | < 15 μm after 1 hour | Cooling, spindle growth |
Fix the machine, then compensate it
Compensation is for repeatable error. If squareness is off or the ballbar shows a reversal spike, fix the hardware first. Send us your drawing and we will quote the machining side.
Questions Engineers Ask
How often should a machining center be calibrated?
A production machine doing tight work gets a ballbar check every 3–6 months and a full laser calibration once a year. A machine that just moved, or one that took a crash, gets a full check immediately.
If your parts pass but the scrap rate creeps up, calibrate before you blame the tooling. Drift of 10 μm rarely shows as a hard reject, but it does show in the rework column.
Can I calibrate a five-axis machine the same way as a three-axis?
No. The three-axis checks cover the linear axes, and they will pass even when the rotary axes are out. You need a ball or DBB test and a rotary centerline measurement on top.
On a trunnion machine, the two numbers that matter most are the C-axis centerline offset and the A-axis tilt error. Both change after a crash and neither appears in a laser linear run.
Does compensation hide a worn machine?
It can, and that is the trap. Compensation is valid for repeatable error. A ballscrew with a worn zone in the middle of travel produces error that changes with load, and a fixed table cannot cancel it.
When the ballbar shows a reversal spike that survives tuning, stop compensating. Replace the nut or the bearing. Compensation after that point just moves the failure to a different part of the travel.
What temperature should the shop be during calibration?
Measure at the temperature your machine actually runs at, and record it. Most shops sit between 18 °C and 24 °C. A 5 °C shift over the day moves a 1,000 mm steel screw by about 6 μm.
If your shop has no climate control, calibrate in the morning and note the drift by afternoon. Then set your process window from the worst case, not the best.
How do I know the calibration worked?
Two ways, and you want both. The laser and ballbar numbers should land inside the limits in the table above, and a certified test piece measured on a CMM should hold your drawing tolerance.
If the laser passes and the test cut fails, the problem is usually the fixture, the tool or the thermal state of the part, not the machine geometry.
What does a calibration report need to contain?
Date, ambient temperature, machine serial number, tool used, measured values before and after, and the name of the technician. Values without the conditions are not usable later.
Keep the previous report with it. Trend matters more than any single reading. A machine that drifts 3 μm per quarter tells you something is loosening.
Parts Machined on a Calibrated Machine
Tolerances to ±0.005 mm, 100% inspection before shipment, and reports on request. Quotation and free DFM analysis within 12 hours.
12-hour quote±0.005 mm100% inspectionNo minimum order
More From the Shop Floor
We publish setup notes, tooling trials and inspection data from the factory floor.
