How to Calibrate a CNC Milling Machine
This guide is for maintenance techs and process engineers who need to bring a vertical or 5-axis mill back inside tolerance. After reading it you can run the geometric checks, read the numbers, and decide what to adjust yourself and what to hand to a metrology service.
In this article
- 1
- 2
- 3
- 4
- 5
- 6
- 7
Key takeaways
What changes when you calibrate a CNC milling machine
A mill that has drifted out of square still cuts metal. It just cuts the wrong shape slowly and in a way that is hard to see until the CMM report lands. The first sign is usually a bore that measures oval, or a face that needs shimming during assembly. By then you have scrapped a batch.
Calibration is not a single event. It is a set of measurements that tell you where the error lives: in the geometry of the machine, in the control compensation, or in the thermal state of the structure. Each one has a different fix and a different cost.
For shops holding ±0.005 mm, the useful question is not whether the machine is calibrated, but how long the calibration stays valid under load. A mill that holds tolerance at 8:00 AM and drifts by noon will pass a spot check and fail a production run.
The checks below are the ones we run at GreatLight before a machine goes back into a job that needs tight tolerance. They work on a 3-axis vertical mill and on a 5-axis trunnion machine. The order matters, because each step depends on the one before it.
- 1Geometry errorsSquareness, straightness and parallelism of the axes.
- 2Compensation errorsBacklash, pitch error and tool-length offsets.
- 3Thermal errorsGrowth of the spindle and ballscrew during the shift.
Tools and conditions you need to calibrate a CNC milling machine
You cannot calibrate a machine that is not clean, not level, and not at a stable temperature. Start with the machine sitting on its pads, the foundation bolts tight, and the way covers clear of chips. A single chip under a linear guide can read as 0.015 mm of straightness error.
The tool list is short but not cheap. A granite square or a precision cylinder square, a dial test indicator with 0.001 mm graduation, a magnetic base that actually holds, a ballbar kit, and a spindle test bar. For the spindle checks you also want a 300 mm test bar and a surface plate if you plan to check taper runout off the machine.
Temperature matters more than most people expect. Cast iron and steel grow about 11 to 12 μm per meter per °C. A 5 °C swing across the day moves a 1 m axis by roughly 0.055 mm. That is ten times the tolerance you are trying to hold, which is why thermal checks come last and get repeated.
Write down the serial number, the date, the ambient temperature and the spindle hours before you take the first reading. Baseline data is what makes the next calibration faster.
- 1Clean firstWipe guides, taper and table. Remove chips from the T-slots.
- 2Let it soakMachine off for at least 8 hours before cold geometry checks.
- 3Same setup each timeSame indicator, same bar, same clamping. Consistency beats absolute accuracy.
How to read squareness, backlash and runout results
Squareness is reported as a deviation over a stated length, for example 0.008 mm over 300 mm. Do not convert it to an angle and compare it to a spec sheet unless the spec sheet uses the same base length. A machine rated at 0.010 mm over 300 mm is not the same as one rated at 0.010 mm over 500 mm.
Backlash shows up as a jump when the axis reverses direction. Measure it with the indicator on the table and the axis jogged in 0.010 mm increments. If backlash is under 0.005 mm, leave the compensation alone. Over-compensating creates a bump on every reversal that is worse than the original error.
Spindle runout is split into two numbers: taper runout at the gauge line, and runout 300 mm from the spindle nose. The first tells you about the taper; the second tells you about the whole spindle and housing. A taper that reads 0.002 mm but a 300 mm bar that reads 0.030 mm means the housing or the bearings are the problem, not the taper.
Ballbar data is the fastest way to see the whole picture. A circularity plot in the XY plane will show squareness as an oval, backlash as a step at each reversal, and servo mismatch as a pair of lobes. One 300 mm circle test at 1,000 mm/min takes about two minutes and tells you where to look next.
- 1SquarenessReport over a fixed length. 0.010 mm over 300 mm is a common working limit.
- 2BacklashUnder 0.005 mm, leave it. Adjust only if it grows.
- 3BallbarUse a 300 mm radius and 1,000 mm/min feed for a baseline plot.
How to keep a CNC milling machine calibrated between services
Calibration is only useful if it survives the next production run. Three habits do most of the work. Keep the machine level and the foundation clean. Warm up the spindle before the first tight-tolerance cut. And keep a log of the checks so you can see a trend before it becomes a scrap event.
Compensation values are not permanent. Backlash compensation hides wear until the wear becomes non-linear, and then the compensation makes parts worse. Review the values every calibration. If backlash grows by more than 0.003 mm between checks, plan for a ballscrew or thrust bearing inspection.
Thermal compensation is worth setting up on a machine that runs long shifts and holds ±0.005 mm. It uses spindle and structure sensors to shift the axis offsets as the machine warms. It does not replace a warm-up cycle. It reduces the size of the correction the operator has to make.
If your shop sends parts out to a machining partner, ask for the calibration records. A supplier who can show squareness, backlash and ballbar data over time is a different risk than one who only sends a certificate. We keep these records per machine and can share them with a project.
- 1Warm-up cycle30 minutes at 8,000 rpm plus 30 at 12,000 rpm before tight work.
- 2Compensation reviewRe-read every value at each calibration. Do not carry old numbers forward.
- 3Trend logPlot the numbers. A slope is more useful than a single reading.
Steps to calibrate a CNC milling machine
Run these in order. Do not skip ahead.
- 1Level and clean the machineWipe the ways and table. Check the level across the table in X and Y with a 0.02 mm/m precision level. Adjust the pads until both axes read within 0.02 mm/m. A machine that is off level will twist as it warms.
- 2Check X-to-Y squarenessClamp a granite square to the table, align it to the X axis with a dial indicator, then sweep the Y axis. Record the deviation over 300 mm. Target is under 0.010 mm. If it is worse, check the table and saddle first before touching the control.
- 3Check Z-to-X and Z-to-Y squarenessMount a dial indicator on the spindle and sweep a precision square standing on the table. Read the deviation over 300 mm of Z travel. Target under 0.010 mm per axis. On a 5-axis machine, also check the trunnion axis to the table.
- 4Measure backlash on every axisSet the indicator on the table, jog the axis in 0.010 mm steps in one direction, then reverse. Record the jump. Repeat five times and take the average. Typical good values are 0.002–0.005 mm. Only enter compensation if the average is above 0.005 mm.
- 5Run a ballbar testFit the ballbar for a 300 mm circle in XY, feed at 1,000 mm/min, and run both directions. Save the plot. Look for ovality, reversal steps and servo mismatch. Repeat in XZ and YZ if the machine does 5-axis work.
- 6Check spindle taper and 300 mm runoutClean the taper with a lint-free wipe and light oil. Insert a test bar. Measure taper runout at the gauge line and at 300 mm. Typical limits are 0.002 mm at the gauge line and 0.010 mm at 300 mm. Anything larger points to bearings or a damaged taper.
- 7Warm up and repeat the critical checksRun the spindle at 8,000 rpm for 30 minutes, then at 12,000 rpm for another 30. Re-check squareness and ballbar. Record the shift. A shift under 0.005 mm is normal. A shift over 0.015 mm means the machine needs thermal compensation or a service visit.
- 8Record and set the next calibration dateLog every number, the ambient temperature, the spindle hours and the tooling used. Set the next check based on how much the values moved. A stable machine can go 6 months. A machine that moved 0.008 mm in three months should be checked quarterly.
Which check catches which fault
Use this to pick the right test when a part fails inspection.
| Symptom on the part | Likely fault | Check to run | Typical limit |
|---|---|---|---|
| Oval bore in XY | Squareness or servo mismatch | Ballbar, XY plane | Under 0.010 mm over 300 mm |
| Step on a face after reversal | Backlash | Dial indicator reversal test | Under 0.005 mm |
| Taper changes with spindle speed | Thermal growth | Warm-up ballbar | Under 0.005 mm shift |
| Bore size drifts over the shift | Ballscrew or thermal | Laser or ballbar, repeated | Under 0.015 mm drift |
| Poor surface finish on one side | Spindle runout | Test bar at 300 mm | Under 0.010 mm |
| Hole position varies batch to batch | Pitch error or scale | Laser interferometer | Under 0.005 mm per 300 mm |
| Trunnion parts tilt | Rotary axis squareness | Ballbar, XZ and YZ | Under 0.015 mm over 300 mm |
Calibrate the machine before you blame the program
Most out-of-tolerance parts come from the machine, not the CAM file. Run the five checks, log the numbers, and fix the geometry before touching offsets.
Frequently asked questions
How often should I calibrate a CNC milling machine?
For a mill holding ±0.005 mm, check squareness and backlash every 3 to 6 months, and run a ballbar test at the same time. Machines running two or three shifts should be checked quarterly.
After any crash, spindle change, or move to a new foundation, calibrate before the next tight-tolerance job.
Can I calibrate a CNC mill myself, or do I need a service company?
You can handle leveling, squareness with a granite square, backlash measurement and taper runout with a test bar. These cover most day-to-day drift.
Laser interferometer work, pitch error mapping and rotary axis calibration on a 5-axis machine need specialist equipment and training. Budget for a metrology service for those.
What tolerance should a calibrated CNC milling machine hold?
A well-maintained vertical mill in good condition typically holds ±0.005 mm on a 300 mm part, with squareness under 0.010 mm over 300 mm.
The machine's own geometry sets the floor. A machine with 0.020 mm of squareness error will not hold ±0.005 mm on a long part no matter how the control is set.
Does backlash compensation fix a worn ballscrew?
No. Compensation adds a fixed offset at each reversal. A worn ballscrew has varying backlash along its length, so a single value is wrong at both ends.
Use compensation only for small, stable values under about 0.010 mm. Above that, inspect the ballscrew, nut and thrust bearings.
Why does my machine cut differently in the afternoon?
Thermal growth. A cast iron structure moves about 11 to 12 μm per meter per °C. A 5 °C rise in shop temperature shifts a 1 m axis by roughly 0.055 mm.
Run a warm-up cycle before tight work and record the ballbar shift between cold and warm states. If the shift is over 0.015 mm, look at thermal compensation.
What records should a machine shop keep for calibration?
Date, ambient temperature, spindle hours, and the measured values for squareness, backlash, ballbar circularity and spindle runout. Note the tooling and the setup used.
Trend data across several calibrations is more useful than a single certificate. It shows whether the machine is stable or wearing.
Send us a print and we will quote it
Upload your CAD file and get a quotation with a free DFM analysis within 12 hours. Tight-tolerance parts run on calibrated machines with 100% inspection before shipment.
12-hour quote±0.005 mm100% inspectionNDA on request
More machining notes
We publish setup notes, tooling trials and inspection data from the factory floor.
