The importance of absolute accuracy in CNC machining
Absolute accuracy is the difference between a machine that repeats well and one that lands on the true dimension. This page explains where that error comes from, how it is measured, and which parts actually need it. Written for engineers and buyers specifying tolerances.
In this article
- 1
- 2
- 3
- 4
- 5
- 6
- 7
What absolute accuracy in CNC machining really means
Absolute accuracy is how close the finished feature sits to the drawing's true position. Repeatability is different. A machine can hit the same wrong spot ten thousand times and still be perfectly repeatable. Absolute accuracy asks whether that spot was right in the first place.
On a mill, the error shows up as a bore that is 0.012 mm off nominal, a pocket wall that leans, or a hole pattern that drifts across a 600 mm plate. The controller says the tool is at X 120.000. The part says otherwise. That gap is the error budget you are trying to close.
The gap comes from stacking. Machine geometry, ball screw pitch error, spindle thermal growth, tool deflection, workholding shift and probe uncertainty all add up. Some are systematic and repeatable, so the controller can compensate them. Others wander with temperature and cutting load, and no amount of compensation in the program will hold them.
This is why two shops can quote the same ±0.05 mm part and deliver very different results. One has mapped its machines and controls its environment. The other is relying on the repeatability spec on the brochure.
Where the error actually comes from
Squareness and straightness errors dominate on large parts. If the X axis is out of square with Y by 10 µm over 300 mm, a 600 mm bolt pattern inherits roughly 20 µm of position error before the cutter touches metal. On a 4,000 mm travel machine, small angular errors become large linear ones.
Thermal growth is the quiet one. A spindle running at 12,000 rpm warms 5–10 °C in the first hour. On a 400 mm steel or aluminium part, that is tens of microns of drift. Shops that hold tight tolerances warm up the machine before the first cut and keep the shop within a few degrees.
Tool deflection scales with length cubed. A Ø6 mm end mill hanging 60 mm out of the holder will push away from the wall under load. The same tool held in a shrink-fit holder at 35 mm projection cuts a straighter wall at the same feed. Short and rigid beats slow and hopeful.
Workholding moves too. Thin plates relax when clamps come off. Castings shift when the first cut releases residual stress. A dimension measured in the fixture is not the dimension on the bench, and the two can differ by more than the tolerance itself.
How you prove accuracy instead of claiming it
A CMM with a stated accuracy of 1.8 + L/350 µm is a measuring tool, not a verdict. Its own uncertainty eats into the tolerance you are trying to verify. When a feature is specified at ±0.005 mm, the measurement method needs roughly a 4:1 ratio to the tolerance, which pushes you toward a temperature-controlled room and a calibrated machine.
On-machine probing closes the loop differently. Touch probes on the 5-axis centers let us find the datum on the actual part, not on the fixture corner. For a second operation on a mill-turn part, that removes stack-up from the vise jaws and the previous setup.
For high-volume runs, the practical check is capability. Take 30 consecutive parts, measure the critical feature, and calculate Cpk. A process running at Cpk 1.33 or better is stable enough to trust. A process at 0.9 will pass inspection today and fail next week.
Reports go out on request with raw material certificates, in-process records and final dimensional data. If your quality team needs the numbers, ask for them at quote stage. It is cheaper than finding out after the parts ship.
When ±0.005 mm is the wrong answer
High absolute accuracy costs money in three places: machine time, inspection time and scrap risk. If a bracket locates on a clearance hole and bolts into a slotted frame, holding its hole pattern to ±0.005 mm buys nothing. The assembly tolerates ±0.2 mm, and the extra precision just moves cost into the part price.
Some features cannot be held at that level by milling at all. A deep Ø4 mm bore at 10× diameter will deflect regardless of the machine. Grinding, honing or EDM holds the size better. The right answer is often a process change, not a tighter machine.
Material matters. Aluminium 6061 and 7075 cut clean and hold size well. Inconel and Ti-6Al-4V spring back, work-harden and generate heat at the edge. Expect the achievable tolerance band to widen, and plan a finishing pass with fresh tooling.
Thin walls are the classic trap. A 0.5 mm wall on a 100 mm aluminium housing will move when the material around it is removed. Rough, stress relieve if needed, then finish in a sequence that keeps the part balanced. No controller setting fixes a part that wants to bend.
The honest engineering position: specify the tolerance the function needs, then ask what process holds it. That conversation at quote stage saves weeks later.
What GreatLight does differently to protect accuracy
We run 127 high-precision CNC machines across three wholly-owned plants, including 16 simultaneous 5-axis machining centers, 16 mill-turn centers and a Ø400 mm rotary table. Maximum processing size is 4,000 mm, with travel options of 4,000 × 400 × 150 mm for long parts and 750 × 1,150 × 550 mm for plate work.
Standard shop tolerance is ±0.005 mm (±0.0002 in) on parts that allow it. Surface finish runs from Ra 1.6–3.2 μm as machined down to Ra 0.2–0.8 μm where the drawing calls for it. Qualification rate on shipped parts is 99.99%, backed by 100% inspection before shipment.
Inspection is not a final gate only. We check raw material on arrival, monitor in process, and run a final dimensional inspection with reports on request. Four certifications sit behind the process: ISO 9001:2015, IATF 16949:2016, ISO 13485:2016 and ISO 27001:2022.
Quotation and free DFM analysis come back within 12 hours. Production can start within 24 hours, and parts ship in 3–5 days. There is no minimum order quantity, so a single prototype and a 10,000-part run go through the same setup discipline. Uploads are secure and confidential, and an NDA is available on request.
Step by step: holding tight tolerances on a real part
A workable sequence for features in the ±0.005 mm to ±0.02 mm band.
- 11. Sort features by toleranceSplit the drawing into bands: loose (±0.1 mm), moderate (±0.05 mm), tight (±0.005–0.02 mm). Only the tight band drives machine choice, fixturing and inspection. Everything else follows.
- 22. Pick the machine for the tight bandFor tight bores and patterns on one face, a 3-axis mill with a warm spindle works. For tight features on five faces, use a simultaneous 5-axis center so you avoid re-fixturing.
- 33. Warm up before the first cutRun the spindle 30–60 minutes at cutting speed. Check the shop temperature. If it swings more than 3 °C across a shift, the tight features will move with it.
- 44. Probe the datum on the partTouch off the actual workpiece, not the vise. Record the offsets. Re-probe after any clamp change or long roughing cycle.
- 55. Rough with margin, finish lightLeave 0.3–0.5 mm on tight walls. Finish with small radial depth (0.1–0.2 mm) and a short, rigid tool. This keeps deflection in the low microns.
- 66. Measure before unclampingProbe the critical feature in the fixture. If it is out, you can still correct it. After unclamping, measure again to catch relaxation.
- 77. Log the resultRecord the actual value, not just pass or fail. Trends tell you when a machine needs calibration, long before a part is rejected.
Typical error contributions and how to control them
Ranges are typical shop-floor values, not machine specifications.
| Error source | Typical magnitude | Control method |
|---|---|---|
| Axis squareness | 5–20 µm over 300 mm | Laser calibration, compensation tables |
| Spindle thermal growth | 10–40 µm in first hour | Warm-up cycle, climate control |
| Tool deflection | 10–80 µm at 4× diameter | Shorter projection, lower radial depth |
| Workholding shift | 5–50 µm after unclamping | Light finishing passes, stress relief |
| Probe and setup error | 2–10 µm | On-machine probing, datum re-check |
| Ball screw pitch error | 3–15 µm per 300 mm | Pitch error compensation |
The verdict
If the feature is functional and the assembly is tight, pay for absolute accuracy and probe the part on the machine. If the feature is clearance or cosmetic, widen the tolerance and put the money into finish and lead time. Tightening a tolerance that does not matter buys scrap, not quality.
Questions engineers ask about absolute accuracy
What is the difference between accuracy and repeatability?
Accuracy is the distance between the commanded position and the true position. Repeatability is the spread when the machine returns to the same position many times.
A machine can be highly repeatable and still inaccurate. Compensation tables fix systematic inaccuracy. Random error from heat and load is what repeatability specs do not capture.
Can a CNC machine really hold ±0.005 mm?
Yes, on the right feature, in the right material, with a warm machine and light finishing passes. We hold ±0.005 mm (±0.0002 in) as a standard shop tolerance.
It is not universal. Deep small bores, thin walls, titanium and Inconel all push the achievable band wider. Bring the drawing and we will tell you which features can hold and which need a process change.
Does a tighter tolerance always cost more?
Usually, yes. Tighter tolerances mean slower finishing passes, more probing, more inspection time and higher scrap risk. The cost curve is not linear.
Going from ±0.05 mm to ±0.02 mm is often modest. Going from ±0.01 mm to ±0.005 mm can double the inspection effort because measurement uncertainty starts to matter.
How does temperature affect the result?
Steel expands about 11 µm per metre per °C. Aluminium is roughly twice that. A 5 °C shop swing moves a 500 mm aluminium part by around 55 µm.
That is why tight work is done after a spindle warm-up and why we watch shop temperature across the shift. The part and the machine need to be at the same temperature when you measure.
When should I move to grinding or EDM instead?
When the feature geometry defeats milling. Deep small bores, sharp internal corners, hardened tool steel above 50 HRC and very tight roundness all point to a different process.
Milling gets the part close and sets the datum. Grinding or EDM finishes it. Planning both steps early avoids a redesign after the first article fails.
What inspection documentation comes with the parts?
Raw material certificates, in-process records and final dimensional inspection data are available on request. We inspect 100% before shipment.
Tell us at quote stage which features need reported values and what measurement method your quality team accepts. That way the inspection plan matches your receiving process.
Send the drawing, get a real tolerance answer
Upload your part and we will return a quotation with free DFM analysis within 12 hours, including which features can hold ±0.005 mm and which need a process change.
12-hour quoteFree DFM analysisNo minimum orderNDA on request
More machining notes
We publish setup notes, tooling trials and inspection data from the factory floor.
