Precision of Machining: What Actually Sets Your Tolerance
A working explanation of the precision of machining for design engineers and buyers: what accuracy and repeatability really measure, how a shop holds ±0.005 mm, and where that number stops being realistic. Read it before you put a tolerance on a drawing.
In this article
- 1
- 2
- 3
- 4
- 5
- 6
Accuracy, Repeatability, and Why They Are Not the Same
Machining accuracy is the gap between the size, shape and position you asked for and what the part actually measures. Repeatability is something else: how tightly the machine reproduces the same result when it runs the same feature again. A lathe can be very repeatable and still be inaccurate if the tool offset is wrong. It will cut fifty identical bores, all 0.04 mm oversize.
Hold on, resolution matters too. Resolution is the smallest increment the control and feedback system can see, and it is usually far finer than what the machine can hold under cutting load. A ball screw might resolve 0.001 mm while the finished bore moves 0.01 mm once the tool heats up.
When engineers talk about the precision of machining, they usually mean the combined result of all three. That is why two shops can quote the same ±0.005 mm and deliver very different batches. One controls the process; the other only controls the drawing.
Rigidity, thermal growth and tool wear set the real floor. A light finishing pass on a rigid 5-axis center behaves differently from a deep roughing cut on a thin wall. Same machine, same program, different result. The number on the drawing does not travel with the part; the process does.
What the Machine Contributes
A CNC machine has three error sources that matter at the micron level: geometric error, thermal error and dynamic error. Geometric error comes from axis straightness, squareness and spindle runout. Thermal error comes from spindle bearings, ballscrews and the workpiece itself warming up during a run. Dynamic error shows up as chatter and tool deflection when the cut is aggressive.
Geometric error is the easiest to measure and the hardest to fix. It is baked into the machine at build time and drifts as the machine ages. A shop that maps its machines and compensates in the control will hold tighter numbers than a shop that simply trusts the nameplate.
Thermal error is the reason the first part of a morning shift and the two-hundredth part after lunch do not match. Spindle growth of 0.01 to 0.02 mm over a few hours is normal. On tight features, we let the machine warm up, then cut.
Dynamic error is a cutting-parameter problem, not a machine problem. Reduce radial engagement, increase spindle speed and the chatter stops. Push the feed and the same tool leaves a rough wall that no amount of inspection will fix.
This is also where the precision of machining separates from the precision of measurement. A CMM in a temperature-controlled room can resolve 0.001 mm. The machine that cut the part cannot always hold that, and the two numbers should not be confused.
How the Precision of Machining Is Held in Production
Tolerance is held by controlling the process, not by inspecting at the end. Inspection only tells you what already happened. The shop has to know which variables move the feature and correct them before the next part is cut.
Tool wear is the slowest and most predictable variable. A carbide end mill wears gradually, then reaches a sharp-wear stage and starts cutting oversize fast. We change or offset the tool before that knee in the curve, not after.
Fixturing is the variable most often ignored on drawings. A part clamped on a soft jaw can spring 0.02 mm when released. A thin rib clamped too hard will measure right in the fixture and wrong on the bench.
Temperature is the third. Aluminium expands roughly 23 μm per metre per degree Celsius. A 500 mm part that warms 5 °C during a long run grows about 0.06 mm. On a ±0.005 mm callout, that is the whole budget.
We inspect 100% of parts before shipment, with raw material checks, in-process monitoring and final reports on request. Reports are useful, but the process is what keeps the numbers.
What Tightening a Tolerance Really Costs
Cost does not rise in a straight line as tolerance shrinks. From ±0.1 mm to ±0.02 mm, most of the extra cost is inspection and slower cutting. From ±0.02 mm to ±0.005 mm, the setup changes: more rigid fixturing, warm-up cycles, finer tools, sometimes a second operation.
Below ±0.005 mm, the discussion shifts to the measurement itself. Gauge repeatability, temperature and operator technique start to eat the budget. Holding a number you cannot reliably measure is not a quality gain.
The precision of machining is therefore a system property. Machine, tool, fixture, material, program and inspector all contribute. Improving one while ignoring the others rarely moves the final number.
Materials matter here too. Aluminium 6061 and 7075 cut cleanly and hold tight tolerances well. Stainless 316 and 17-4PH work-harden and deflect more. Titanium Ti-6Al-4V and Inconel move under heat and need slower passes. Plastics like POM and PEEK spring back after the cutter passes, so the finished bore is smaller than the tool.
For most projects, the useful question is not how tight we can go. It is which features need to be tight, and what happens to the assembly if they are not.
Step by Step: Specifying a Tolerance That Holds
- 1Start from functionList only the features that touch another part or carry load. Everything else gets a general tolerance.
- 2Assign the tightest callout to the fewest featuresOne or two critical dimensions per part is normal. Ten tight callouts multiply cost fast.
- 3Check the datum schemePosition tolerances only work if the datums match how the part is set up and measured. Mismatched datums add error.
- 4Match finish to functionRa 0.8–1.6 μm covers most sealing and sliding faces. Ra 0.2–0.8 μm is for bearing seats and optical contact.
- 5Confirm the material and heat treatHardened 17-4PH or 4140 moves after heat treat. Finish machine after, or add a stress relief step.
- 6Ask for a capability checkBefore a 10,000-part run, machine a small lot and review the measured spread, not just the mean.
- 7Set the inspection planAgree which features get measured, on what equipment, and how often during the run.
Which Tolerance Can the Process Actually Hold?
Ranges below are shop-floor capability on rigid setups, not a promise on every geometry.
| Feature or process | Typical achievable | Where it stops working | Best fit |
|---|---|---|---|
| 3-axis milling, flat face | ±0.01 mm | Deep pockets, long tools | Brackets, plates |
| 5-axis milling, contoured | ±0.005 mm | Very thin walls under 1 mm | Aerospace, medical |
| CNC turning, OD and ID | ±0.005 mm | Long slender shafts | Shafts, bushings |
| Bore and hole position | ±0.01 mm | Stacked datum chains | Housings, manifolds |
| Surface finish, fine | Ra 0.2–0.8 μm | Soft aluminium, gummy cuts | Seals, bearing seats |
| Surface finish, standard | Ra 1.6–3.2 μm | Cosmetic faces only | General parts |
| Part size, large | 4,000 mm travel | Tolerance grows with length | Frames, rails |
Where the Line Sits
If the feature touches another part under load, pay for ±0.005 mm and the inspection that proves it. If it is a cover, a bracket or a clearance face, a general tolerance of ±0.1 mm will do the same job for far less money.
Common Questions
Is ±0.005 mm the same as ±0.0002 in?
Close enough for quoting. ±0.005 mm equals roughly ±0.0002 in. The difference appears when you convert a callout in inches back to millimetres, so pick one unit system for the whole drawing and stay in it.
Does a tighter tolerance always need a finer surface finish?
No. They are separate requirements. A bore can be held to ±0.005 mm with an Ra 1.6 μm wall, and a cosmetic face can be polished to Ra 0.2 μm with a loose tolerance. Specify each one where it is needed.
How does part size affect achievable tolerance?
Error accumulates with length. Thermal expansion, machine geometry and tool deflection all scale with the distance travelled. A 4,000 mm frame will not hold the same number as a 50 mm bracket, even on the same machine.
Can you hold tight tolerance on a prototype?
Yes, and it is often easier. One part can be inspected fully and adjusted by hand. The harder case is holding the same number across a 10,000-part run, which depends on tool wear control and in-process checks.
What do you need to quote a tight-tolerance part?
A 3D model, a 2D drawing with datums and tolerances, material, finish and quantity. A DFM review within 12 hours flags features that are hard to hold before cutting starts. No minimum order quantity applies, from one prototype to 10,000+ parts.
Do I need an NDA before sending drawings?
Uploads are secure and confidential, and an NDA is available on request. Many customers send a signed agreement before the first file transfer, especially for medical and aerospace work.
Send the Drawing, Get a Straight Answer
Upload your model and tolerances. We review the callouts, flag the ones that will cost more than they are worth, and return a quotation with free DFM analysis within 12 hours.
12-hour quote100% inspectionNo minimum order quantity
More CNC Knowledge
We publish setup notes, tooling trials and inspection data from the factory floor.
