Common Methods to Obtain Dimensional Precision in Machining

• 1
  Key takeaways
• 2
  What dimensional precision in machining actually measures
• 3
  Trial cut method: cut a little, measure, correct
• 4
  Gauge setting: fix the size before the first part
• 5
  In-process probing and closed-loop correction
• 6
  Fixture and datum control: the part cannot move
• 7
  Thermal control: the error you cannot see
• 8
  Step by step: picking the right method for a job
• 9
  Which method fits which job

Machining accuracy is how closely the finished part matches the drawing. It covers three things: the size you measure with a micrometer, the form of a feature (roundness, flatness, straightness), and the position of that feature relative to a datum. A bore can be a perfect 20.000 mm and still be scrap because its center sits 0.05 mm off the datum.

Error is unavoidable. Every part leaves the machine with some deviation from nominal. The job is to keep that deviation inside the tolerance band, and to make it repeatable so the operator can see it drifting before parts go out of spec.

For a typical machined part, the error budget splits roughly like this: thermal growth, tool wear, machine geometry, fixture deflection, and measurement uncertainty. On a tight part, thermal and wear usually dominate. Fixing those two moves the whole process more than chasing spindle runout.

This page covers the five methods we use to control that budget: trial cut, gauge setting, in-process probing, fixture-based location, and closed-loop temperature control. Each works. None works everywhere.

The trial cut method means you machine a small amount of material, measure the result, then adjust the tool offset before cutting the rest. You repeat the loop until the size lands in the middle of the tolerance band. The loop is: cut, measure, calculate, adjust, cut again.

It is slow but very accurate. Because you are correcting against a real measurement on the real part, machine geometry errors and most thermal drift get absorbed. We use it for one-off parts, thin-wall features, and any dimension tighter than ±0.01 mm where the setup cost is already paid.

The catch is that it depends on the operator and the measuring instrument. A 0–25 mm micrometer reading to 0.001 mm gives you a much better correction than a vernier caliper reading to 0.02 mm. If the measurement is wrong, the correction is wrong, and you may have to scrap the feature. Calibrate the gauge before the first cut, and measure at the same temperature as the machine.

A common mistake is cutting too much on the first pass. Leave 0.2–0.5 mm radial stock on a finishing pass, take 0.1 mm, measure, then dial in the rest. On a 20 mm bore with a ±0.01 mm tolerance, the last cut should be 0.05 mm or less for the correction to be predictable. Deep cuts load the tool and push the size around.

Gauge setting, sometimes called the adjustment method, uses a master gauge, a pre-set stop, or a standard sample to establish the exact tool-to-work relationship before the run starts. Once the machine is set, every part comes off the same size without a trial cut. The size is held by the setup, not by the operator's eyes.

This is the standard approach for production runs. On our 127 high-precision CNC machines, a typical batch job has the tool offsets locked against a master gauge, and the operator checks the first part plus periodic samples. If the machine drifts, the operator stops and re-sets rather than adjusting every part.

The method is fast and consistent, but it is only as good as the machine and the gauge. Machine repeatability, tool wear rate, and thermal stability all feed directly into the part. On a long run, tool wear can move the size 0.02–0.05 mm before the next scheduled offset change.

It also cannot correct for a part that is already wrong. If the casting has 0.3 mm of stock variation, gauge setting does not help; you need a roughing pass that removes the variation before the finishing pass sees it. For parts with tight tolerances and stable stock, it is the most economical method on the list.

A spindle-mounted touch probe measures the part while it is still on the machine. The control reads the result, updates the work offset or tool offset, and continues cutting. No operator intervention, no unclamping, no re-fixturing.

This is the method for long runs and complex parts. A probe can check a datum, a bore position, or a pocket depth every 10–20 parts and correct the offset before the size walks out of tolerance. On a 4,000 mm part, checking a reference feature between operations catches fixture shift that a manual check would miss.

Probing has limits. The probe itself has a repeatability budget, typically 1–2 μm on a good setup, and the stylus must reach the feature without interference. It cannot measure a surface the probe cannot touch. It also adds cycle time: each probing cycle is 10–40 seconds, which matters on a high-volume job.

Use it when the tolerance is tighter than the machine's natural drift over the run, or when re-fixturing a part is expensive. On a part with a ±0.02 mm tolerance and a stable process, manual sampling is usually enough. On a ±0.005 mm part, probing pays for itself.

Every accuracy method above assumes the part is located the same way every time. If the fixture lets the part shift 0.05 mm, no amount of probing or gauge setting will fix the resulting position error. Fixture design is the foundation, not an afterthought.

The rule is 3-2-1 location: three points on the primary datum, two on the secondary, one on the tertiary. For a round part on a Ø400 mm rotary table, the primary datum is usually a face plus a bore, and the clamping force must not distort the wall. Over-tightening a thin-wall part is a common cause of out-of-round bores.

For a batch of parts, we use dedicated soft jaws or a fixture plate machined on the machine that will run the job. That way the fixture's own error is cut into the setup, not added to the part. Off-the-shelf vises are fine for ±0.05 mm work but not for ±0.01 mm position tolerances.

Check the fixture before the run, not after. Indicate the locating surfaces, confirm the clamp repeatability, and run a first-article inspection. If the first article is good but part 20 is not, the fixture is usually the reason.

A machine tool grows as it warms up. The spindle, ballscrews, and bed all expand, and the part expands too. On a 300 mm aluminum part, a 5 °C rise moves the size by roughly 0.035 mm. That is more than the total tolerance on a tight job, and it happens without any warning on the display.

The fix is to let the machine reach thermal equilibrium before the first measurement. A warm-up cycle of 15–30 minutes at cutting speed is standard practice on tight work. For parts held to ±0.005 mm, we run the spindle through a warm-up program and take the master gauge reading only after the machine has stabilized.

Coolant temperature matters too. A chiller holding coolant at 20 ± 1 °C removes heat from the cutting zone and keeps the part from growing during the cut. Without a chiller, a heavy roughing pass can raise the part temperature enough to move the finish cut by 0.02 mm or more.

Measure at the same temperature as the machine. A part that measures 20.000 mm on a cold granite plate may measure 20.018 mm on a warm shop floor. For tight work, let the part cool to room temperature before final inspection, and record the temperature in the inspection report.