Smoothing an animation curve can be done through the animation curve's “Edit/Filter” option.
However, this will move the original keyframes which seems a little destructive and doesn't allow a nice fine tuning workflow (other than jumping up and down on the undo button and trying again with a different value).
So here is an expression based solution that uses a user knob value to smooth the curve “live”:
In my Tracker node I have this animation curve:
I have also created two floating point sliders called “smooth” and “smoothedCurve”, one to adjust the amount of smoothing and the other to hold the resulting smooth curve respectively:
The magic in this whole smoothing business is using the integral of the knob that holds the animation (in this case it's track1.x) over a certain range. In other words we want to use something like integrate(a,b). Let's do this just for the range of three frames (previous frame to last frame). Dividing the whole thing by (b-a) will give the average value and thus a box filter (thanks to Bill Spitzak for posting this solution on the nuke ailing list once, I never paid enough attention at high school to even know what this is exactly doing). I do know however, that the next frame minus the previous frame always equals 2, so here we go:
The resulting curve:
This is hard coded to a 3 frame range though (from previous to the next frame of each frame), so let's hook it up to our custom “smooth” knob to make it more interactive:
And that is all. Now you can drive the smoothness of the custom curve by cranking the "smooth" knob: