With Kinetic control, certain toolpaths seem to result in lost steps in X and Y axis

Troubleshooting the Pocket NC
#1

Before I dig into this further, I thought I would post to see if anyone has seen anything similar. I have Kinetic Control v0.0.6 (I know I should update) with my relatively early v2-50.

I’m running a toolpath generated by F360’s adaptive clear that has, in places, a lot of small moves and retracts. In certain geometries, it sure looks like I can make the PNC reliably lose steps in the X and Y direction, even when not cutting anything. In my limited tests so far, this manifests as step-downs in the adaptive clear not lining up, or in crashes when the PNC decides to take off more than the nominal cut width. It seems to happen in regions where the control has to slow down to follow the desired path. This is with the new post-processing smoothing factor set to 0.01 and 0.005.

If it were just a bad wire, I’d expect it to happen more randomly, rather than at the exact same places in the program. However, given that it is a seemingly random amount of steps lost, it seems less likely to be something in software. Also, since I’m seeing lost X and Y steps, hardware failure seems less likely.

Has this been seen anywhere before or have a known resolution? If not, I suspect I can come up with a reproduction recipe of sorts.

#2

Hey Josh,

Sorry to hear you are running into an issue with your machine! We have not had any other reports of this from other Kinetic Control users but I would be happy to help you investigate it further.

The first thing I would do is “uninstall” Kinetic Control and then run the same program again to see if you get the same results. This can be done by simply removing the microSD card that Kinetic Control boots from and then connecting to the machine as usual (you may just have to clear your browser’s cache to get the old UI to show up).

Keep me posted on what you find and then we can go from there!

Best,

Q Rothing
Applications Engineer, Pocket NC

#3

Q,

Sounds good. That was in my plan for debugging. I’ll try to get a somewhat reliable reproduction recipe and then try downgrading to see if that changes anything.

Josh

#4

Sounds good, Josh!

Let me know if there is anything I can do to help.

Thanks,
Q

#5

I now have a partial understanding of what is happening, and believe it is related to the G64 path smoothing.

First, I’m testing with a Fusion 360 adaptive clear toolpath that results in lots of complex curves to carve out a cross shape with a circle in the middle. So it clears the slots with a bunch of small moves, then gradually expands to clear out the center before moving on to each of the remaining slots. It uses an optimal load of 0.25mm, so it is supposed to engage the tool at most that much with each pass.

With the legacy controller, I always ran the Fusion 360 post leaving the “Smoothing Tolerance” parameter at -1, which put no smoothing control directives into the gcode at all that can I see. So no G64 or G61 in the gcode. The legacy controller’s UI reports that G64 of some kind is active from power up, but I don’t know how to tell what values it is using. When running my toolpath on the legacy controller in this mode, the adaptive tool paths seem to run at full speed all the time with no slow downs. I also have never observed axis slipping in this configuration.

With the Kinetic control, G64 is also reported as active at power up. Using the same gcode from above with it results in extraordinarily slow movements, once the tool path is anything more complicated than a very gradual curve. So its at-boot parameters seem to be different from the legacy controller. I have been setting semi-random values for the “Smoothing Tolerance” in the post-processor to work around this. I was using 0.01 (which translated to G64 P0.01, so I think 0.01mm since I’m in G21) typically. For the tests with the biggest problems I selected 0.005 for some reason. With the 0.005 smoothing parameter the control not only moves more slowly, but does lots of other bad things and seems to follow the paths much worse, or at least much less evenly. The adaptive clear was supposed to take 0.25mm per pass, but I saw it oscillate from probably 0.05mm to >=0.5mm. This setting was where I saw the worst axis slipping as well. With the 0.010 that I had been mostly using, there is still some stuttering in curves and some amount of pass-to-pass unevenness but not as bad and the entire toolpath would sometimes complete without issue.

The LinuxCNC documentation indicates that the default should be “G64 P0” which should “sacrifice path following accuracy in order to keep the feed rate up”, which is what I assumed the legacy controller was using - Important User Concepts . However, with the Kinetic control, if I commanded “G64 P0”, I got behavior that looked pretty close if not indistinguishable from G61 (exact path mode). I did this by hand, as it is not possible to specify a smoothing tolerance of “0” in the F360 post-processor, it just omits the smoothing control directive in that case the same as for the default of -1.

Note that this isn’t really problematic for tool paths that don’t rely on many small short line segments like the adaptive ones do.

So, I now have several questions:

  1. What path smoothing mode does the legacy controller start in?
  2. How can I configure the Kinetic control in the same way?
  3. Why does the Fusion 360 post-processor omit the smoothing control entirely by default? This was a decent source of confusion for me, as in some tests I was inadvertently using the smoothing parameters from a previous test when I thought I was using some kind of default.
  4. Why does specifying a smaller value for the smoothing tolerance result in paths that track worse and lose steps? I possibly don’t care about this if I can reliably get back to legacy controller behavior which was totally adequate for my use cases. For anything precision, I always use a F360 primitive that emits a single arc or line, and the legacy controller at least tracked that pretty accurately and so has the Kinetic control in my more limited testing.

I took video of all the above experiments that I can annotate and share if helpful.

#6

Hi Josh,

Thanks for the thorough testing and explanations! This sounds like it could be among the most nuanced problems we’ve encountered and appreciate all the experimentation you’ve put in. Would you be willing to share your G code with us to investigate further? With Kinetic Control we upgraded to LinuxCNC’s new trajectory planner so there is potential for some unexpected differences. Unfortunately, I don’t currently have good answers for all of your questions. As far as I know the default smoothing setting should be the same as before, G64 P0. This should result in the fastest, but not necessarily the most precise path following. Once tolerances are set, I would expect things to slow down, but better track the defined paths. I would expect smaller smoothing values to track better, but move slower (it sounds like this is what you are expecting as well, but isn’t what you’re experiencing). For most tool paths, I expect people don’t need to fiddle with the smoothing settings at all, so not having a default in the Fusion post processor isn’t missed by most people. This is something we could get added. As for skipped steps and poor following, one hunch I have, is that this could be related to acceleration values we had in there that weren’t necessarily being reached in the legacy trajectory planner, but now may be and it’s possible this could be causing steps to be skipped. Have you tried slowing your feed rate down in any of these examples (just using the feed override slider)? Smoothing is used to balance speed and accuracy. If you slow your feed rate considerably, does the skipped steps and poor tracking go away? If so, we could try turning down some max acceleration values and perhaps that could help.

For what it’s worth, the version of LinuxCNC that you should reference is 2.8. I don’t think it differs much in this case, but for reference:
http://www.linuxcnc.org/docs/2.8/html/user/user-concepts.html#sec:trajectory-control

http://www.linuxcnc.org/docs/2.8/html/gcode/g-code.html#gcode:g64

If you send me the G code, I can try to take a closer look.

-John

#7

Sure, the g-code I’ve been experimenting with is linked below. Unfortunately, there is a fair amount of just cutting away at the outside diameter before the interesting part happens. However, even that outside part slows down significantly under the default Kinetic parameters, even if it doesn’t cause significant problems.

The slowing down and stuttering is obvious even cutting air. The path-to-path unevenness and lost steps are easier to see when cutting as you see the chip size change or the endmill crash.

#8

I’ve been reading through this LinuxCNC thread about the new trajectory planner. This post says that the G64 behavior has changed slightly:

It sounds like in the previous planner passing only a P parameter will automatically assign the same parameter to Q, which controls the naive cam detector which collapses points into a single segment when the points deviate less than Q. In the new planner, Q must explicitly be assigned, otherwise it will be 0. So, to match the old behavior, you would need to specify both the P and Q parameters. I still don’t see how this could explain the slow down in default behavior of G64 P0, but it may explain your experiments with adjusting the P parameter.

#9

It looks like you should be able to disable the new trajectory planner. See the TRAJ section on this page:
http://linuxcnc.org/docs/2.8/html/config/ini-config.html

I’ll try to do some experimenting next week, but hopefully this gives you something to try.

#10

No worries. Unless you have a problem reproducing, I’ll just stick with the legacy software unless I need TCPC or M999 for a particular job so I’m fine for now.

#11

Hi Josh,

This morning I was able to experiment with this a bit. I didn’t run the program on the legacy controller, but I did run your program (cutting air) on Kinetic Control. I see you have a feed rate of 750 mm/minute. The initial moves do hit 750 mm/minute, but then it starts to sputter and I see the feed rate jump between 200-600 mm/minute in the feed panel. This is a calculated speed based on the current position of the machine, so it indicates a problem. I read through the documentation on the [TRAJ]ARC_BLEND_OPTIMIZATION_DEPTH and it provides some suggestions for when strange slowdowns occur: INI Configuration

Specifically, it suggests setting the naive cam tolerance based on the desired speed (750 mm/minute = 12.5mm/second) and servo period of the machine (.001 seconds). Their suggestion is to set the naive cam tolerance (the Q parameter of G64) to half of 12.5*.001, so 0.00625. When I do that (G64 P0.00625 Q0.00625), I see the feed rate stay a constant 750mm/minute. That suggests to me an improvement, but I was only cutting air so I can’t know for sure how it affects the cut quality.

I couldn’t find any documentation that points to a difference in functionality when P and Q are 0, but it definitely seems like it doesn’t do what it used to (and what is currently documented). This suggests to me we should include a different default for the smoothing parameters and should look into updating the post with smoothing options. I think the key thing you were missing in your tests was matching the P and Q parameters, which are now set separately (despite the documentation which says otherwise) when before the Q parameter was set automatically to match the P parameter.

I hope that works for you. Let me know how it goes.

-John

#12

This document suggests the default smoothing parameter is .001 (see the RS274NGC_STARTUP_CODE parameter):
https://linuxcnc.org/docs/2.8/html/config/ini-config.html#gcode:ini-features

I didn’t have any success overriding that RS274NGC_STARTUP_CODE with different smoothing parameters (any changes I make, simply seem to be ignored). I did try running your code after running G4 P0.001 Q0.001 and it also resulted in a steady 750mm/second feed rate, though.

#13

A minor update.

I tried running G64 P0.001 Q0.001 and while it was better, there were still slowdowns in some paths (I didn’t run it long enough to see if there were any accuracy issues). Same with G64 P0.005 Q0.005.

I’m at G64 P0.05 Q0.05 at the moment, for which I’ve seen no issues, although my testing is not super exhaustive. For roughing passes 50um is fine, but it would be nice to know how this differs from the old default eventually.

#14

I’m still trying to get a better understanding myself. In the mean time, I did push out an update (v5.1.1) that defaults the smoothing to 0.001 inches (I only set the P value, but upon further experimentation it does seem like the Q parameter is automatically set, as well). Also, in the active codes section the G64 parameter now also shows the active P and Q parameters.

#15

Am I correct that the values are measured in user units? Since my gcode is all in G21, that makes the values in mm instead of inches?

#16

Yes, the values passed to G64 are in user units. By default the machine starts up in inches and that is when the defaults are set. When you issue a G21 it will perform the conversion and you will see the P0.0254 Q0.0254 in the active codes. If you follow up with another G64 P0.001 Q0.001, then it will be 0.001 mm.

#17

Unfortunately, after a bit more digging it’s looking like these slow downs are due to a bottleneck in the interpreter of Kinetic Control. We’ve been aware that there is a performance hit across the board in all the major processes started by LinuxCNC, but didn’t realize it impacted the speed of the interpreter so much in cases with many small moves. It’s hard to determine the exact source of the performance hit because there were so many updates from the legacy software to Kinetic Control (upgrade from Debian Stretch → Debian Buster, upgrade from Python 2 → Python 3, upgrade from MachineKit → MachineKit-HAL+EMCApplication, etc.). Luckily, the new trajectory planner does seem to be more performant than the old one (you can switch the old one back on in Kinetic Control, but it performs worse than it did in the legacy software) and tuning smoothing parameters can help, but for now switching back to the legacy software is the only way to get same level of performance for programs with many small moves. I’ll keep you posted with any updates as I work towards improving performance.

#18

Thanks for the update!

At least for me, fiddling with the smoothing parameters seems to have gotten Kinetic to a usable state in that I don’t observe missed steps or crazy slow downs. It would of course be nice to be able to set that arbitrarily so that I could have closer following without worrying about inappropriate accelerations and skipped steps, but it isn’t critical for me.