Merging two ski activities yields incomplete results

[robby]

Hi there,

I signed up as a donor hoping to merge two Garmin FIT files that together, comprise a single day of resort skiing. However the output has a few issues.

First, a small thing, but I'd love to see the body battery metric supported. I believe Garmin calculates this in Garmin Connect in realtime, by subtracting the ending value from the starting value (vs storing a static value), so ideally a merge would contain the starting value from the first file and the ending value from the last file.

More importantly, the merged file loses the notion of downhill runs vs uphill lifts. The lift portions no longer show up as such on the Garmin Map, and the numbered "run" markers for each run are no longer present on the map when using the merged file.

I also noticed the merged file is about 148kb while one of my source files alone is 412kb, a bit of a red flag.

Any chance this ski run issue is resolvable, or is the current tool not up to this task? Should I be worried about the ~300kb of data that didn't make it into the merged file?

[fulmar2]

Hi Robby -

Some of this is true. To get the maximum number of data fields today, you want to check this box:

viewtopic.php?p=515&hilit=rarely#importConnectIQ

However, I know that downhill / uphill is not one of the fields currently implemented. The list of currently supported fields can be found here:
viewtopic.php?p=501&hilit=supported+fields#p501

With regard to the gap: can you show me a screenshot of where Garmin does not fill in the line between gaps? My experience has been that all viewers display one continuous line:
viewtopic.php?p=61&hilit=continuous+line#p61

However... if Garmin has started to respect stopped time while drawing the map, then for sure you want to check the "Try to insert stopped time" checkbox, and also check the "Ignore Big Gaps" checkbox (and set the gap distance/time that you want to leave out). That way, GOTES Will insert the stopped time.

As far as file size is concerned, yes, GOTOES files often are smaller than some of the Garmin files. This is because GOTOES only exports fields that Garmin has documented (and a few I've been able to "reverse engineer"). Surprisingly, Garmin has not described what all the fields do in their SDK! It's a big mystery!

But there is a light at the end of the tunnel - I have been working on rewriting the entire website from scratch. I've been working on it mostly because of an ongoing DDoS attack, and had to make the site much more resilient to bots - rejecting millions of connections a day... but because I'm redoing the architecture entirely, that means that I've rethought how the tools works. A lot has changed since 2015 when I began this, and I never envisioned all of the FIT fields that Garmin has enabled. In the old tool, I had to painstakingly add FIT fields one at a time. In the new architecture, it will autodetect (any fields in the SDK).. and I hope to give users the option to blindly include the undocumented fields (with a warning that it might not work). I've been working about 17 hours per day since early December. My health has suffered greatly from this marathon push, but I hope to give you what you want in the upcoming weeks.

[robby]

Hi Robby -

Some of this is true. To get the maximum number of data fields today, you want to check this box:

viewtopic.php?p=515&hilit=rarely#importConnectIQ

However, I know that downhill / uphill is not one of the fields currently implemented. The list of currently supported fields can be found here:
viewtopic.php?p=501&hilit=supported+fields#p501

With regard to the gap: can you show me a screenshot of where Garmin does not fill in the line between gaps? My experience has been that all viewers display one continuous line:
viewtopic.php?p=61&hilit=continuous+line#p61

However... if Garmin has started to respect stopped time while drawing the map, then for sure you want to check the "Try to insert stopped time" checkbox, and also check the "Ignore Big Gaps" checkbox (and set the gap distance/time that you want to leave out). That way, GOTES Will insert the stopped time.

As far as file size is concerned, yes, GOTOES files often are smaller than some of the Garmin files. This is because GOTOES only exports fields that Garmin has documented (and a few I've been able to "reverse engineer"). Surprisingly, Garmin has not described what all the fields do in their SDK! It's a big mystery!

But there is a light at the end of the tunnel - I have been working on rewriting the entire website from scratch. I've been working on it mostly because of an ongoing DDoS attack, and had to make the site much more resilient to bots - rejecting millions of connections a day... but because I'm redoing the architecture entirely, that means that I've rethought how the tools works. A lot has changed since 2015 when I began this, and I never envisioned all of the FIT fields that Garmin has enabled. In the old tool, I had to painstakingly add FIT fields one at a time. In the new architecture, it will autodetect (any fields in the SDK).. and I hope to give users the option to blindly include the undocumented fields (with a warning that it might not work). I've been working about 17 hours per day since early December. My health has suffered greatly from this marathon push, but I hope to give you what you want in the upcoming weeks.

Thank you for the quick response! I did use the "Parse Rarely Used FIT Fields" checkbox, but yes, I see some of the fields I'd like are not supported by design right now.

With respect to the gap, I think I misunderstood how the "Try to Insert Stopped Time" checkbox works. Intuitively I thought I would want that unchecked for what I'm describing, but you're right, when I check it, the issue I described seems to go away. PEBCAC, it seems, on that one -- sorry!

It would be great if there were a generic way to include all data, even that which isn't understood. But I understand that's a hard problem and probably impossible to get right, especially when it needs to be summarized and you don't know whether to capture the min, the max, the first, the last, an average, etc. FWIW, I've had pretty good success getting Cursor and ChatGPT to manipulate FIT files, working backwards from what I know I want the output to be to reverse engineer what some of the fields do. It's often specific to a FIT file or an Activity type, though -- not generic enough to work across the board.

Appreciate the quick response and best of luck with the new version. Looking forward to trying it out when it's ready.

[fulmar2]

If you want, you can upload the files you want to merge and let me run them against the new tool. I've been collecting "non standard" files to truth test the new tool, and your files fit the bill. The foundation of the tool was designed to support Strava (not Garmin)... and cyclists/runners/swimmers. Hence the rebuild to support the new sports Garmin has added to their devices. Getting it right surprisingly challenging. You can upload FIT files directly to this forum, or you can contact me via the web contact form.

[robby]

Sure thing, here they are. Even if it doesn't work out this time, feel free to use these files as a future test case if helpful.

[fulmar2]

Thanks! and one more thing: Can you please send a screenshot of an instance where Garmin Connect does NOT show a line in-between runs?

[robby]

Thanks! and one more thing: Can you please send a screenshot of an instance where Garmin Connect does NOT show a line in-between runs?

To be clear, it always shows the uphill lines, but it grays them out in a separate color and adds numbered markers at the start of each run. When I used the GOTOES combine tool, it got rid of the markers and made the track show up as one big speed heatmap (inclusive of the uphill). Here's what the original FIT file looks like in Garmin Connect:

[fulmar2]

OK, I ran it through the new tool, and it works. I appreciate having your files as "test files" before releasing the new tools!! Thanks. I attached it here, and hope to publish the new tool in about a week. Meanwhile, happy skiing!

[robby]

Awesome, this is much better and yes, both the table and map now show correctly - thank you!

There are some data missing in the new version (namely elevation, and the file size is still quite a bit smaller). Not making any request here, but sharing the old + new screenshots in case it helps you down the line at pulling all the data in.

[fulmar2]

Thanks for the reply! Yes, I'm still fine-tuning the new site. Your original files did not have elevation recorded; they just had "enhanced_elevation" - so as a default I set the new tool to substitute in enhanced_elevation for regular elevation if regular elevation is missing.

Also, one more thing: the new tool (coming in a few days) lets you optionally import/export fields that are NOT defined by the Garmin SDK. This is kind of a "use at your own risk" but it allows you to export all of the original data at your option/peril (i.e. file bloat)

Also, don't be TOO concerned about file size. GOTOES is rewriting every file from scratch. The reason for this is that then I can verify each and every data field against known parameters (base type, units, range). A big part of GOTOES is repairing corrupted files, and a rewrite from scratch allows me to ensure non-corrupted files. A rewrite from scratch is a little bit like defragmenting your hard drive - the data gets ordered more efficiently, so file size can be smaller if it was initially stored in a less efficient manner.

At a teaser, here are the options available in the new tool (this is just a partial list; I had to super-minimize my screen to just show a tiny fraction of the data fields in just one of your files).

Also, I have added the replacement file; I ran it through the new tool with all options selected (including non-defined fields) so you can get closer to your bigger file size.