How to handle very large FIT files on GOTOES. (Solved)

A place for the community to help each other out with getting the most out of the Tool to convert FIT files to CSV format.
Post Reply
User avatar
fulmar2
Site Admin
Posts: 228
Joined: Wed Nov 25, 2020 4:21 am
Contact:

How to handle very large FIT files on GOTOES. (Solved)

Post by fulmar2 »

Here is the solution to handling FIT files that are too big for Garmin/Strava/GOTOES:

The FIT file format allows for the storage of a massive amount of data in a relatively small file. Frankly, it's amazing how much information you can pack into such a small file. A problem arises when there is SO MUCH data in the file that ordinary tools can't handle all of the data points. Once a FIT file exceeds about 2.5Mb in size, it probably contains more information than standard approaches can handle. Today, for example, I was looking at a 5.4Mb file that had a quarter million time stamps. Each stream (power, heart rate, latitude, longitude, etc.) had its own set of 1/4 million points. Add that all up and you have several million points. On top of that, by recording HRV (Heart Rate Variability), you now have points recorded in-between-seconds. All together, it's too much to process.

To resolve this, I built the FIT -> CSV tool which extracts just the important data form the FIT file and ignores all of the extra "junk". Here is how to use that tool to get your files processed on GOTOES.



Step 1:
Upload your problem file to the FIT -> CSV tool.
Upload your problem FIT file to the converter.
Upload your problem FIT file to the converter.


Step 2:
After the upload, download the converted file by clicking the big blue button:
Download the converted FIT file that has been converted to CSV.
Download the converted FIT file that has been converted to CSV.


Step 3:
You will notice that the CSV file is MUCH bigger than the original FIT file. That is OK... The file is big because the data is not compressed. In reality, though, we have actually removed a lot of the "junk" data from the file. Often the CSV file is so big that you will need to compress it in order to re-upload. GOTOES has a 15Mb upload limit. You can shrink the CSV file for re-upload by zipping it. If you don't know how to zip a file, here are instructions. My screen shot below shows me making a ZIP file on my Mac by right-clicking on the GOTOES CSV file:
Zipping a GOTOES CSV file to re-upload to the Merge Tool
Zipping a GOTOES CSV file to re-upload to the Merge Tool


Step 4:
Now that you have a smaller file, go back to the merge tool and give it another go with the zipped file. Often, even with enormous files with millions of points, GOTOES can handle processing within a few seconds. See screen shot below. This was a 5.4Mb FIT file that became a 42.7Mb CSV file (4Mb after I zipped it). Processing of the zipped CSV took 5 seconds, where as the FIT file times out after 180 seconds. Note how the tool tries to encourage you to reduce tracepoint density if you're going to export as FIT. This will make your file more "digestible" by apps like Strava. You can override that setting, but there is no guarantee that Strava will be able to handle it. Enjoy!

Success!  Processing a super big FIT file on GOTOES.
Success! Processing a super big FIT file on GOTOES.
Post Reply