Managing Really Large FIT Files

[fulmar2]

About 0.06% of the files uploaded to this website fail. That's not a lot, but most of these failures are caused by really huge FIT files. I've noticed that Wahoo devices in particular can create bad FIT Files. Furthermore, people who have "Log HRV" turned on and "1 Second Recording" turned on in combination with really long activities can have a failed upload / merge. If you don't know what "Log HRV" or "1 Second Recording" is, I'm not referring to you!

I have wanted to help these people on the fringe, so I am busy writing a new tool that can grab all of the important streams from these huge FIT files - and discard the junk that is weighing my tool down. The tool Converts FIT Files to CSV Files..

Let's say you upload a FIT file to the merge tool, and you get a crash screen like this:

If you get a 500 error like this, maybe you need to convert your BIG FIT File to a CSV file.

The 500 error with a FIT file is a signal that you may have too much junk data embedded in your FIT file for my tool to parse it all out. You may have noticed that FIT files are very small in size (something like 1Mb sometimes). The resulting CSV file would actually be quite large (maybe something like 14Mb). How could a CSV file be better than a FIT File?

It turns out that FIT Files need a lot of processing to determine the value for each message. It takes a lot of server time and power. For example, a person sent me a 1.3Mb FIT file, and it took my tool over 90 seconds to process. That same file - once converted to CSV - was able to be processed in 2 seconds. The trick is to Zip Compress the CSV file before uploading. (how to ZIP on a Mac or PC) Once your CSV file is zipped up, it will be tiny: usually smaller than the original FIT file.

Here is how to do this:

1) Go to the Convert FIT Files to CSV Files Tool.
2) Upload your troublesome FIT file (the tool ONLY accepts FIT files; nothing else).
3) Once the file is uploaded, download the CSV file.
4) (optional) Zip the CSV file if you don't want to wait a long time to upload it back to the Merge Tool
5) Your file should now process VERY quickly.

Check out the difference in time between uploading a FIT file directly vs converting it to CSV first!

Hope that helps you 0.06 percenters!!


Here is a discussion on other strategies for uploading large GPS files.