Does the Merge Tool also combine developer fields from two FIT files?

[Hellkeeper]

Hi everyone,
quick question about the Merge Tool:

Can it merge developer fields from two FIT files (e.g., TymeWear ventilation data FIT + a normal activity FIT)?

I’m seeing that the default (standard) fields are merged, but the developer fields don’t seem to come across. Is that expected?

Specifically, can developer fields whose names start with tyme* from the secondary FIT file be included in the merged result?

If this isn’t supported, is there any workaround to preserve those developer fields during a merge?

Thanks!

[fulmar2]

Yes, I have a FAQ about this already. Rather than create a new one, I will direct you to the existing one:

viewtopic.php?p=515&hilit=rarely#importConnectIQ

Does that answer your question?

[Hellkeeper]

Hey @fulmar2,
thx for your answer - according to https://forum.intervals.icu/t/merging-f ... hnederle-w - it's not working because of:
Quote MedTechCD:

Code: Select all

I tried, but Gotoes has a problem with the US formatted numbers from TymeWear. 
Gotoes expects Euro metric values in the FIT. Comma/dot is causing wrong interpretation.

[fulmar2]

Thanks for your reply. That is insightful. Can you please upload a sample FIT file to this forum? Just FYI, GOTOES is format agnostic when it comes to Euro/US data values. That is because FIT files are binary, not text based like CSV, JSON, GPX, TCX, etc. In other words, that answer doesn't "really" apply... however, it is possible that the developer data fields are not formatted currently (like maybe they are using a text field to represent numeric values)? In that case, the downstream interpreter may be interpreting them incorrectly.

If that is the case, the CORRECT solution would be for "TymeWear" to properly format their files.

Having said that, though, I would be happy to look at your source FIT file and see if that is really the issue. Moreover, I might consider writing a work-around to adjust to TymeWear's improperly created data fields. Often I can write code to fix other vendor's problems much faster than the original vendors can get to it. In fact, sometimes I find that rather than adhering to proper formatting, vendors just tell their users to use GOTOES as a work-around! So send the file and I'll look into it.

[j4m1eb]

I’ve just come across this post as I’m to looking for a solution to merge an original fit file from either TrainingPeaks virtual or hammerhead Karoo with a tymewear fit file.

The issues are. Tymewear has merge facility on their own website but it strips all gps data from merged file. Then when uploaded to intervals everything including breathing data power hr etc is there but no distance elevation or map.

The fields tymewear writes are fine and show in both intervals and TrainingPeaks.

I’m looking for a merge that respects the TPV or Karoo data but just adds the tymewear specific fields.

Tymewear are slow fixing this. Happy to donate if this feature is possible. I know a few other users with the same issue looking for a solution.

Happy to send you 2 files from same ride. “Master” and “tymewear”

Thanks

[fulmar2]

In order to help, I'm going to need as much information as possible:

1) A raw time wear FIT file
2) a raw Karoo FIT file.
3) Screenshots of what you want, and what you're not getting from GOTOES.

Yes, GOTOES can do this, and yes, GOTOES creates valid files. In a sense, GOTOES is a file validator as It won't export invalid fields. A previous user posted a quote from Tymewear, but that is not a correct response from them because FIT files simply do not have "localization" (i.e. commas vs decimals) unless it is a text field (i.e. not a data field). I'd be surprised if Tymewear is using text fields to store data... Though I have definitely seen stranger things from new device makers! Upload all you can to the forum so I can begin working on a solution for you.

[j4m1eb]

Hi

Lets see if this helps.
This a CX race from yesterday.

Karoo would be the "master" even though the time elapsed is shorter as its the actual race.
To record the tymewear data, you start an activity on your phone, leave it in the car then finish it and upload once back at car, therefor the elapsed time is way longer.

The data I would like merged into the Karoo or TPV fit file is just the the tymewear fields, for example "tyme_ve_zone"
It's fine that tymewear data get chopped off as I only ever want it for the actual recorded activity.

The tymewear fit files all work and show data currently in TP and intervals with the naming they use so no need to change them.

I can record a TPV fit file and corresponding tymewear one tomorrow and share but the requirement is the same, leave "master" fit as is and add tymewear fields wear timestamp matches.

Maybe this already works, I haven't donated to try so was just asking if it did before donating

Thanks

[Hellkeeper]

Thanks for your reply. That is insightful. Can you please upload a sample FIT file to this forum? Just FYI, GOTOES is format agnostic when it comes to Euro/US data values. That is because FIT files are binary, not text based like CSV, JSON, GPX, TCX, etc. In other words, that answer doesn't "really" apply... however, it is possible that the developer data fields are not formatted currently (like maybe they are using a text field to represent numeric values)? In that case, the downstream interpreter may be interpreting them incorrectly.

If that is the case, the CORRECT solution would be for "TymeWear" to properly format their files.

Having said that, though, I would be happy to look at your source FIT file and see if that is really the issue. Moreover, I might consider writing a work-around to adjust to TymeWear's improperly created data fields. Often I can write code to fix other vendor's problems much faster than the original vendors can get to it. In fact, sometimes I find that rather than adhering to proper formatting, vendors just tell their users to use GOTOES as a work-around! So send the file and I'll look into it.

Hey there,
thank you for your work on this!
Here are the Files in question!
Bye
Andy

[fulmar2]

Jaime -

Oh, sorry; I thought you were suggesting that GOTOES does not work; not that you wanted to test out the platform. I took your files and loaded them into the GOTOES tool. I did a smart (overlap) trim, and then selected the ConnecIQ fields from both of the files. There were quite a few developer data fields, as you can see from the screenshots, so I am guessing you have A LOT of sensors connected. GOTOES definitely successfully merged the data (you can see it in the GOTOES FIT file viewer):

GOTOES merged the Developer Data fields. Notice how we have Tyme AND Karoo data fields in the merged file.

I then opened up a Tymewear account online, but could not find any way to upload the file I had created for a test. It seems as if you might need to be supplying files directly off the device? If this is the case, GOTOES might not be able to help, as I see no way to manually import files. Maybe you know how to do this?

Here is a shot of all your connected sensors (at least what the devices are recording)!

I also uploaded the merged file to Garmin, but I saw that none of the developer data fields have been registered with Garmin (this includes the Karoo developer data). In order to "see" the developer data, it must be registered with the platform that you're viewing it on. So, if tymewear is able to read a file that has both Karoo and Tymewear data, this attached file (merged-6.fit) should work for you. I tried it in Garmin and could not see the Tymewear data.. but I could not see the Tymewear data in your original file either because Tymewear has not registered with Garmin. Tymewear has also not registered a Manufacturer ID, so the device is currently "unidentified" across platforms. If they get registered, I can add them to GOTOES (just let me know when this happens).

[j4m1eb]

hi the merged file doesn't work, it has some data spikes but is missing the actual data and the spikes are 100x higher than possible.

Nothing shows in Garmin buyt I guess they don't support it and I don't use Garmin anyway.

Both Intervals and Trainingpeaks do support all the Tymewear fields.

The easiest way for you to test would be to upload to TP and look at their beta charts which support all the metrics.

Tymewear have the ability to merge a fir file from a headunit or whatever on their site but then I'm back to the main issue, it takes power HR etc, but strips all GPS data so no distance elevation grade or maps which makes it pretty useless.

Do you want one of their merged files for the 2 I supplied?

[j4m1eb]

bit more info having done some digging.

I've compared 2 timestamps and although GOTOES has merged the data the figures are all wrong which is probably why it is all wrong in Intervals and TP.

this one is to show the fields recognised in TP from a Tymewear file

this is the merged files but wrong numbers

this is the merged file uploaded to Intervals, there are just a few spikes

this what the data should look like in intervals - this is the un merged tymewear file

[fulmar2]

Ah, sounds like I can use Training Peaks to test. Are you a paid member of Training Peaks? The reason I ask is because even when I upload your original Tymewear file, I cannot see the Tymewear data in my account. Also, I was unable to import files to Tymewear for testing..

However, I do see from your screenshot, the most helpful one of the raw data, that there does seem to be some sort of units difference in the GOTOES file. GOTOES does not manipulate the developer data, but instead inserts it as-is. However, GOTOES does read the field types assigned by the file and puts them in the output in the same format as GOTOES reads in. So, there does seem to be a discrepancy there.

The short answer to your question seems to be that as of today (November 24) GOTOES needs to be updated to work around this data format anomaly. I'm going to be out of town for 3 weeks, but when I return, I'll work on making GOTOES work with Tymewear files.

[j4m1eb]

I am a premium TP member, just to do a free account to see if it works but if gives you 14 days premium free

You could test on intervals.icu following the below image. You upload file and view it, click on charts and search for VENT and then add the one that comes up. There loads already made in there if you search VE or tymewear, which will allow you to see the data.

Thanks for your help on this.

[gloscherrybomb]

[fulmar2]

This is helpful, especially the AI response. I examined both the input as well as the output, and both fields are being treated the same way (as float32)... But actually, I do see how the response (above) from Tymewear is actually correct. It turns out that GOTOES is seeing the arrays as single numeric values. I will fix it. Probably won't be until I get back, but I do see the issue.

[j4m1eb]

Just to say Tymewear has updated their dashboard and removed the ability to merge completely which is unbelievable. So if you can get this working I think it'll be every useful to a lot of customers.

Happy Thanks giving.

[fulmar2]

OK, I will start working on this today. May take a week or so.

[j4m1eb]

thanks, Tymewear still have no merge at all now

[fulmar2]

OK, after several hours of work, I think I may have identified the issue. I am not able to easily test on the tymewear website, as there is not a way for me to easily upload and delete files. I get the feeling that their site is very much a work-in-progress.

I did follow your instructions on intervals.icu, and if I'm not mistaken, I have fixed the issue with GOTOE's interpretation of the values. Indeed, this was a bug with GOTOES (Assuming it is now fixed). Can you please report back to me to make sure that this issue is truly resolved?

[j4m1eb]

Hey, I become a donor and tried it. It's very nearly there.

It has combined the data an d the value look correct but having tried 2 activities the data is about 6 minutes out in terms of timestamp.

The tymewear breathing data is too soon.

See below, you can see were power is applied and HR rate rises, breathing is a very similar graph to HR but is way too soon.
Red is HR and breathing is blue. You can see the final sprint and peak HR, breathing is about 6 mins before, and you can see where HR drops at end of activity, again breathing finish way too early.

As a second request would it be possible to toggle on/off tymewear fields that are added as their HR data is funky and I only need the breathing related fields

[fulmar2]

Hi j4m1eb -

GOTOES does not alter timestamps within the files; it uses the embedded time. In fact, time is the ONE field that GOTOES never changes unless you're using the timestamp tool. Typically, with outdoor units (like GPS), the device captures the precise time from the satellites. However, it appears that you're using two non-GPS units (I don't know where your TrainingPeaks file is capturing position data from, but perhaps it is an indoor activity?). Regardless, GOTOES bases everything off of the time - so you will need to set the time to be correct (or at least matching) in both devices.

Moreover, I can see from your screenshot that you might also want to use the "smarttrim" feature to cut off the head and tail of the overlaps. This makes your uploads more tidy.

With Regards to your request about removing developer data fields - I will consider that; but before I do, how much of an inconvenience is it to have an extra data stream? Can you just hide/show that in your viewer? I'm asking because adding the ability to remove the datastreams is probably another 2+ days of work for me, and I've recently been tasked with some other important feature requests. I am guessing that this isn't something that will help many people - but please correct me if I'm mistaken.

[j4m1eb]

Thanks for the info.

I've merged another activity, this time main data recorded with Karoo, Tymewear is recorded by the strap then synced back to their phone app for upload, time is out again so I assume Tymewear is the issue, not sure how to fix.

Is time shift possible?

The dodgy HR bit in in this file. Not super important. Think it's just how tymewear deals with no data. I was fixing a mechanical, for some reason HR stay at last record then jumps to new value. Throws metrics off a bit, but hopefully it's a one off.

Tymewear merged

Karoo only

[fulmar2]

For the first issue, I am not familiar with Tymewear devices - but is it possible for you to set the time on that device? This is almost certainly the issue there.

For the 2nd issue (the flat line heart rate after the merge), this is a feature of GOTOES where if your device is turned off/not recording, it continues with the "last known heart rate". The purpose of this feature is to repair files where the heart rate monitor drops out frequently. Anything below a heart rate of ~26 is "carried over".

However, in a FIT file, you can insert start/stop time. Depending on the viewer, start/stopped time may or may not be respected. I am not sure how your viewer handles stopped time, but you probably want to check the insert stopped time box anyway to make your output be accurate.