| Jim Reisert : DX4WIN | Identity QSOs missing from LoTW |
by
updated 7 May 2013
Identify all QSOs in your logbook that have not been uploaded to the ARRL Logbook of the World (LoTW) web site, so they can be uploaded again. It is better to upload only the missing QSOs, rather than your entire logbook (less work for the LoTW server to do). And as you'll see below, many of the same steps will be required no matter which method you choose, only the pool of QSOs you start with changes.
Whenever a log is uploaded to LoTW, the data uploaded is copied to a temporary, internal file. This temporary file is then placed into a queue for processing by Logbook. Since the inception of Logbook, the name used for this temporary file was based on the identifier of the process (PID) that handled the upload. The defect inherent in this design choice was that PIDs on the Logbook sever (and hence temporary file names) get recycled and re-used every 32k processes. Since a new process (and a new process id) is used every time a web page is loaded from the Logbook server, these 32k PIDs can be cycled through in just 15 minutes during high periods of web traffic.
Even so, the choice of PID as a "unique" id for a temporary file worked fine until processing delays built up a backlog of thousands of logs that resided in the processing queue at any moment. At that point there was a finite chance that each time a new upload arrived, the new log would be assigned the same temporary file name as a log already waiting in the queue to be processed. Whenever this happened, the newly arrived log would over write the previously queued log.
The fix was quite simple once the defect was identified. The changes to correct this defect were to:
Extend the temporary file name generation to
include both a date-time stamp and a process id making it effectively
unique (at least until we are processing thousands of uploads per
second)
and, to be doubly sure:
Perform a check of whether a file with the same name is already present in the queue before attempting to add a new file to the queue.
This was a latent defect that had been present in Logbook since day one. Its effect only became apparent (and occurred frequently enough to debug) during the last quarter of 2012 when the size of the log processing queue became large. The code changes to fix this issue were deployed on 28 November 2012 at approximately 21Z; uploads made on 29 November 2012 (UTC) and after would not be affected by this defect.
There are other cases that can cause missing uploaded QSOs. One way is to change QSO data in your log (callsign, date, time, band and/or mode) but do not export the QSO again after the change. This may happen when you log a station incorrectly, then later receive a QSL with the correct information. This often occurs during contests by "busting" the callsign you logged.
The procedure for identifying missing QSOs and uploading them again is made up of many small, detailed steps. Do not be overwhelmed by the number of steps, most of them will take no more than a few seconds each.
First, upload any pending QSOs to LoTW. Then wait until you can confirm that those QSOs have been processed by the LoTW system (choose "Your QSOs" then click on the "Most Recent QSOs button"), or check the LoTW Queue Status.
Save your log. Copy the <call.DXL> file to a safe place, just in case you make a mistake.
Download your full LoTW log using a query form written by Rick Murphy, K1MU:
Type in your LoTW username (callsign) into the "LotW Username" box, then click the Submit button at the bottom of the form. Save the resulting ADIF file when prompted. You will need this file in step #5 below.
Clear all the LoTW confirmations in your logbook, using:
LoTW | Clear LoTW Confirmed
Answer "Yes" to the Confirmation question. It will report how many QSs were changed.
Import the ADIF file created in step #3 above to "confirm" your QSOs via LoTW, using:
LoTW | Import from LoTW
Depending on the size of your log, this may take a while. You will get an Information box reporting how many QSOs were confirmed, not found, etc.
Search for QSOs not confirmed in LoTW by using F8 and searching for 'N' in the "Cnfm" box on the LoTW line of the QSO window
Filter | Count will show you how many QSOs were not confirmed. This is the same as the number of QSOs in your logbook that were not uploaded to LoTW. In my case, I had just over 3000 QSOs out of 114,000 QSOs, or about 2.5%.
Export the filtered QSOs to a DX4WIN8-format file, using:
File | Import/Export
select DX4WIN8
File | Export
Type in a file name (I used "MISSING"), then click Save. DX4WIN will ask if you want to use the active filter for export. Answer YES.
At this point, you have exported the subsection of your log that you will have to upload to LoTW a second time.
Delete all the QSOs in your logbook, using:
QSOs | Multiple QSO Operations | Delete QSOs
Verify that the logbook contains no QSOs using:
Filter | Count
INFORMATION: Why do this, instead of creating a new logbook file? Because it will preserve all your group selections, which will be useful for steps #12-16 below.
Import the QSOs saved in step #8 above, using:
File | Import/Export
select DX4WIN8
File | Import
Navigate to the file you saved in step #8 above, then double-click to import it. The number of QSOs you imported should be the same as the count in step #7 above. Save this logbook file using a different name than your original logbook (I used "MISSING").
Clear the LoTW upload flags and Upload Date for all QSOs:
LoTW | Clear LoTW upload flag
You will be prompted to clear the flags, answer Yes.
LoTW | Clear LoTW upload date
Again, you will be prompted to clear the flags, answer Yes.
Now here is where things can get a little tricky. In many cases, you cannot sign your entire logbook using the same LoTW certificate and station record. This may be because you changed callsigns or the location that the QSOs were made from. You will need to find a way to filter out (select) only the QSOs you want to sign using each combination of LoTW certificate and station record that's applicable to your logbook. If you have existing group selections (these were preserved in step #9 above), that is the best way to separate the QSOs. Or you use a Date Range search, as long as the QSOs do not overlap.
NOTE: If you do not have group selections corresponding to the different callsigns or time periods of your logbook, now might be a good time to create them. You do not know when you may need them again, and it will save you having to re-create the same selection(s) at a later time. You will need to set these up in your real logbook, then restart at step #4 above.
To filter the QSOs using a Group Selection, use:
Filter | Selection
Using the drop-down arrow in the Current selection box, choose one of your group selections, then click OK.
To filter the QSOs using a Date Range, first bring up your Preferences. On the QSO tab, under Searching, make sure that the "QSO Date" field is set to "Range". Save your preferences.
Now using F8, search for the QSOs by starting and ending date.
You can use Filter | Count as a sanity check to ensure that only QSOs in the chosen selection or date range (and not all) are selected.
TIP: It is easiest to work from inside to outside when selecting QSOs to be exported. Here is an example: AD1C operated the 1992 CQWW CW contest from Derry, NH, not his home QTH (different state, different county). QSOs made during this contest need to be signed with a different LoTW station record than the one for the home QTH. Export these contest QSOs FIRST (before exporting any other AD1C QSOs) and sign them (see below). Now when the remaining AD1C QSOs are exported using a group selection by that also encompasses the 1992 CQWW CW contest, those contest QSOs will not be exported again, because they were already exported (LoTW Upload date was set).
NOTE: It is possible that NO matching QSOs are found in a selection or date range, meaning that all those QSOs are already in the LoTW system. If that is true, just move on to a different selection or date range.
Mark the selected QSOs by setting their LoTW upload flag:
LoTW | Set LoTW upload flag.
If you using a Date Range filter, DX4WIN will ask if you want to use the active filter for export. Answer Yes (you will not get this question if you are using a Group Selection). When DX4WIN is finished setting the upload flags, it will report the number of QSOs that were changed. If the number is zero, or lower than you expect, it could be because you forgot to clear all the LoTW upload flags in your log, see step #11 above.
Now, export those flagged QSOs so they can be signed for LoTW, using:
LoTW | Export to LoTW
Type in a file name. I suggest using the callsign as the file name if these QSOs are all associated with a single callsign, or a location name (i.e. "Austin").
When you click on Save, DX4WIN will ask if you want to use the active filter for export. Answer Yes. Then DX4WIN will tell you how many QSOs were saved.
NOTE: If DX4WIN says that no QSOs were exported, it may be because you forgot to clear the LoTW Upload Date in your logbook. Revisit step #11 above.
After DX4WIN has finished writing the QSOs, it will ask if you want to clear the Upload flags. Answer Yes.
At this point you have two choices:
Export to LoTW all the other QSO "groups" from your log before signing any file for LoTW
Sign each file for LoTW after every export
Use whichever method will help you the most to export all the missing QSOs from your logbook.
When signing each exported group of QSOs for LoTW, make sure to use the correct LoTW certificate and station location.
If you are using TrustedQSL v.1.14 or later to sign the QSOs, it will automatically upload them to LoTW after they have been signed. If you are using an earlier (older) version, you are strongly encouraged to update to the latest version.
Trusted QSL v1.14 and later includes a feature that will prevent you from uploading duplicate QSOs. It does not query the server, but instead relies on a local database of updated QSOs that it manages on your local computer. Therefore, it's possible that you could export some QSOs from LoTW, sign and upload them, but they are not processed by the LoTW system. You will not be able to upload them again. There may be work-around for this (TBD).
Wait until all of the QSOs have been processed by the LoTW system. You can get an idea by using the "You have ### QSO records" report in the upper right-hand corner of the ARRL LoTW main window. Or you can wait a day. Or check the LoTW Queue Status to see if your uploads have been processed.
Do not save your logbook at this point! Otherwise it will over-write your existing logbook. If you accidentally do this, hopefully you have another copy as a result of step #2 above.
To check if all of the missing QSOs were exported for LoTW, use F8 to search for 01/01/1901 in the "Upload Date" field on the LoTW line in the QSO window. You can also use F8 to search for 'Y' in the LoTW field on the LoTW line in the QSO window to see if QSOs were marked for upload, but were not exported. If you find that QSOs were not exported, go back to step #12 above.
Finally, to see if all your logbook QSOs have been uploaded, repeat starting at step #3 above.
NOTE: It is possible that the number of QSOs in LoTW still does not match the number of QSOs in your DX4WIN log. (TBD)
For more information about working with LoTW, please read this application note.