Global Fishing Watch has launched an improved version (v3.0) of our automatic identification system (AIS) data pipeline. The AIS data pipeline is the series of processes that we use to convert the raw satellite AIS ingested data to the data products, such as machine learning estimates of fishing activity, identity data that combines merged external registry data, and related vessel activity at sea that is then applied by users.
In this new release of the AIS pipeline, we have implemented several improvements to AIS data. While some of these enhancements may not be immediately noticeable to users, they are crucial for the ongoing and frequent improvement of our data quality.
- AIS Segmenter: We have updated one of the key components of the data pipeline: our process to create vessel positions into longer tracks, known as the AIS segmenter. We have worked to remove invalid positions caused by bad satellite timing. While this change may not be immediately obvious, it could result in alterations to the events associated with those AIS segments. You may notice minor changes in a vessel and its activity that you have reviewed previously. We have conducted an extensive review of these data changes and have noted any significant impacts below.
- Anchorage database: The GFW team manually reviewed anchorage labels in 2022, updating over half of the anchorages. This updated dataset was used to recalculate the ‘distance_from_port’ metric. The latest dataset removed inaccurate anchorages near Peru and off the West Coast of Africa.
- Vessel data: We resolved an issue related to vessel characterization and updated our vessel characterization and identity modules within the AIS data pipeline.
- Enhancing stability: We’ve integrated automated quality assurance metrics into the development of AIS data pipeline version 3.0. Furthermore, we’ve enhanced the underlying infrastructure of the data pipeline to ensure greater stability and flexibility in the future.
- The Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- Our public APIs and gfwr (R package)
We will be soon working on updating:
Below, you will find examples and more detailed information for each dataset.
Table of Contents
Examples of data changes
An example of improvements to the AIS segmenter
In the first image, the previous AIS data pipeline segments a track, resulting in no data from June to November 2023:
In the new AIS data pipeline version (3), you can now see the entire vessel track. In addition to the benefits of viewing entire tracks, longer AIS segments allow for cleaner data products, enhancing analysis of fishing and other vessel activities.
An example changes to apparent fishing efforts
In the first image, the previous AIS data pipeline calculated the apparent fishing effort as 1517325 fishing hours:
In the new AIS data pipeline version 3, you can now see around ~ 3% less, see dynamic report below.
Vessel data
In the previous AIS data pipeline version, we identified an error in our vessel class estimates, which had not been functioning correctly since December 2020. These errors have been resolved in the new AIS data pipeline version 3.
What is the impact of those fixes? Apparent fishing hours have increased as follows:
- 2021: +1%
- 2022: +3.4%
- 2023: +10.5%
How did vessel classes change and what vessel classes were we previously missing?
- We have classified an additional 263,627 vessels, of which we classified 29,619 as fishing.
Vessel IDs from the previous AIS data pipeline version are not compatible with those in the new AIS data pipeline version 3.
AIS apparent fishing effort
Some minor changes have been applied to our fishing detection algorithm to improve pipeline stability and reproducibility. The effects are small and for the most part eliminate inconsistencies in the previous pipeline version. As a result, until 2020 the amount of fishing effort by year is 0% to 3% lower compared to the previous pipeline version. Since 2021, the biggest contributor to changes in fishing effort stems from fixing an issue in how we estimate vessel classes after 2020. The bottom line differences of fishing effort in the new pipeline are:
- 2021: +1.6%
- 2022: +3.3%
- 2023: +8.1%
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS fishing events
The filter for vessels considered in apparent fishing events has been updated to align more closely with the filter used for apparent fishing effort.
In the previous AIS data pipeline version, apparent fishing effort was estimated only for vessels identified as fishing, which we referred to as the highest-confidence fishing list. However, we previously displayed fishing events for vessels not only on that highest-confidence list but also for those with lower certainty.
However, in the new AIS data pipeline version 3 and within our products we want to ensure the highest confidence in vessels we estimate to be fishing. Therefore, if a possible vessel is also on the manually reviewed lists of bunker, support or carrier vessels (see definitions), we replace the vessel class with the label from the reviewed list. Lastly, we do not include vessels only reporting on AIS as `fishing` in our product displayed list of fishing vessels.
Event IDs from the previous AIS data pipeline version are not compatible with those in the updated AIS data pipeline version 3.
The impact of this is a large decrease in fishing events, which fluctuates particularly since 2020. Before 2021 the yearly decrease is about 8-9% and in 2022 it is about 6%. In 2021, 2023, and in the current year to date we observe a decrease of about 15%.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS encounter events
In summary, due to core changes in the new data pipeline version 3, we’re now more likely to flag data as “potentially bad,” resulting in a slightly lower number of encounters (around 3% overall).
If you’re interested in more details, these core changes are related to how Global Fishing Watch processes “segments” of vessel tracks, which now generates longer segments for a vessel. This increases the likelihood of overlapping segments, which are removed in the encounters algorithm (see definitions), except for the longest overlapping segment.
Event IDs from the previous AIS data pipeline version are not compatible with those in the updated AIS data pipeline version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS loitering events
There are no major semantic changes between the previous AIS data pipeline version and the new version.
Event IDs from the previous AIS data pipeline version are not compatible with those in the updated AIS data pipeline version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS port visits - Small issue and fix incoming shortly
During our final review of the new pipeline we identified an issue that leads to a small amount of false positives in port visits, which indirectly impacts voyages and coverage. We will apply a fix shortly after the pipeline release at which point we will also provide further information.The issue incorrectly identifies a port visit when a vessel moves for more than 4 hours nearby different anchorages while at the same time not stopping. This can happen when a vessel transits nearby coastal areas with a large number of anchorages, like the Singapore strait. We estimate an increase of 1-3% additional port visits.
We have already implemented a fix that we’re currently testing on all historic data before we will soon release it.
Event IDs from the previous AIS data pipeline version are not compatible with those in the updated AIS data pipeline version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS voyages - Small issue and fix incoming shortly
Voyages are calculated from port visits and are therefore subject to the same issue with port visits that we’re currently fixing. The impact is a small increase of voyages. Once this has been resolved, we will provide further information of the changes of voyages in the new pipeline.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ❌ Public APIs and gfwr (Global Fishing Watch R package)
AIS vessel presence
Our review indicated that there is nearly no change (<1%) across time in vessel presence results in the new AIS pipeline version 3 as compared to the previous pipeline version.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and gfwr (Global Fishing Watch R package)
AIS vessel insights
The vessel insight data is considered prototype stage data (see definition). Learn more about available vessel insights.
AIS coverage - Insight
An issue was identified in the previous AIS data pipeline version that we are still working to resolve in version 3. This issue caused a considerable overestimation of this metric only in recent periods. We will soon release another fix to ensure more accurate values for the most recent period.
Further, coverage is indirectly calculated from port visits and is therefore subject to the same issue with port visits that we’re currently fixing. Once this has been resolved, we will provide further information of the changes of coverage in the new pipeline.
AIS off events - API and insight
There are no changes in AIS off events to report between the previous AIS data pipeline version and the new version 3.
Event IDs from the previous AIS data pipeline version are not compatible with those in the new AIS data pipeline version 3.
Dataset available in:
- ✅ Global Fishing Watch and TMT Vessel Viewer
- ✅ Public APIs and ✅ gfwr (Global Fishing Watch R package)
VMS fishing effort and vessel details
The vessel monitoring system (VMS) is a system for monitoring vessels that is regulated at national and regional levels. VMS data is owned by states, but governments are increasingly sharing this data publicly to increase fisheries transparency.
There are no changes to report between the previous AIS data pipeline version and the new version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ❌ Public APIs and gfwr (Global Fishing Watch R package)
Night light vessel detections (VIIRS)
The night lights vessel detections layer, known as visible infrared imaging radiometer suite or VIIRS, shows vessels at sea that satellites have detected by the light they emit at night.
There are no changes to report between the previous AIS data pipeline version and the new version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ❌ Public APIs and gfwr (Global Fishing Watch R package)
Radar vessel detections (SAR)
Synthetic aperture radar (SAR) can detect vessels and structures at sea in any weather conditions. We utilize SAR imagery to identify vessels in each scene. The only change implemented is the use of the new AIS data pipeline version 3 for the matching logic.
Detection IDs from the previous AIS data pipeline version are not compatible with those in the new AIS data pipeline version 3.
Dataset available in:
- ✅ Global Fishing Watch platform, including map, marine manager portal and vessel viewer
- ✅ Public APIs and ❌ gfwr (Global Fishing Watch R package)
How will the AIS data pipeline version 3 updates affect the data I download from the Data Download Portal?
The Global Fishing Watch data download portal has not yet been updated as part of the release on 1 August 2024. However, not all datasets are affected by this new AIS data pipeline version. Below is the list of datasets that we plan to update and the estimated date:
| Dataset | Updates | Estimated Date |
|---|---|---|
| Vessel detection from Sentinel-1 SAR | The current version is using the previous AIS data pipeline version (a.k.a pipev20231026). With the new update, we won’t update previous vessel detections files, we will start creating new ones using AIS data pipeline version 3 and the files will have this naming convention, for example in Oct 2024 sar_vessel_detections_pipev3_202410.csv | End of Sept 2024 |
| Anchorages | The latest version is from December 2022, we are planning to add more labels and run across all new data from AIS data pipeline version 3. | End of Dec 2024 |
| Carrier Vessel Portal events | The current version uses the previous AIS data pipeline version. With the new update, we won’t update previous files, we will start creating new ones using data pipeline version 3. | End of March 2025 |
| All Vessels Voyages Confidence 4 | The current version uses the previous AIS data pipeline version. With the new update, we won’t update previous files, we will start creating new ones using data pipeline version 3. | End of Dec 2024 |
| Fishing effort dataset | The current version uses the previous AIS data pipeline version. With the new update, we won’t update previous files, we will create new ones using data pipeline version 3. | End of Jan 2024 |
Important note – The previous AIS data pipeline will be updated for 6 months, until 28 February 2025. After that, the data will remain available for 3 years, until 1 March 2028.