Did you know that every record submitted to eBird goes through the eBird data verification process? Using a combination of automated data filters and a network of local experts, eBird tackles the issue of data quality in Citizen-Science. In order for us to maintain the integrity of the database, and for it to be used fully by the science and conservation communities, we as observers must fully understand and strive to reach the highest level of data quality. eBird's procedures facilitate communication between eBird observers and experts, including some new and improved review tools for our editors. Through our combined effort to maintain high data quality, eBird has become one of the most valuable large-scale data sets on bird distribution and abundance in the world.
What do we do when you report birds that are rare and far outside their range of known occurrence? What about records of early/late migrants, or out of season records of lingering birds at local scales? As with any large-scale citizen-science project open to the public, there is the possibility that erroneous data will be submitted. At eBird we consider data quality to be paramount, and we're taking every step possible to ensure that our data are the best they can be. Using advanced data vetting technology, we've developed a combination of automated filters and a network of regional editors that work together to verify eBird data. Each eBird submission, regardless of observer or location, is checked for data quality using the same process.
The data quality process is complex and important to understand, so we hope you read the below article in full, but here is a quick summary of important things to understand:
- Be respectful. Our reviewers aren't paid. They volunteer for an often thankless task that is difficult. Interact with reviewers in a respectful manner and always assume a positive intent. We do not tolerate abusive treatment of our reviewers or others in the eBird community. Abusive language or threats are not tolerated and will result in probation or permanent removal of your account.
- Reviewing records takes time. We have 500+ volunteer reviewers who devote many hours of personal time to this cause. These reviewers have families, jobs, and also enjoy going birding themselves. They participate because they share our belief in the purpose and goals of eBird. Please treat them kindly and respectfully. They do their best to act with dignity and they offer their expertise freely, often handling challenging situations with skill.
- Review is not always chronological. Our review team may work on different chunks of data at different times. Sometimes records may be dealt with chronologically, but at other times a reviewers may tackle all records of a certain species, from a certain location, or tackle just those with photos or just those with notes. This may make the process appear arbitrary and sometimes it may be true that your report has yet to be addressed while more recent reports from others are already showing as accepted. Please be patient and understand that these apparent inconsistencies may relate only to how the pending records were sorted by a reviewer.
- Records awaiting a decision are not shown on maps and graphs. There is a difference between "not accepted" and "awaiting review", but neither are shown on maps and graphs. Oftentimes a record will not be shown in eBird output. When you see "Unconfirmed" in eBird Alerts or smartphone apps, this simply means that the reviewer has yet to act on the record. Patience is the best policy.
- Please do not be offended. Being questioned about a record is not personal. It is simply a request for more information so that your record will hold up to scientific scrutiny. Some submitted records are typos, geographical errors, date errors, the result of taxonomic confusion (e.g., Common Snipe vs. Wilson's Snipe), or another simple error.
- Documenting rarities is up to you. The best way to ensure acceptance of a rarity that you observe is to provide unequivocal proof. An embedded photo is ideal, but links to photos or a description of the key field marks is also enough in many cases. Adding details to your noteworthy records is key for expedient review.
- Non-public records. Reviewers cannot change, edit, or delete records and anything you submit will always be in your personal account (and appear on your checklists) unless you change it. Some records are treated as "unconfirmed", which means they are not shown on maps, graphs, charts and other data output. No record is ever deleted from the database by reviewers, and all records are available for reassessment.
- Please trust us. Ensuring data quality for millions of records per month is a huge task. We are constantly working towards more refined and speedier review, but sometimes it can takes days or even weeks before a record is acted upon. Rare bird sightings are shown in a timely fashion via eBird Alerts; your other sightings are a benefit to eBird even if the review is delayed and even if a record is flagged as "unconfirmed".
- Go birding. Please don't stress over whether a record is being acted on. Much better to go birding and find some more unusual birds for us! We'll get to them all in time and your best contribution is to keep submitting observations.
eBird filters and how they work
eBird filters are the lynchpin of the eBird review process. The filters provide an automated check on every record submitted to eBird.
The filters are a matrix that checks the species, count, location, and date for every observation. Anything that exceeds expected totals for a given species at a location on a given date, will be 'flagged' for review. Filter thresholds for species that have occurred in the area but are very rare are set at zero, so that any record will be reviewed. These species are marked "Rare" on the data entry checklist and can be exposed or hidden by toggling the "Show rare species" checkbox.
Fig. 1. Example of an eBird checklist filter from Lubbock Co., Texas. The species set at zero (e.g., Violet-green Swallow) would be flagged on any date, but Barn Swallow would only be flagged in winter or when the counts exceed the numbers on the filter (e.g., 105 would be flagged on August 25).
The eBird filters are assigned to certain geographic areas, and eBird has thousands of such filters in place around the world. In many places in the USA, Canada, and United Kingdom we have very fine-scale (county-level) filters and some places have even finer scale filters (e.g., IBA, bioregion within a county). In many other countries (e.g., Mexico, Guatemala, Australia, etc.) we have state-level filters. In some areas (e.g., Africa) we still have broad-scale checklists, but we are moving to make the filters more site-specific and accurate.
As soon as you define the date and location of your list the eBird filters start to work. The data entry checklist is always customized by date and location so that you only see species appropriate for the area you went birding and the date of your observation. This makes data entry quicker and easier (imagine trying to navigate a list of 10,500 bird species!). The checklist is also customized so that rarities are not shown. It is simple to add them to the checklist using "add a species" (right under the "jump to species" box) or by clicking "show rare species". If you saw something that is definitely not on the list, the "add a species" box allows you to add it (but you will be asked to enter notes to confirm the sighting!).
Once you have filled out your checklist and clicked continue, the filter goes to work again, providing the first-level of data validation. Any rarity or count you submit that exceeds the filter values will return an eBird message asking for documentation to support the record. For example, a Shoebill would trigger such an alert in many places:
The messages come in two forms: either the species is unusual or the count is unusually high. Watch for these messages to let you know if the record is being questioned because the species is unexpected, or because the count is higher than expected.
Anytime a record gets flagged, please stop and double-check. Do you know why this record was flagged? If so, and if you are certain of the identification, please add appropriate details and continue. If you don't know why it was flagged, please double check the date, location, and species. Did you make a typo? If traveling internationally, are you following the correct taxonomy? If you are sure, please do enter as many details as you can to support the identification. IMPORTANT!--DO NOT take these messages as an indication to remove a species you know you saw, or to lower a count that eBird has flagged for review. Instead, take the opportunity to provide the required details to support your observation, such as notes on the species' identification or an explanation of how you made the high count and the circumstances of the observation.
Records are flagged for one of the following reasons:
Rarity - a species that is rare or unusual in the region
Out of season report - a species is reported outside its normal date range in a region
High count - species count exceeds what one might expect to find in a typical day's birding in the region on a particular date
Not on filter - any species you add to the checklist using "Add a species" (you can add any species worldwide to any eBird checklist -- be careful!)
Once flagged, a record is withheld from the public view of the database until it can be confirmed by our expert reviewers. The flagged records are placed in a separate series of tools where they can be reviewed in detail by our reviewers.
eBird reviewers and what they do
eBird reviewers are set up for certain regions where they can review records and change the filters. When eBird reviewers travel outside their region, their records are reviewed by the same process as everyone else's. Our reviewers often have a long history of birding in their region. We try to select reviewers that are enthusiastic, work hard, and are kind people with good communication skills. It bears stressing that this is a corps of volunteers and we are incredibly grateful for the hard work they do. eBird would not exist without this team.
Each reviewer has login credentials that allow them to see all flagged records for their region in one place. Records are treated as "not valid, unreviewed" (see the red type at right in Fig. 3) until acted upon, at which point they are considered "reviewed". Reviewers have the option of marking records as unconfirmed or confirmed, and have all the information from each submission available: species, count, date, location, and observer are all shown and the full checklist (and written details) can be seen as well. By clicking the "mail" icon, they can quickly send you an email request for more information.
Fig. 3. Example of the review list for Vietnam. This is the list that reviewers use to make decisions. Note that Sri Lanka Green-Pigeon has been entered erroneously presumably because the species has been split and the current taxa are confusing. Reviewers follow up on rarities, check for location accuracy, correspond about typos and taxonomy errors like this, and much more. The little mail icon sends the review email.
Sometimes a reviewer has 10 minutes to spare and will review a handful of recent records. Sorting the list can help to identify records in different ways. We know users are sometimes confused about why some records were accepted quickly but theirs took more time. Perhaps the reviewer sorted taxonomically and acted on everything up to Barn Swallow, but not on your bird. Perhaps it was a sort by date and acted on everything but yours before being called away for dinner or work. This list never goes away until a decision is reached though, so the reviewer is sure to return and address your bird, even if it takes a while.
Another part of each reviewer's responsibility is maintaining the filters in their region. New filters are put in regularly and sometimes an error on the filter is corrected and the data are re-run. This can result in records from 10 years ago suddenly getting flagged for review. This is an important part of the ongoing process, and while we understand that it may be hard to provide details many years after the fact, we also should point out that the bird is no less rare (so should have been documented at the time!).
We have a long document (40+ pages!) that gives guidelines for reviewers for how to deal with records, how to maintain the filters, and how we would like them to correspond with the eBird community. We also have an online forum where Regional Editors can discuss the process and how to better serve the needs of eBird and its users. These people believe strongly in the project, as do you, so please consider that we are all part of the same team.
eBird Data Verification Email
The ability to quickly email eBird users is very important for our editors who must deal with tens or hundreds of records in a single sitting. Please be sure to read the beginning of the email, since reviewers often put short notes there about the significance of the record or specific questions (which may relate to the location or date, and not to the identification). We know form letters can sometimes be viewed negatively, but in order for our volunteers to be able to review so many records, an automated process is required to initiate discussion.
Here is an example of an automated email that regional editors might send to you regarding one of your records:
Observation date: Mar 17, 2013
Location: stakeout Fieldfare Greenough Land, Carlisle (2013), Middlesex, US-MA
Submission ID: S13429505
In your description, it is often very helpful to include how similar species were eliminated. If you saw and remember relevant field marks please take note of those and tell me about them. If you identified the bird by voice, a description of this is often helpful. The best supporting evidence for any bird record is a photograph, which you can simply attach in an email response to me, or embed directly into your eBird checklist (click http://help.ebird.org/customer/portal/articles/973966-adding-photos-videos-and-recordings-to-checklists for details on that process). If you do embed a photo, please drop me a note to let me know that you did.
If you were not sure about the identification, please let me know. Without sufficient supporting details, the sighting will be left as unconfirmed and therefore not included in the output for scientific analysis, but your personal records and lists will be kept intact. If you do wish to edit the observation, you can do so from the checklist link above (just click 'edit species' on that page and change it as needed).
We strive for a high level of data quality at eBird not only because eBird is a scientific database, but also because we want to provide the best resources for the birding community. Birders use eBird every day to learn about bird distribution and occurrence, and your data are a valuable part of that process. If you'd like to learn more about the eBird data quality process click here: (http://help.ebird.org/customer/portal/articles/1055676-understanding-the-ebird-review-and-data-quality-process)
Thanks for your understanding and for any help that you can provide. And most importantly, thank you for using eBird. Your participation is valued, and the observations you submit are an important part of helping us better understand birds and biodiversity in your area and around the world.
Why worry about data quality?
eBird is increasingly being used by scientists (hundreds of data downloads/month) and conservation, so it is important to provide those communities with an accurate dataset. The eBird review process intentionally errs on the conservative side, so that we are more likely to treat a possibly correct record as not accepted than we are to treat a possibly incorrect record as accepted. This means that the maps and graphs in eBird can be trusted to be correct and that we can vouch for any record in the system. If anything, the current maps and graphs under-represent the ranges and seasonal patterns of birds, and it should be very rare to have over-estimations in the system. If a less conservative approach were taken, it would be much harder to know which records can be trusted. We recognize that this may be disappointing to some users who have their correctly-identified birds not included, but we hope that the larger scientific goals can be understood and appreciated by all. If nothing else, when hoping to answer questions about bird occurrence, we are all happy to get accurate data back.
eBird is accessible to anyone around the world and as a result has a user community that runs the gamut from rank beginner to expert ornithologist. Your eBird reviewer has no easy way to know if a submitted record is coming from someone submitting their first checklist and inexperienced with the challenges of bird identification, or a professional birding tour leader who has been studying birds for decades. Reviewers make an effort to treat everyone equally and to assess records based on the quality of the description of the bird and to assess records objectively. It is certainly true that individuals that demonstrate care and caution in their identifications and respond promptly and helpfully will quickly earn reputations as skilled eBirders who respect the process. Those that do not respond to queries, do not enter comments, or do not participate in the process in a constructive way may find that fewer of their records are accepted. We want eBird to be accessible to everyone, but typographical errors, upload errors, misidentifications, and taxonomic confusions account for tens of thousands of mistakes annually. Please understand that the review process is very important and works very well when the whole eBird team -- from users to reviewers --participates in the process and tries our collective best. eBird is truly a global team of tens of thousands of bird-lovers, so thanks for playing your part!
We recognize that your records are your own, and reviewers cannot truly know what you saw. Reviewers can only decide whether the evidence to support the record is strong enough to withstand objective scrutiny 100 years from now, when all of our personal experience will long since be forgotten!
Recent revisions (summer 2012)
Prior to 2012, the eBird filters were specific only to a given month. In North America, many eBirders may remember the days when an expected spring migrant would be flagged as rare on 30 April but not on 1 May. This was because reviewers had the difficult decision of whether to consider that migrant "rare" or "expected" in April, even if it was rare before 25 April and expected thereafter.
In 2012 we adapted to the daily filters seen in Fig. 1. This means that our rare bird flags are much more accurate and allow us to set arrival dates that better match bird biology.
Along with this, we made a number of revisions to make the job of our editors easier. These revisions continue to pay dividends as more records are getting reviewed more quickly than ever before.
How can I document my birds better? eBird has several stories to help with information on how to document your birds. The best source is the "Reporting Rarites--Elements of a Bird Description". Our "How to use the eBird comments" fields gives advice on using written comments to better document birds for reviewers. Our story on "Embedding rich media" gives instructions for how to embed links to photos and videos in your checklist.
How can I tell if a record has been reviewed? In general, if you can find it on the eBird point maps, it is being treated as a "public" record. If you cannot, it may be: 1) blocked by another pin (be sure to look in satellite mode, where you can zoom in farther); 2) still awaiting review; 3) reviewed and treated as "not public". Note that there are several reasons to treat a record as not public that do not relate to the ID of the bird, including imprecise location. Plotting your location correctly is of paramount importance (See more on location specificity).
How can I dispute a review decision? In general this is not possible, but any record is open to re-review. Amending and adding to your field notes after the fact is not good practice since you should have given a complete description at the time, but if you have photos that you did not provide before, please embed those and let your reviewer know. If you know of others who saw the bird, please put your reviewer in touch with them.
Do reviewers have to send me the email? We strongly recommend that reviewers give everyone a chance to defend their records and that reviewers send everyone an email. If you consistently respond to review requests with a positive demeanor, you are sure to always be given a chance to provide extra information for rarities.
Does eBird follow records committee decisions? Our reviewer guidelines recommend following Records Committees for species on the review list for a given area. We do recommend that eBird editors make preliminary judgments for rarities in real time, since eBird review proceeds much faster than most records committees.
Why is this species flagged in my area? Our reviewers use a number of considerations to set the common/rare species lists. Sometimes regular species that are hard to identify, or species that are only found at one or two sites, will be flagged so that potential errors are more easily caught. Your records from expected places will be easily processed.
Birds have wings, so anything can happen, right? We do hear this defense at times for reported rarities. As of 2015, we have more than a century of careful bird study in many parts of the world. We have a good sense for what birds are prone to long-distance vagrancy and which ones essentially never move. For example, South American antbirds and antpittas tend not to stray from their home ranges, while sandpipers, swallows, and gulls often turn up thousands of miles away. Spring migrants often have established very punctual arrival schedules (for example, Willow Flycatcher has never been documented in the U.S. and Canada in March), so an early arrival might be more unlikely than a far-flung vagrant from the opposite coast. Please listen to our reviewers; it is not true that "anything is possible" with birds, and some things are far more probable than others. Everyone makes mistakes (eBird project leaders included).
Why were expected subspecies or spuhs flagged? Since the eBird filters also define the checklist entry list, editors sometimes must strike a balance between showing the most often used options and every conceivable option for the area. eBird allows entry of lots of subspecies including obscure things such as Mallard (Northern), and Osprey (American). These are not always worth showing on the checklist filters, so they may get flagged as a "rare bird", even if the subspecies is perfectly expected. Likewise for some rarely used "spuh" options, such as Blackpoll/Bay-breasted Warbler or woodpecker sp. These are not really rarities, but since they are not shown on the list they may get flagged. Don't worry, these will always be easily accepted. If you have a "spuh", a hybrid, or slash combo that you feel should be added to the filter, try writing "Please add to filter" in the species comments along with your observation. Editors see these comments, so you are giving them information about what additions are needed on the regional checklists.
Fig. 5. Embedded photos, like the one of this Fieldfare in Massachusetts, can be seen by reviewers and simplify the review and acceptance of rare species considerably. Please see the first FAQ above for instructions on how to embed photos. Thanks to Steve Mirick for his checklist with images of this exciting bird.
As eBird continues to grow, data quality will remain a challenge, and we thank you for your participation and understanding of the process. We believe that eBird reviewers are not only helping with scientific accuracy, but providing a valuable service to birders to help keep their data accurate, especially the instant feedback during data entry and subsequent direct communication.
A few closing thoughts bear repeating:
- Please don't stress about review. Our editors are doing the best they can. Time lags can occur. Your records are certainly appreciated, and they will be reviewed as soon as possible.
- Provide the best documentation you can. Providing good notes and photos is the best way to have your records shown publicly. Make sure that your documentary notes include information on field marks and how similar species were eliminated. Notes that say "Seen in treetop then flew away" really do not add much to help support the record. See the FAQ above for how to better document records for links that help improve your documentation habits.
- Nothing is final. If a record is treated as unconfirmed that doesn't mean it isn't valuable. As the status of birds change, your record may be reassessed.
Given today's fast-changing landscape and the wide variety of anthropogenic factors affecting bird populations (e.g., global climate change), now more than ever we need the help of birders in monitoring birds. These factors are certainly increasing the rate at which changes in bird distribution and abundance are taking place. Only through large-scale citizen-science efforts like eBird will we be able to fully understand these changes globally. With that we ask you to participate in eBird, to understand that every observation you make is valuable, to realize that data quality is also important, and to bear with us as we continue to help narrow the gap between birding and science!
Research and development of the eBird Review Process, especially the recent update to an advanced emergent filter process, is funded in part by the National Science Foundation.