In particular, the contrast between TV advertising -- where advertisers know exactly what content their ads are playing during -- and digital video is stark.
"It's really the introduction of video that has made brand safety top of mind for advertisers," Ben Winkler, chief investment officer for advertising agency OMD, told CNBC.
"TV isn't just about video. It's about ultra-high quality content and context. Digital can't quite deliver on that yet. Advertisers expect a higher level of brand safety from video, especially if they're expected to spend TV money there."
This could turn into a significant problem for Google and Facebook, which together will account for about 60 percent of the $83 billion U.S. digital ad budget in 2017, according to eMarketer.
"Appearing next to bad or unwanted content has always been a problem, especially as ad networks and exchanges (ad tech platforms that sell ads) grew and as more and more websites emerged," Matt Borchard, media director at media agency Noble People, told CNBC.
"The sheer size of the Internet means there is a small fraction that is premium, a lot that is mediocre, and a small amount that is truly awful, especially for advertisers. This isn't a Google issue, it's a digital network/exchange/inventory issue."
Borchard previously said adjacency is also a problem with Facebook videos, as companies do not have much control over what content their ads appear in-between or on.
But OMD's Winkler argues the ad adjacency problem is more pronounced for YouTube than Facebook because ads run in between back-to-back videos.
OMD has found that people associate commercials with content when they appear in a linear format, meaning a bad video could taint a company's image. However, people don't associate negative posts in Facebook's news feed with nearby ads. For example, a person may see a negative statement from a friend, but understands that the next post in their feed -- which might be an advertisement -- has no association with that statement.
In Noble People's experience, some money is moving out of digital video and digital display advertising, but it's not going back to TV. "Instead, it's usually distributed across a variety of channels: video on demand, streaming services like Hulu, audio ads like radio, streaming radio and podcasts, outdoor or print," said Brochard.
Another source at a big buyer disagrees and says there's no significant outflow of money from digital right now, but advertisers are starting to press for better solutions.
Meanwhile, big publishers stand the most to gain if advertisers begin to have doubts about Google and Facebook, so it's no surprise they're jumping on board.
Robert Thompson, the CEO of Times of London publisher News Corp, condemned the whole digital ad industry in a statement sent to CNBC today:
"Ad fraud is being perpetrated ad nauseam. It is rife throughout the digital world, and is facilitated by some ad agencies, which themselves make money from artificial audiences and pretend page views," said Thompson. "Advertisers need to go back to basics to protect their brands from serious damage and to protect themselves from being involved in potentially criminal activity, whether it be supporting extremist groups or funding hardcore pornography."