braedon.dev

Mastodon Isn't Twitter

2022-11-11T06:00:00+08:00

I’ve spent the last couple of years being very on Twitter. Along with spending too much time scrolling/reading, I write there frequently and at length. It’s where I publish most of my adtech/disinformation research, and where I talk about my software projects.

Like a lot of Twitter users recently, I’ve been trying to work out if Mastodon is an alternative if/when it’s no longer viable to stay on Twitter. I have my doubts.

To understand why, we first need to look at why the bird site had such a draw with a certain group of people — what set it apart from the various other types of mass communication.

Twitter

While there are undoubtedly many factors in play, a key part of Twitter’s appeal is that tweets have unlimited potential reach, and are persistent.

Tweets are entirely public. On Twitter, you don’t really post to your friends, family, or any other community. You’re tweeting to everyone, at least aspirationally. Retweets, replies, and quote tweets all help spread a tweet to a wider audience, and are themselves tweets that can be spread.

Perhaps more importantly, tweets spread far beyond Twitter itself. They can be viewed by anyone without an account, linked to from anywhere, and they’re easy to embed or screenshot. The ability of a tweet to reach the wider world, and the frequency at which that happens, is core to Twitter’s appeal.

Tweets have a short half-life in the Twitter feed, but remain accessible via links and embeds, and can be readily found via searches. Retweets, replies, and quote tweets push them back into the feed and start the spread again. This persistence preserves the value of the work that goes into creating tweets — particularly long threads.

Reach and persistence are why Twitter is the social network for brands, politicians, and journalists; why its influence has always been outsized for its user base. And they’re why I’m there, building a user-base for a niche research tool, and sharing screenshots of brands advertising on disinformation.

Twitter is where you go if you want your ideas to spread as widely as possible, and to be able to refer back to them in the future.

Clearly, this is not what everyone on Twitter is looking for. And these dynamics cause a lot of issues — it’s often called “the hellsite” for a reason. But I do think these factors are key to understanding Twitter’s niche, and whether any potential replacement can fill it.

Mastodon

Superficially, Mastodon looks a lot like Twitter, just open source and federated. But it has a significant number of differences, many of which are directly at odds with Twitter’s recipe of reach and persistence.

One of the first things a Twitter user will notice is the lack of quote tweets. This is an explicit decision to prevent quote-dunking, an oft-lamented part of Twitter culture.

But while quote tweets can be used to dunk, they can also be used to reference old tweets (your own, or others’) as context for a new tweet, or just to take a conversation in a slightly different direction without hijacking the original one. They’re key to how ideas spread, expand, and evolve across Twitter (and beyond).

Another major departure is searching for posts. Most Mastodon servers only allow searching by hashtag — full-text search just isn’t provided. An external search service was recently created and then shut down almost immediately due to community backlash.

Once again, this is a deliberate choice — harassers often find targets by searching for specific key words or phrases. But it also makes it hard to find old posts you want to reference (even your own) — once they’re out of your feed, they’re largely gone.

Automatic post deletion is a built-in feature, and used by a lot of people. When you migrate your profile to a new server you can bring your followers, but not your posts. Long-term post persistence is clearly not the goal here.

Mastodon servers are designed to be distinct communities, with their own rules and norms. While you can follow people from other servers, it’s highly recommended you find a server that matches your interests. Mastodon doesn’t have a recommendation algorithm, so discovery is largely facilitated by your server’s local timeline — a feed of all the posts from users on your server.

You’re not posting to everyone, aiming for maximum reach — you’re talking to your local community. Trying to talk to followers on other servers spams your community with posts they don’t care about.

Reach is further curtailed when server admins block entire other servers, preventing users from following each other. While there are often obvious reasons for this — there are some toxic servers out there — other cases are due to more complex cultural issues.

For example, a new server for journalists is apparently being blocked by a number of servers. I expect these kinds of culture clashes to be a problem for a lot of Twitter users trying to migrate, not to mention the existing Mastodon communities being impacted.

Overall, a Mastodon server seems a lot closer to a forum, or even something like a Discord server, than Twitter. Posts are essentially transient, and work best when you’re talking to your local community.

I have no doubt many Twexiters will find and create great communities on Mastodon, but I’m not convinced it can slot into the niche Twitter has filled for so long.

Ads.txt OwnerDomain and ManagerDomain

2022-04-27T20:00:00+08:00

IAB Tech Lab recently released a new version of the ads.txt standard for comment. This 1.1 update defines two new variables — OWNERDOMAIN and MANAGERDOMAIN — that attempt to solve some outstanding issues with how seller relationships are expressed.

While these new variables provide useful extra data points, I’m concerned they won’t achieve the wide adoption necessary to make a meaningful difference, and that some of the additions are moving the standard in the wrong direction.

OwnerDomain

The definition of the OWNERDOMAIN variable from the updated ads.txt spec.

OWNERDOMAIN allows a site to declare who owns it by specifying the owner’s “business domain”. While this can just be the site domain, it’s often different, particularly when a company owns multiple different sites. For example, Gizmodo could add OWNERDOMAIN=g-omedia.com to their ads.txt, indicating they’re owned by G/O Media.

Some of the PUBLISHER ad accounts authorized as DIRECT sellers by Gizmodo.

A site’s OWNERDOMAIN can be compared with an ad account’s domain from its sellers.json entry to determine if they have the same owner. This will be helpful in checking if a PUBLISHER account is actually a valid seller for a particular site, for example.

A related use case is determining whether an account labelled BOTH in sellers.json is acting as a PUBLISHER or INTERMEDIARY on a given site. If the site’s OWNERDOMAIN matches the account’s domain, it’s acting as a PUBLISHER; otherwise it’s an INTERMEDIARY.

Intuitively, OWNERDOMAIN should also help in checking if an account is actually DIRECT. However, this isn’t mentioned by the spec (though there is a brief mention in the “Implementation Guide”), and the definition of DIRECT is a fraught topic.

ManagerDomain

The definition of the MANAGERDOMAIN variable from the updated ads.txt spec.

MANAGERDOMAIN is similar, but rather than declaring the site’s owner, it declares the company exclusively responsible for managing the site’s ad inventory (if there is one). Different companies can be declared per country if needed.

For example, meetup.com’s ad inventory is managed by Freestar. They could add MANAGERDOMAIN=freestar.com to their ads.txt to indicate this relationship.

Part of a graph of meetup.com’s authorized supply chains.

A management company’s ad accounts can be identified by comparing account domains to the manager’s domain. As an exclusive manager’s accounts should be the most direct sources of the site’s inventory, this can help with Supply Path Optimisation (SPO).

Note that the spec falls short of saying the manager’s accounts can be labelled DIRECT, however, and they are labelled RESELLER in the implementation guide’s examples.

Adoption and Enforcement

These new variables are only useful in preventing ad fraud if DSPs actually enforce them, e.g. by not buying inventory via PUBLISHER accounts that don’t match a site’s OWNERDOMAIN. But DSPs aren’t going to turn off a large proportion of their supply, so wide adoption is required first.

So what’s the incentive for sites to adopt the new variables if DSPs aren’t using them? DSPs could add validations that only run on sites that provide them, giving some reason for their existence without overly affecting supply. However, this effectively just adds new restrictions to sites that adopt the variables. Sites that are benefiting from the current paucity of validation effectively have an incentive to not add them.

Usually, the ability to say you comply with a spec provides some incentive in itself. However, the spec doesn’t actually require either variable. OWNERDOMAIN is “recommended” and “should” be checked against the domains of PUBLISHER accounts. MANAGERDOMAIN is “optional” and “should” be provided when a manager is used. A site that should provide them can simply not, and still claim they are following the spec. Similarly, a DSP can claim to support the ads.txt spec, but ignore these variables entirely.

Even if a DSP wants to break from the pack and start enforcing them, they aren’t standalone. For example, they need to be compared to account domains, but the account domain field is optional in the sellers.json spec. Currently, more accounts are missing a domain than provide one. Enforcing OWNERDOMAIN and MANAGERDOMAIN requires first enforcing account domains, and that hasn’t happened in the almost 3 years since sellers.json was released.

MANAGERDOMAIN isn’t really enforceable per se — it’s essentially just a SPO hint. The question is more if DSPs will actually meaningful change their buying decisions based on it. SupplyChain objects were released years ago alongside sellers.json, and there’s been little evidence they are being widely used for SPO, or even validated.

It’s also important to note that OWNERDOMAIN and MANAGERDOMAIN are set by sites, and there’s no programmatic way for DSPs to check they’re actually correct. I can think of a few ways this could be exploited by less than virtuous sites (though determining what would work in practice is always difficult).

Owner vs. Site

The definition of OWNERDOMAIN states that it should match the domain of any PUBLISHER accounts — i.e. account domains in sellers.json should be the owner domain, not the site domain. This isn’t a change — it’s essentially what the sellers.json spec already says, though it often isn’t followed currently (see the Gizmodo example above).

However, I’m concerned that this doubles down on the idea that SupplyChain objects should start at the site’s owner, not the site itself. This means that different sites with the same owner can have identical supply chains.

This is a complex issue — see my posts on ads.txt and sellers.json for more details — but the core problem is sites that share supply chains can spoof each other’s ad inventory at will.

When I first wrote about this issue it was just a theoretical possibility, but then I found Gannett doing it (accidentally) in the wild. If they were using SupplyChains that started at the site level, the bug could have been caught by basic SupplyChain validations. Instead, all their sites shared the same supply chains, and no one noticed the issue for 9 months.

Requiring SupplyChains to start at the site level wouldn’t make this kind of authorized domain spoofing impossible — the SupplyChain itself could be spoofed by a malicious actor (or a particularly unfortunate bug) — but that’s a problem with these specs in general, not something specific to this issue.

What makes ads.txt, sellers.json, and SupplyChain objects (potentially) useful is the ability to cross-check data from different sources. A SupplyChain can be checked against other fields provided in the bid request (which are often pulled from different places), the sellers.json entries provided by the ad systems, and the ads.txt file provided by the site.

Not requiring SupplyChains to start at the site level eliminates opportunities to validate the link between the site and the first seller account, and will result in less issues being detected.

Final Thoughts

This ads.txt update only introduces two relatively simple variables, but they slot into a complex web of existing standards, and their often less than stellar implementations. I’ve only touched on what I think are the most important points, but there’s plenty more there to dig into.

One aspect I’ve avoided for brevity (and because I’m not sure how to say it nicely — sorry) is a number of issues that basically boil down to a need for significant copy editing. Key sentences that don’t really make sense, incorrect field names, new terms that seemingly duplicate existing ones, paragraphs that are replicated from elsewhere but with differences, etc. I’ve tried to avoid being pedantic in this post, but if pedantry is ever required it’s when you’re writing specifications.

OWNERDOMAIN patches one of the holes in validating DIRECT PUBLISHER accounts, but others remain. MANAGERDOMAIN helps break the DIRECT/RESELLER binary, if DSPs will use it. Adoption and enforcement will be key, but that’s exactly where the existing specs have faltered. Unfortunately, I don’t see anything in this update that would fundamentally change that.

Finally, ads are displayed on sites. Ad inventory supply chains should start at the site.

Domain Spoofing on Gannett Sites

2022-03-09T08:17:00+08:00

This research was done in collaboration with Krzysztof Franaszek. Read his post here.

Domain spoofing — where ad inventory is misrepresented as being from a different site — is often talked about as a solved problem by adtech insiders. Despite this, USA Today and hundreds of local newspapers owned by Gannett were sending spoofed bid requests to multiple ad exchanges for over 9 months.

The various companies involved didn’t appear to notice this behaviour — or didn’t act to resolve it if they did — until late last week when it suddenly stopped.

A tweet from a senior engineer at a major DSP.

The primary weapon against domain spoofing is the ads.txt standard from IAB Tech Lab. This allows a site to authorise specific ad accounts to sell its inventory, preventing other accounts from spoofing it. But ads.txt has its limitations, and one is that it only protects against domain spoofing by unauthorized accounts. What if the spoofing is coming from inside the house?

Gannett owns a large network of newspaper sites that share a set of DIRECT ad accounts — everything from USA Today to the Wisconsin Rapids Tribune. As they’re the “publisher” for all these sites, technically they’re allowed to do this — the accounts aren’t considered mislabelled.

However, the shared accounts are authorized to sell inventory for all the sites, so they can theoretically also spoof inventory for any of the sites without failing ads.txt validations. This kind of “authorized spoofing” is exactly what appeared to be happening until Friday the 4th of March 2022.

I’m not going to speculate as to whether this spoofing was deliberate on Gannett’s part, or if they derived any benefit from it. Instead, I want to focus on how the spoofing worked, and the implications of it.

Finding the Spoofing

Gannett sites use header bidding to run ad auctions on different exchanges when you load a page. Scripts running in your browser make the requests that trigger those auctions, so it’s possible to inspect the requests your browser sends to see how Gannett is representing their ad inventory. In September 2021 I noticed that the domains and page URLs included in some of those requests didn’t seem to match the actual page being loaded.

Let’s look at the Detroit Free Press, a local newspaper owned by Gannett. I loaded this local news article about mistreated dogs on the 17th of February 2022.

The actual article on the Detroit Free Press.

The spoofed article on USA Today.

Header bidding requests to multiple ad exchanges, such as Pubmatic and IndexExchange, reported the page as a USA Today college football article. That looks a whole lot like domain spoofing.

A header bidding request to Pubmatic.

A header bidding request to IndexExchange.

A search of the page’s source code shows where the spoofed URL is coming from — a minified inline script. With the help of a javascript formatter we can more easily see the offending code.

Source code of the actual article.

Part of the inline script, formatted.

pbjs stands for Prebid.js, a JavaScript library that’s commonly used to implement header bidding. The a.setConfig({...}) method call is used to configure Prebid, and the ortb2 configuration field provides data about the inventory being sold in OpenRTB 2.5 format.

In OpenRTB, the site field should contain data about the page the ad will be shown on. So why was Gannett providing data about a totally different page?

Further, it wasn’t only the domain and page being spoofed. Data about the article section and subsection, keywords, and brand safety was also being provided based on the spoofed page, not the actual page. This spoofed data was all showing up in requests to affected exchanges.

Definition of fields in the site object from the OpenRTB 2.5 spec.

The mapping between actual pages and spoofed pages in the Prebid config seemed to be cached. Loading a given page multiple times within a short period, or from different browsers or computers, usually yielded the same spoofed page. The mappings did eventually change after some period of time — presumably after the page expired from the cache.

It is important to note that the spoofed data didn’t show up in requests to all exchanges. While some exchanges were only getting spoofed page data, others got actual page data, or a mixture of the two.

That’s because each exchange has its own Prebid adapter that constructs requests from available data. It’s up to these adapters to decide how to use the ortb2 field from the Prebid config, and some didn’t end up incorporating the spoofed data.

Similarly, exchanges that were sent spoofed data may not have passed it on to advertisers. To test this, Krzysztof Franaszek of Adalytics examined ads served in response to header bidding requests that included spoofed data. He found evidence suggesting advertisers were bidding on spoofed ad inventory via multiple exchanges.

Scoping the Problem

Based on archived versions of the USA Today homepage on The Internet Archive’s Wayback Machine, the ortb2 field was first added to the Prebid config sometime between 11:15am and 5:22pm EDT on the 25th of May 2021.

That first observed ortb2 field contained data for the article “Arizona has seen the worst of its winter weather — for now” from Canton Repository — i.e. the data was spoofed right from the start.

The USA Today homepage started consistently including correct data in the ortb2 field sometime between 9:16am and 2:37pm EST on the 4th of March 2022.

In the months since I noticed the spoofing I’ve run dozens of scans checking thousands of articles on Gannett sites to see how widespread the issue was. These scans excluded paywalled articles and a small number of special features that have different ad setups, but all public “regular” articles — ones where the path starts with /story/ — were candidates.

The last scan completed before the spoofing stopped was run on the 3rd of March 2022, and checked articles listed on the homepages of Gannett sites. Of 6,983 articles checked across 275 sites, the domain in the Prebid config was different to the actual domain 99.46% of the time. The page URL in the Prebid config was different to the actual page URL 99.99% of the time. Only a single page wasn’t spoofed.

These results are consistent with the many other scans I ran. The rare cases where a page wasn’t spoofed were so infrequent as to likely be flukes — i.e. the spoofed page happened to match the actual page.

There didn’t appear to be any relationship between the actual and spoofed pages. However, there was a clear pattern to which domains were spoofed. In this scan, spoofed pages were from USA Today 19.05% of the time. The next most common domain was the AZ Central at 2.36%, with other domains decreasing from there.

Here’s the scan’s top 10 spoofed domains, how many of their articles were checked, how often they were spoofed, and their spoof frequency:

Domain	Actual	Spoofed	Spoof Freq.
usatoday.com	48	1330	19.05%
azcentral.com	29	165	2.36%
desmoinesregister.com	30	125	1.79%
heraldtribune.com	20	101	1.45%
delawareonline.com	164	92	1.32%
app.com	26	88	1.26%
theledger.com	23	81	1.16%
freep.com	25	78	1.12%
seacoastonline.com	21	76	1.09%
dispatch.com	49	73	1.05%

Once again, these results are consistent with other scans done on other days, including ones using different methods of sampling articles, e.g. selecting up to 25 articles from each day of a month for each site, or all articles from a single site for a whole year.

USA Today was always the most frequently spoofed domain by a wide margin. It was typically spoofed on 20% of checked articles, plus or minus a few percentage points.

The Impact

Gannett’s ad inventory was being misrepresented to advertisers using affected exchanges for over 9 months.

This might not be a concern for programmatic campaigns that purely target users, but most advertisers do care about the sites and content their ads end up on to some degree or another.

If you thought you had bought ad space on USA Today, would you be ok with your ad actually displaying on Detroit Free Press? As of writing, USA Today is ranked #189 on the Tranco list of top sites. Detroit Free Press is #1,983. How about another Gannett site that was spoofed a handful of times in each scan, The Galesburg Register Mail? It’s #98,198. A significant proportion of Gannett’s sites don’t even make the top 100,000.

Is ad space on an article about animal neglect in northern Michigan the same as ad space on a college football article? Not if you’re trying to reach sports fans, for example. Then there’s brand safety and suitability concerns — I suspect certain brands would not be happy to be placed next to the animal neglect article.

The Big Picture

This is unlikely to be the only case of this kind of authorized spoofing in the wild. Exchanges, DSPs, and anti-fraud vendors need to take a good look at why it seemingly went undetected for so long, and where else it might be happening.

Part of what allowed Gannett’s ad inventory to be spoofed is their use of shared DIRECT accounts — if the sites didn’t share accounts the spoofing could have been detected by buyers when validating the inventory against ads.txt files.

Many other multi-site publishers have similar setups, and even more sales houses and other adtech intermediaries regularly mislabel their accounts as DIRECT across large networks of sites. Any of them could be engaging in spoofing — accidentally or otherwise — that won’t be caught by ads.txt validations.

Updating the definition of DIRECT in the ads.txt spec to close the multi-site publisher loophole, along with enforcing it to stamp out mislabelling, would make it easier to detect this kind of authorized domain spoofing (or at least make it harder to implement).

Finally, it would be helpful if certain segments of the adtech industry were a little more cautious about unfurling the “Mission Accomplished” banner.

Ad Specs Part 2: Sellers and Supply Chains

2021-10-18T22:00:00+08:00

This is part 2 in a series on IAB Tech Lab adtech standards. Read part 1 here.

In the previous post I covered the ads.txt standard, proposing that the account relationship field be re-defined so that DIRECT accounts could only be used on a single site. This change would enable the industry to clean up mislabelled accounts and help tackle domain spoofing.

But it’s been over 4 years since ads.txt was released, and in the meantime it’s been joined by a pair of newer standards — sellers.json and SupplyChain object. Let’s take a look at these standards, and how they fit in with the proposed ads.txt update.

The Sellers.json Specification

Ads.txt exposes what accounts are authorized to sell a site’s inventory, but very little about the accounts themselves.

In July 2019 the IAB Tech Lab released the sellers.json standard. Where ads.txt is quite limited in its focus — tackling domain spoofing — sellers.json has more nebulous goals. It exposes extra information about the accounts used to sell ad inventory, in service of a “transparent marketplace”.

The abstract section of the sellers.json spec.

The audience section of the sellers.json spec.

Sellers.json files are hosted by “ad systems” (sales houses, SSPs, ad exchanges, etc.), and list details of all the accounts used to sell ad inventory via each system.

These accounts are the same as the accounts listed in ads.txt files. In theory, the details for any account in an ads.txt should be able to be looked up in the relevant ad system’s sellers.json file.

The core information provided is the name of the company that is paid when inventory is sold via the account, along with its domain if it has one. Accounts can be marked as “confidential”, in which case these fields are omitted.

The definitions of the name and domain fields in the sellers.json spec.

Each account also has a “type”. If the company getting paid by the ad system (i.e. listed in the name and domain fields) owns the inventory being sold, it’s a PUBLISHER account. If the inventory is owned by a different company, it’s an INTERMEDIARY account. BOTH accounts sell both types of inventory.

The definition of the seller_type field in the sellers.json spec.

PUBLISHER and INTERMEDIARY seem quite similar to DIRECT and RESELLER from ads.txt, but the definitions are based on different concepts — payment vs. control — so they don’t always align.

A small example sellers.json file for a fictional ad exchange:

{
    "contact_email": "admin@exchange1.com",
    "contact_address": "Exchange One Inc., 1 Queen St, London, United Kingdom",
    "version": "1.0",
    "sellers": [
        {
            "seller_id": "foo",
            "name": "Publisher One",
            "domain": "publisher1.com",
            "seller_type": "PUBLISHER"
        },
        {
            "seller_id": "bar",
            "name": "Sales House One",
            "domain": "saleshouse1.com",
            "seller_type": "INTERMEDIARY"
        },
        {
            "seller_id": "baz",
            "name": "SSP One",
            "domain": "ssp1.com",
            "seller_type": "INTERMEDIARY"
        },
        {
            "seller_id": "ekki",
            "name": "Exchange Two",
            "domain": "exchange2.com",
            "seller_type": "INTERMEDIARY"
        }
    ]
}

The SupplyChain Object Specification

Ad buyers have traditionally only been able to see the account they are buying off — the “last seller”. When buying direct, it’s this last seller account that needs to be DIRECT.

The SupplyChain object OpenRTB extension was released alongside sellers.json. It adds a list of upstream accounts to bid requests, enabling buyers to see the other accounts the inventory has passed through. For “complete” SupplyChains, this list starts all the way back at the “first seller” — the account of the site owner.

The introduction section of the SupplyChain object spec.

Once again, these are the same accounts as those in ads.txt and sellers.json files. Any account in a SupplyChain object should be authorized in the site’s ads.txt, and be listed in a sellers.json file.

An example SupplyChain object for an RTB request on our fictional ad exchange:

{
    "ver": "1.0",
    "complete": 1,
    "nodes": [
        {
            "asi": "ssp1.com",
            "sid": "123",
            "hp": 1
        },
        {
            "asi": "exchange1.com",
            "sid": "bar",
            "hp": 1
        }
    ]
}

Scenarios

Now we’ve covered what sellers.json and SupplyChain objects are, let’s take a look at how they interact with the proposed ads.txt update. I’ll focus on two scenarios that are affected by the new definition of DIRECT.

Sales Houses

Sites that use a sales house to sell their inventory often list all the sales house’s accounts with downstream ad systems as DIRECT. This wouldn’t be allowed under the updated definition.

The site must list the sales house’s accounts with downstream ad systems as RESELLER. The sales house should create a dedicated PUBLISHER account for the site in its sellers.json, with the domain set to the site domain. The site can then list their account with the sales house as DIRECT.

saleshouse1.com/sellers.json:

{
    "contact_email": "admin@saleshouse1.com",
    "version": "1.0",
    "sellers": [
        {
            "seller_id": "1111",
            "name": "Site One",
            "domain": "site1.com",
            "seller_type": "PUBLISHER"
        }
    ]
}

site1.com/ads.txt:

saleshouse1.com, 1111, DIRECT
exchange1.com, bar, RESELLER

Site One SupplyChain object:

{
    "ver": "1.0",
    "complete": 1,
    "nodes": [
        {
            "asi": "saleshouse1.com",
            "sid": "1111",
            "hp": 1
        },
        {
            "asi": "exchange1.com",
            "sid": "bar",
            "hp": 1
        }
    ]
}

Note that while this is a change from how many sales houses are set up currently, it’s actually what they should be doing already. This was confirmed by a representative of the IAB Tech Lab in response to the previous post. The new definition of DIRECT simply clarifies this scenario.

Tweets from the SVP, Product Management & Global Programs at IAB Tech Lab.

Multi-Site Publishers

Publishers that own multiple sites often have a single set of accounts with downstream ad systems that are listed as DIRECT by all the sites. This wouldn’t be allowed under the updated definition. To keep using these shared accounts, the publisher would need to act as an ad system.

Each site must list the publisher’s accounts with downstream ad systems as RESELLER. The publisher should create a sellers.json containing a PUBLISHER account for each site, with the domains set appropriately. The sellers.json must be hosted on the domain used in the publisher’s accounts. Each site can then list their account with the publisher as DIRECT.

publisher1.com/sellers.json:

{
    "contact_email": "admin@publisher1.com",
    "version": "1.0",
    "sellers": [
        {
            "seller_id": "aaaa",
            "name": "Site A",
            "domain": "site-a.com",
            "seller_type": "PUBLISHER"
        },
        {
            "seller_id": "bbbb",
            "name": "Site B",
            "domain": "site-b.com",
            "seller_type": "PUBLISHER"
        }
    ]
}

site-a.com/ads.txt:

publisher1.com, aaaa, DIRECT
exchange1.com, foo, RESELLER

Site A SupplyChain object:

{
    "ver": "1.0",
    "complete": 1,
    "nodes": [
        {
            "asi": "publisher1.com",
            "sid": "aaaa",
            "hp": 1
        },
        {
            "asi": "exchange1.com",
            "sid": "foo",
            "hp": 1
        }
    ]
}

site-b.com/ads.txt:

publisher1.com, bbbb, DIRECT
exchange1.com, foo, RESELLER

Site B SupplyChain object:

{
    "ver": "1.0",
    "complete": 1,
    "nodes": [
        {
            "asi": "publisher1.com",
            "sid": "bbbb",
            "hp": 1
        },
        {
            "asi": "exchange1.com",
            "sid": "foo",
            "hp": 1
        }
    ]
}

Supply Path Optimisation

These account labelling changes will reduce the number of sites with DIRECT last seller accounts. This could result in a significant shift in ad supply and demand if buyers continue to prioritise buying direct.

If these buyers legitimately don’t want to buy from any intermediary, then this would simply be buyers actually getting what they thought they were buying all along. But it’s possible many buyers would be OK with certain indirect paths, if they could target them.

Buying direct is effectively using the ads.txt relationship field for a type of Supply Path Optimisation (SPO). But it’s a binary field — paths are direct or indirect. It just isn’t a good tool for the job, regardless of how it’s defined.

The SupplyChain object is clearly a better tool for SPO. With it, a buyer can check that all the accounts are authorized in the site’s ads.txt, not just the last seller. They can check the first seller is DIRECT. They can look them up in sellers.json files to see who the intermediaries are, who the first seller was, and determine whether they trust them.

You get the picture. SupplyChain objects, combined with data from sellers.json and ads.txt files, enable far more sophisticated SPO than the ads.txt relationship field ever could. With this kind of SPO, the importance of the last seller account being DIRECT is significantly reduced. A valid SupplyChain with trustworthy intermediaries could be just as valuable as a DIRECT account, depending on the buyer.

We should be aiming to get buyers using SupplyChain, rather than worrying about whether sites can be bought direct.

Is DIRECT Needed?

At this point, you may be wondering if it’s worth updating the definition of DIRECT at all. If buyers should be checking SupplyChains rather than buying direct, does DIRECT mislabelling matter? Do we need the ads.txt relationship field?

If ads.txt was still being drafted, I think there’d be a good argument for simply not including the relationship field. But ads.txt has been in the wild for 4 years, and the relationship field is being actively used by buyers. That’s not going to change overnight.

Using SupplyChain for SPO is not something new I’m proposing — that’s what it was designed for, and it was released over two years ago. Buyers are going to keep buying direct for some time yet, and I think it’s important to make sure they start getting what they’ve been paying for.

Even once using SupplyChains is standard and the direct/indirect binary is a thing of the past, buyers will need to validate those SupplyChains against information from sellers.json and ads.txt files. With its updated definition, the ads.txt relationship field provides information that isn’t available elsewhere: whether the account is specific to a site. This information is important for assessing the risk of domain spoofing by authorized accounts.

Sellers.json focuses on who gets paid for inventory, not what site it’s from. It could be updated to incorporate this information — something I think we should consider — but that is a much bigger change.

A Way Forward

DIRECT mislabelling has been a long standing issue with ads.txt. Cleaning it up is essential to weeding out fraud, and building a level playing field for legitimate publishers.

Narrowing the definition of DIRECT to enable this clean-up will impact some sites, but only while buying direct remains a prominent strategy. SupplyChain objects, paired with sellers.json and ads.txt, give buyers the data required to move away from this reliance.

Transparent, accurate supply chains are better for both advertisers and publishers. If you’re an advertiser, talk to your agency/DSP about utilizing SupplyChain object data in your campaigns. If you’re a publisher, check your ads.txt for mislabelled accounts and develop a plan for correcting them.

Changes like this won’t happen overnight, but the sooner we start, the better.

Ad Specs Part 1: Ads.txt Ambiguity

2021-10-04T22:00:00+08:00

Programmatic advertising is a massive, complicated, and largely opaque industry. Thousands of companies buy and sell ad space across the web in real-time auctions, determining what ads you see and how much publishers get paid.

This system funds a large proportion of the web as we know it, but has produced a variety of harms along the way; from the defunding of journalism in favour of clickbait and disinformation, to encouraging and facilitating ever more invasive user tracking, to the prevalence of outright ad fraud.

The ads.txt and sellers.json standards from the IAB Tech Lab have been touted as tools to provide much needed transparency and help tackle some of these issues, but problems with the interpretation, implementation, and enforcement of the standards have blunted their impact.

This is the first in a series of posts diving into these standards and their problems, and suggesting some potential solutions (or at least improvements).

The Ads.txt Specification

The obvious place to start is with the oldest standard, ads.txt. The first version of the ads.txt specification was released in June 2017, and it has had 3 minor revisions since then. The latest is v1.0.3, released in March 2021.

Its stated goal is to tackle domain spoofing — a type of ad fraud where someone sells ad inventory (ad space) claiming it’s for one site, but the buyer’s ad is actually displayed on a different site. (Yes, most details about ad inventory are self-declared by the seller, including the domain the inventory is on.)

The introduction section of the ads.txt spec.

Ads.txt addresses this problem by getting publishers to add a plaintext file (called ads.txt) to their site listing all the ad accounts that are authorized to sell inventory on that site. Before a buyer bids on ad inventory, they can check that the seller’s account is authorized by the relevant site’s ads.txt. If it isn’t, the domain is probably being spoofed.

Here’s an example of an ads.txt file, from the New York Times:

amazon-adsystem.com, 3030, DIRECT
appnexus.com, 3661, DIRECT
google.com, pub-4177862836555934, DIRECT
google.com, pub-9542126426993714, DIRECT
indexexchange.com, 184733, DIRECT
liveintent.com, 130, DIRECT
openx.com, 537145107, DIRECT
openx.com, 539936340, DIRECT
openx.com, 539052954, DIRECT
openx.com, 544071378, DIRECT, 6a698e2ec38604c6
rubiconproject.com, 12330, DIRECT
rubiconproject.com, 17470, DIRECT
triplelift.com, 746, DIRECT
pubmatic.com, 158573, DIRECT, 5d62403b186f2ace
pubmatic.com, 158945, DIRECT, 5d62403b186f2ace
media.net, 8CU2553YN, DIRECT
aol.com, 55861, DIRECT, e1a5b5b6e3255540
yahoo.com, 55861, DIRECT, e1a5b5b6e3255540
aol.com, 55792, DIRECT, e1a5b5b6e3255540
yahoo.com, 55792, DIRECT, e1a5b5b6e3255540
google.com, pub-1793726897772453, DIRECT, f08c47fec0942fa0
aps.amazon.com, 3030, DIRECT
indexexchange.com, 196165, DIRECT, 50b1c356f2c5c8fc

Beyond listing the authorized ad accounts, ads.txt provides another piece of information: the “relationship” between the publisher and account. This can be “DIRECT” or “RESELLER”.

DIRECT means the “…publisher (content owner) directly controls the account…”. RESELLER means the “…publisher has authorized another entity to…resell their ad space…”.

The definition of the relationship field in the ads.txt spec.

That all sounds fairly straightforward, right? If you’re a publisher, just add an ads.txt file that lists your own ad accounts as DIRECT, and the accounts of any resellers you work with as RESELLER. However, despite the apparent simplicity, there are at least two major contentious issues with ads.txt.

Authorized Spoofing

When checked by buyers (not a given), ads.txt does prevent spoofing by unauthorized accounts. I can’t open an ad account and sell spoofed nytimes.com ad inventory, as my account isn’t in their ads.txt. But what if the spoofer is in the spoofed site’s ads.txt?

Adtech vendors (sales houses, SSPs, resellers, etc.) act as intermediaries for multiple sites, so their ad accounts are authorized by all of them. For example, if a vendor works with a highly prestigious news site that commands high CPMs, what’s to stop them spoofing that prestigious site when selling ad inventory from their other sites?

Checking a site’s ads.txt won’t help buyers identify this kind of spoofing; the vendor’s ad accounts are in the spoofed site’s ads.txt, so it is authorized to sell the inventory. The same possibility arises when a publisher owns multiple sites, or colludes with other publishers (i.e. a dark pool sales house).

But let’s put a pin in this for a minute, and move on to…

Account Mislabelling

Ad inventory sold by DIRECT accounts is frequently preferred by buyers. Programmatic ad campaigns can be configured to prioritise buying from DIRECT accounts, or exclude RESELLER accounts entirely.

Each intermediary between the buyer and the publisher site increases the risk of fraud (e.g. domain spoofing), so “buying direct” is seen as a way to reduce that risk. It also has the potential to reduce costs by bypassing intermediaries that clip the ticket.

Response to a question on Twitter about why marketers prefer buying direct.

Follow-on to above response.

This makes inventory sold by DIRECT accounts more valuable, providing a clear incentive for publishers (often under instruction from the intermediaries) to label accounts as DIRECT, regardless of what the real relationship is.

Mislabelling accounts is ad fraud, as the buyers aren’t getting what they paid for. It’s also rampant. Almost any ad-enabled site you can think of will likely have multiple DIRECT accounts in their ads.txt file that are shared by tens, hundreds, or thousands of other unrelated sites.

They’re easy to spot with Well-Known, an open ads.txt index I run. Here’s a particularly bad case (log in to see counts of sites sharing each account), but the problem isn’t limited to disinformation sites.

Sample of mislabelled accounts from breitbart.com’s ads.txt. The numbers in parentheses are the number of sites that list each account as DIRECT.

Tackling this fraud requires three things: an agreed definition of when an account can be labelled DIRECT, an entity with the power and will to enforce this definition, and the ability for violations to be detected at scale. Currently, none of those things exist.

The Definition of DIRECT

For simple cases, the definition of DIRECT in the ads.txt spec seems clear. For example, if I create an account with Google AdX to sell ads on a site I own, I have direct control of the account, so it’s a DIRECT account.

But what if a publisher owns multiple sites? They still “directly control” the account, so can list it as DIRECT on all of them, right? What about a parent company that owns hundreds of companies that each have their own sites?

If an account isn’t DIRECT, does that make it RESELLER? It must, as there isn’t any other option, but RESELLER isn’t defined in opposition to DIRECT — it has its own separate definition.

Publishers often contract a vendor to manage their ad inventory for them. The publisher doesn’t “directly control” the vendor’s accounts, but the vendor arguably isn’t re-selling the inventory, as they never bought it — they’re selling it on behalf of the publisher. Should the vendor’s accounts be DIRECT or RESELLER? You can guess what most of these vendors argue.

A creative interpretation of DIRECT from an adtech vendor.

The term “publisher” is an essential part of the definitions, but it’s never explicitly defined. The spec refers to “content owners” (abstract and introduction sections), “publisher content distributors” (start of the specification section), and “publishers” (everywhere else) seemingly interchangeably.

The problem with this lack of consistency is that there are cases (e.g. syndicated content) where the content owner isn’t the site owner (“content distributor”). Either could be reasonably called the “publisher”. It’s the site owner that controls the ads.txt file, but should the content owner determine if an account is DIRECT? While there would be many problems trying to implement that interpretation, the definition of DIRECT explicitly includes “content owner” in parentheses after “publisher”, confusing the issue.

The RESELLER definition even seems to include an outright mistake. When I quoted it above I removed a section from the middle to help it make sense. It actually says: “…the Publisher has authorized another entity to control the account indicated in field #2 and resell their ad space…” (removed section in italics). This implies that the publisher controls the RESELLER accounts and has delegated that control to the reseller. This simply isn’t true in any case I know of. RESELLER accounts are owned and controlled by the reseller itself.

At this point, it would be helpful to look at the intention behind the account relationship field as a guide in interpreting the definition. However, the spec contains no information about the purpose of the field, nor how it should be used by buyers.

These fundamental issues with the ads.txt spec result in legitimate ambiguity about which accounts can be labelled DIRECT. This ambiguity is then leveraged to manufacture reasonable doubt whenever mislabelling is highlighted by researchers and activists. While everyone is stuck arguing about whose interpretation is correct — a clash of opinion that’s never resolved — the fraud continues.

Enforcement

This is where an enforcement body could step in. Even without a clear definition of DIRECT in the ads.txt spec itself, some interpretation could be selected and enforced, becoming the de facto “correct” interpretation.

The IAB Tech Lab itself is the obvious candidate here, as they developed the ads.txt standard in the first place. However, they constantly insist they aren’t an enforcement body, and would likely argue they don’t have power to actually enforce their standards even if they wanted to.

A Tweet from the SVP, Product Management & Global Programs at IAB Tech Lab.

Large scale DSPs and ad exchanges could act as enforcers by not buying from mislabelled accounts. There’s precedent for this — the general adoption of ads.txt was largely driven by Google’s announcement in September 2017 that their products would only buy from ads.txt authorized accounts.

So far, Google and the other big players don’t seem to be interested in seriously addressing account mislabelling. But even if they were, how would they detect it reliably enough to take enforcement action?

Under any reasonable interpretation of the spec, determining whether a publisher has “direct control” over an ad account requires knowing who the site’s publisher is, who owns the account, the relationship between the two, and how much “control” that relationship affords the publisher. That simply can’t be done programmatically.

Well-Known currently has ads.txt data for around 467k sites, which collectively list 537k unique accounts as DIRECT. Then there’s the long tail of ad-enabled sites that aren’t in Well-Known’s dataset. Manually investigating every potential case of mislabelling wouldn’t make a dent in the problem.

A Better Definition of DIRECT

Now that we have some idea of the shortcomings of the ads.txt standard, what can we do about it? Here’s where the pin we put in authorized domain spoofing comes in.

We don’t know what the original purpose of the relationship field was, but given that the purpose of the standard as a whole was to tackle domain spoofing, and ad buyers are trying to use it to limit fraud by avoiding intermediaries, why not use it to tackle authorized domain spoofing?

With this purpose in mind, here’s a new definition of the relationship field:

A value of DIRECT indicates that the ad account is only authorized to sell ad inventory on this site.

A value of RESELLER must be used in all other cases.

(Clearly the term “RESELLER” is far from ideal, but keeping it for backwards compatibility is worth the definitional dissonance.)

We also need a definition of “site” to go with this new relationship definition:

A site is determined by its root domain, with the exception of declared subdomains.

A subdomain that is declared with a SUBDOMAIN variable in its root domain’s ads.txt is its own separate site. Its ads.txt file therefore can’t share any DIRECT accounts with the root domain.

Undeclared subdomains are part of the root domain’s site.

Note that the ads.txt spec already defines the “root domain”, and how to handle SUBDOMAIN variables:

The definition of root domain from the ads.txt spec.

The definition of the SUBDOMAIN variable in the ads.txt spec.

The Benefits

This new definition is enforceable, less prone to misinterpretation, meaningfully reduces opportunities for domain spoofing, and is more in line with what ad buyers expect.

It doesn’t matter who the “publisher” is, what sites they own, or how much “control” they have over the account. All that matters is whether the account is exclusive to the site or not.

Mislabelling is easily detectable — every DIRECT account must only appear in the ads.txt for a single site. This can be automatically checked by anyone with a database of ads.txt files. Any ads.txt file can be checked for mislabelled accounts on Well-Known right now, and DSPs and exchanges could enforce this at any point with minimal effort.

With an unambiguous definition and wide enforcement, buyers could be sure that when they buy from a DIRECT account their money is going to the site the seller claims the inventory is from, and isn’t going through any intermediaries. This doesn’t entirely remove the possibility of domain spoofing — that would require enabling the buyer to validate the inventory details independently of the seller — but it’s far better than what we have currently.

The main limitation with this approach is that it’s a blunt tool. It’s based on what’s essentially a boolean field, so accounts are either DIRECT or they’re not. Buyers can’t choose how direct they want to buy. Many sites won’t be able to be bought DIRECT at all, as they’re too small to have their own accounts.

Thankfully, newer standards provide tools to address these limitations, which I’ll cover in a future post.

The second post in this series is now available here.