I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).
When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.
This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.
Just to be clear, you are a Bluesky engineer, right?
off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.
> and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline
I think then you still have the 'weird user who follows hundreds of thousands of people' problem, just at read time instead of write time. It's unclear that this is _better_, though, yeah, caching might help. But if you follow every celeb on Bluesky (and I guarantee you this user exists) you'd be looking at fetching and merging _thousands_ of timelines (again, I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users).
Given the nature of the service, making read predictably cheap and writes potentially expensive (which seems to be the way they've gone) seems like a defensible practice.
> I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users
Random sampling? It's not as though the user needs thousands of posts returned for a single fetch. Scrolling down and seeing some stuff that's not in chronological order seems like an acceptable tradeoff.
Do you know the name of the problem or strategy used for solving the problem? I'd be interested in looking it up!
I own DDIA but after a few chapters of how database work behind the scenes, I begin to fall asleep. I have trouble understanding how to apply the knowledge to my work but this seems like a useful thing with a more clear application.
To serve a user timeline in single-digit milliseconds, it is not practical for a data store to load each item in a different place. Even with an index, the index itself can be contiguous in disk, but the payload is scattered all over the place if you keep it in a single large table.
Instead, you can drastically speed up performance if you are able to store data for each timeline somewhat contiguously on disk.
Think of it as pre-rendering. Of pre-rendering and JIT collecting, pre-rendering means more work but it's async, and it means the timeline is ready whenever a user requests it, to give a fast user experience.
(Although I don't understand the "non-celebrity" part of your comment -- the timeline contains (pointers to) posts from whoever someone follows, and doesn't care who those people are.)
Perhaps I misunderstanding, I thought the actual content of each tweet was being duplicated to every single timeline who followed the author, which sounded extremely wasteful, especially in the case of someone who has 200 million followers.
I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot.
So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time.
I should make a blog post to record all the ones I can remember.
As a systems enthusiast I enjoy articles like this. It is really easy to get into the mindset of "this must be perfect".
In the Blekko search engine back end we built an index that was 'eventually consistent' which allowed updates to the index to be propagated to the user facing index more quickly, at the expense that two users doing the exact same query would get slightly different results. If they kept doing those same queries they would eventually get the exact same results.
Systems like this bring in a lot of control systems theory because they have the potential to oscillate if there is positive feedback (and in search engines that positive feedback comes from the ranker which is looking at which link you clicked and giving it a higher weight) and it is important that they not go crazy. Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly.
Reading this description of how user's timelines are sharded and the same sorts of feedback loops (in this case 'likes' or 'reposts') sounds like a pretty interesting problem space to explore.
I guess I hadn’t considered that search engines could be reranking pages on the fly as I click them. I’ve been seeing my DuckDuckGo results shuffle around for a while now thinking it’s an awful bug.
Like I click one page, don’t find what I want, and go back thinking “no, I want that other result that was below” and it’s an entirely different page with shuffled results, missing the one that I think might have been good.
That's connected with a basic usability complaint about current web interfaces, that ads and recommended content aren't stable. You very well might want to engage with an ad after you are done engaging what you wanted to engage with but you might never see it again. Similarly, you might see two or three videos that you want to click on on the side of a YouTube video you're watching but you can only click on one (though if you are thinking ahead you can open these in another tab.)
On top of that immediate frustration, the YouTube style interface here
collects terrible data for recommendations because, even though it gives them information that you liked the thumbnail for a video, they can't come to any conclusion about whether or not you liked any of the other videos. TikTok, by focusing on one video at a time, collects much better information.
I don't use DDG, but in my (very limited, just now) testing it doesn't seem to shuffle results unless you reload the page in some way. Is it possible you're browser is reloading the page when you go back? If so, setting DDG to open links in new tabs might fix this problem.
This behavior started happening for me in the last few months. If I click on a result, then go back, I have different search results.
I've found a workaround, though – click back into the DDG search box at the top of the page and hit enter. This then returns the original search results.
Hi - I work on search at DuckDuckGo. Do you mind sharing a bit more detail about this issue? What steps would allow us to reproduce what you're seeing?
Similar to how Google images loads lower quality blurred thumbnails towards the bottom of the window at first so that the user thinks they loaded faster
This is less a question of perfection and one of trade off's. Laws of physics put a limit on how efficiently you can keep data in NYC and London in perfect sync, so you choose CAP-style trade-offs. There are also $/SLO trade-offs. Each 9 costs more money.
I like your example it is very interesting. If I get to work on (or even hear someone in my team is working on) such interesting problems and I can hear about it, I get happy.
Interesting problems are rare because like a house you might talk about brick vs. Timber frame once, but you'll talk about cleaning the house every week!
That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue getting humans to use it was the only way to be net profitable, and you can't spend billions buying traffic and hope to make it back on advertising as the #3 search engine in the English speaking markets.
There are other ways to monetize search (look at Kagi for example) than advertising. Blekko missed that window though. (too early, Google needed to get a crappy as it is today to make the value of a spam free search engine desirable)
Blekko was gone by the time I learned about it. Recently (past few years) I emailed someone who worked on Blekko to get his opinion on a search engine concept I still have yet to start. His advice was to not bother competing with Google (obviously) LOL!
I don’t know if anyone’s embarked on a P2P search engine but that’s essentially my concept. Anyhoo, thanks for the inspiration!
Peer to peer would be tough, you really need a 10G network connection to some tier 1 provider, and about 2500 machines to distribute the crawling/serving load. (that is if you want to do a full stack search engine). And while you can run that infrastructure for on the order of $100K/month (not counting depreciation) that means you need roughly $5K/day in revenue from that cluster. At $10 RPM ($10 revenue per thousand queries) you're looking at a minimum of 500,000 'real' search queries during 'English time' (roughly 7AM to 11PM GMT). That's 31,250 queries per hour or ~9 queries per second (average).
And that just pays to keep the lights on at the colocation center. If you're paying off the development costs (30 - 50 developers over 2 - 3 years) and the cost of an office somewhere. You'll want at least double that revenue or you'll go broke before you break even.
Ideally you are the 'go to' place for people looking to buy something as those queries make money. People researching Douglas Fairbanks for a high school essay consume queries but don't generate ad revenue.
I know "search is hard" in the general sense but context is lacking (not a lot of details online from ex-search teams). It's always been apparent to me that you must have some other high-grossing product if you want to get into search or video, if only to pay for the servers.
Darknet Lantern is a decentralized searchable directory. It's probably not going to take off, but it could inspire something else. Servers spider other servers with the same software, and synchronized their data.
Yup, directory services are a lot easier to do peer-to-peer. Pinboard.in is a good shared directory (sort of Yahoo! without the editorial). They can yield excellent quality when you're searching for something that someone has 'indexed' with them, but poor recall when it comes to the set of all possible answers.
Doing it peer to peer without editorial allows sites to 'get into' the index easily which has its own plusses and minuses.
The lossy timeline solution basically means you skip updating the feed for some people who are above the number of reasonable followers. I get that
Seeing them get 96% improvements is insane, does that mean they have a ton of users following an unreasonable number of people or do they just have a very low number for reasonable followers. I doubt it's the latter since that would mean a lot of people would be missing updates.
How is it possible to get such massive improvements when you're only skipping a presumably small % of people per new post?
EDIT: nvm, I rethought about it, the issue is that a single user with millions of follows will constantly be written to which will slow down the fanout service when a celebrity makes a post since you're going through many db pages.
When a system gets "overloaded", typically it enters exponential degradation of performance state, i.e. performs self ddos.
> Seeing them get 96% improvements is insane
TFA is talking about P99 tail latencies. It does not sound too insane to reduce tail latencies by extraordinary margins. Remember, it's just reshaping of latency distribution. In this case pathological cases get dropped.
> does that mean they have a ton of users following an unreasonable number of people
Look at the accounts of OnlyFans models, crypto influencers, etc. They follow thousands or even tens of thousands of accounts in the hope that we will follow them in return.
I don't see that accommodating this behavior is prosocial or technically desirable.
Can you think of a use case?
All sorts of bots want this sort of access, but whether there are legitimate reasons to grant it to them on a non-sharded basis is another question since a lot of these queries do not scale resources with O(n) even on a centralized server architecture.
Given enough time, you'll end up with a lot of legitimate users who follow a huge number of accounts but rarely interact with more than a handful, similar to how many long-time YouTubers have a very high subscriber:viewer ratio (that is, they have way more subscribers than you would expect given their average view count), and there's nothing inherently suspicious about it. People lose access to their accounts, make new accounts, die, get bored, or otherwise stop watching the content but never bother unsubscribing because the algorithm recognized this and stopped recommending the channel's uploads to them.
Bluesky doesn't have this problem yet because it's so young, so the outsized follow counts are mostly going to be from doomscrollers and outright malicious users, but even if it was exclusively malicious users, there is no perfect algorithm to identify them, much less do so before they start causing performance problems. Under those constraints, it makes sense to limit the potential blast radius and keep the site more usable for everyone.
> Generally, this can be dealt with via policy and moderation to prevent abusive users from causing outsized load on systems, but these processes take time and can be imperfect.
So it’s a case of the engineers accepting that, however hard they try to moderate, these sorts of cases will crop up and they may as well design their infrastructure to handle them.
Ok I'm curious: since this strategy sacrifices consistency, has anyone thoughts about something that is not full fan-out on reads or on writes ?
Let's imagine something like this: instead of writing to every user's timeline, it is written once for each shard containing at least one follower. This caps the fan-out at write time to hundreds of shards. At read time, getting the content for a given users reads that hot slice and filters actual followers. It definitely has more load but
- the read is still colocated inside the shard, so latency remains low
- for mega-followers the page will not see older entries anyway
There are of course other considerations, but I'm curious about what the load for something like that would look like (and I don't have the data nor infrastructure to test it)
Hmm. Twitter/X appears to do this at quite a low number, as the "Following" tab is incredibly lossy (some users are permanently missing) at only 1,200 followed people.
It's insanely frustrating.
Hopefully you're adjusting the lossy-ness weighting and cut-off by whether a user is active at any particular time? Because, otherwise, applying this rule, if the cap is set too low, is a very bad UX in my experience x_x
1200 people is really nothing, specially if you have a job tangentially related to social media (for example journalists). It's really simple, you are not the same type of user. You have 50 "acquaintances", they have 1200 "sources".
The article is talking about people who have following/follower counts in the millions. Those are dozens of writes per second in one feed and a fannout of potentially millions. Someone with 1200 followers, if everyone actually posts once a day (most people do not) gets... a rate of 0.138 writes per second.
They should be background noise, irrelevant to the discussion. That level of work is within reasonable expectation. What they're pointing out is that Twitter is aggressively anti-perfectionist for no good technical reason - so there must be a business reason for it.
You are on the verge of discovering why non-chronological timelines exist :) It's not hard to imagine that there are 1200 people posting at least one thing a week that you would find interesting. The trouble is, if they also post 100 things that are not interesting, how does the software surface the interesting stuff without drowning you in the non-interesting stuff? How do you do that in a way that feels fair to the user ("I never see Friend X's posts because they're drowned out by the interesting stuff posted by the other 1199 people I follow")? It's tough!
Personally, I always use chronological. I like to be able to hop on, and mingle with whatever percent of people are online and posting at any particular time.
Replying to people right after they post is how you actually get to have conversations with people, and get to know them well, imo
They're all people that I know, and 98% of them are mutuals. I regularly go through my list of accounts I'm following every 5-6 months, and get rid of people who have disconnected from me, have gone inactive, or I don't really know.
I think the part you're missing here is that there are certain parts of twitter where the density of interconnection is really high, so you'll know people because you see them in the comments of a lot of threads, or through retweets.
It's really not that hard to end up knowing 1,000+ people if you engage with a group or "corner" of twitter. Even moreso if it's multiple corners. (Like, AI, but also just frens, but also a little political corner, people from specific cities, etc.)
I can come up with 100 people I'd want to follow on Twitter, and I don't even have an account. Don't dismiss other people's use-cases if you don't have or understand them.
> Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new.
While I'm fine with the solution, the wording of this sentence led me to believe that the solution was going to be imperfect chronology, not dropped posts in your feed.
So, let's say I follow 4k people in the example and have a 50% drop rate. It seems a bit weird that if all (4k - 1) accounts I follow end up posting nothing in a day, that I STILL have a 50% chance that I won't see the 1 account that posts in a day. It seems to me that the algorithm should consider my feed's age (or the post freshness of my followers). Am I overthinking?
The "reasonable limit" is likely set based on experimentation, and thus on how much people post on average and the load it generates (so the real number is unlikely to be exactly "2000", IMHO).
If you follow a lot of people, how likely it is that their posting pattern is so different from the average? The more people you follow, the less likely that is.
So while you can end up in such situation in theory, it would need to be a very unusual (and rare) case.
I think the 'law of large numbers' says that it's very unlikely for you to follow 4k and have _none_ of them posting. You could artificially construct a counter-example by finding 4k open but silent accounts, but that's silly.
The other workaround is: follow everyone. Write some code to get what you want out of the jetstream event feed. https://docs.bsky.app/blog/jetstream
Yeah, this seems concerning to me. Maybe now as the platform is new this isn't much of an issue. But as accounts go inactive people will naturally collect "dead" accounts that they are still following. On Facebook it isn't uncommon of to have old accounts of sociable people naturally collect thousands of friends.
It seems that what they are trying to measure is "busy timelines" and it seems bag they could probably measure that more directly. For example what is the number of posts in the timeline over theast 24h? It seems that it should be fairly easy to use this as the metric for calculating drop rate.
Love reading these sorts of "technical problem + solution" pieces. The world does not need more content, in general, but it does need more of this kind of quality information sharing.
Anyone following hundreds of thousands of users is obviously a bot account scraping content. I'd ban them and call it a day.
However, I do love reading about the technical challenge. I think Twitter has a special architecture for celebrities with millions of followers. Given Bluesky is a quasi-clone, I wonder why they did not follow in these footsteps.
You don't need to follow anyone (or even have an account) to scrape content… Someone following a huge amount of accounts usually wants to get a lot of followers quickly this way through follow-backs.
Maybe not hundreds of thousands but I'd follow anybody that looks remotely interesting and then primarily use customized feeds. E.g. if I wanna hear about union news, my personal irl network, etc I check that feed
This does assume that scrapers are smart, and often they're really not. They have infrastructure for scraping HTML from webpages at scale and that is the hammer they use for all nails. (e.g. Wikipedia has to fight off scraper traffic despite full archives being available as torrents, etc.)
In this case I agree though, they're all spammers and/or "clout farmers", or trying to make an account seem more authentic for future scams. They want to generate follow notifications in the hope that some will follow them back (and if they don't, they unfollow again after some interval).
100%. I ran a job board where we provided a nice machine readable XML feed of all of our jobs, but we had bots that insisted on using the standard search box. Searching by city using an alphabetized list.
Geographic search to was the most expensive thing they could have done and no matter what we did we couldn’t get them to use the XML feed.
I even tried returning a link to the feed when we detected a bot. No dice. They just kept working around the bot detection.
BlueSky has starter packs that allow you to mass follow in the click of a button. You join 10 starter packs in one day, you are following over 1000 people. Sometimes following others is the only way to get people to engage with your content.
No matter how high you set a maximum limit for interactions on social media (followers, friends, posts, etc), someone will reach the limit and complain about it. I can see why Bluesky would prefer a "soft limit", where going above the limit will degrade the experience. It gives more flexibility to adjust things later, and prevents obnoxious complaints from power users with outsized influence.
I’m skeptical that the people who would complain about that wouldn’t find something else to complain about if you resolved the first complaint. I’d recommend implementing product features that you think are reasonable and accepting the fact that you will get complaints from people who disagree.
- Make it easy to systematically unfollow people (or degrade them to a different tier, see below, or sort them automatically into a different feed; maybe even allow automatic following of certain people, like your cities mayor or local ice cream parlors). Like based on recent activity, last engagement with a post, type of content (pictures, videos, links ...), on a schedule (e.g. follow for 3 yeard, follow until 2028), special status (family, friends, member of congress, member of city council, mayor...), number/ratio of common followers, regex expressions, recommendations by certain accounts, letter-to-word ratio, season, planetary alignment, weather, age, train departure time, side-chaing based on other accounts, force accounts to play russian unfollow roulette, urgency to pee, healthcare CEO life expectancy derivative, ... or any combination of these.
- Allow different tiers of following someone. Like friends (never unfollow, always fetch updates), family (never unfollow, rate limit high-energy uncles), news (filter based on urgency or current topics of interest), politicians (highlight as untrustworthy, attach link to donation and board membership disclosure, attach term-limit and next election countdown), local businesses (hard rate limit, attach opening hours), bookmark (never unfollow, no updates), ... maybe multiple tiers in each category and allow those being followed to either temporarily boost their tier (or tiers of certain posts) or e.g. once per year.
- Allow people from exempting some of their posts from not being shown to some of their followers. E.g. two per week and an additional 5 per month.
- Allow people to choose which followers should be given a higher priority when writing posts to their feeds.
Nice problem to have, though. Over on Nostr they're finding it a real struggle to get to the point where you're confident you won't miss replies to your own notes, let alone replies from other people in threads you haven't interacted with.
The current solution is for everyone to use the same few relays, which is basically a polite nod to Bluesky's architecture. The long-term solution is—well it involves a lot of relay hint dropping and a reliance on Japanese levels of acuity when it comes to picking up on hints (among clinets). But (a) it's proving extreme slow going and (b) it only aims to mitigate the "global as relates to me" problem.
The basic idea is to assign each user to multiple shards, decreasing the changes of another user sharing all their shards with the badly behaving user.
Fixing this issue as described in the article makes sense, but if they did shuffle sharding in the first place it would cover any new issues without effecting many other users.
I think shuffle sharding is beneficial for read-only replica cases, not for writing scenarios like this. You'll have to write to the primary and not to a "virtual node". Right? Or am I understand it incorrectly? I just read that article now.
When I go directly to a user's profile and see all their posts, sometimes one of their posts isn't in my timeline where it should be. I follow less than 100 users on Bluesky, but I guess this explains why I occasionally don't see a user's post in my timeline.
If another user you follow reposted or replied to a post, it can affect its order in your following feed. You shouldn't be seeing any loss as described in the article from following only 100 users.
I've experienced it with "first-party" posts, not replies. A post wouldn't show in my timeline but would on the user's profile. This is the official android app, but there has been an update or two so I'll have to double check again
Various clients (I’m writing one) interpret the timeline differently, as a feed that shows literally everything includes could things that most people would find undesirable or irrelevant. (replies to strangers, replies to replies to replies, etc)
I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this,
"In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page."
Basically means that they block on each page, process all the items on the page, and then move on to the next page. Why wouldn't you rather decouple page fetcher and the processing of the pages?
A page fetching activity should be able to continuously keep fetching further set of followers one after another and should not wait for each of the items in the page to be updated to continue.
Something that comes to mind would be to have a fetcher component that fetches pages, stores each page in S3 and publishes the metadata (content) and the S3 location to a queue (SQS) that can be consumed by timeline publishers which can scale independently based on load. You can control the concurrency in this system much better, and you could also partition based on the shards with another system like Kafka by utilizing the shards as keys in the queue to even "slow down" the work without having to effectively drop tweets from timelines (timelines are eventually consistent regardless).
I feel like I'm missing something and there's a valid reason to do it this way.
I interpreted this as a batch write, e.g. "write these 10k entries and then come back". The benefit of that is way less overhead versus 10k concurrent background routines each writing individual rows to the DB. The downside is, as you've noted, that you can't "stream" new writes in as older ones finish.
There's a tradeoff here between batch size and concurrency, but perhaps they've already benchmarked it and "single-threaded" batches of 10k writes performed best.
I found it odd to base the loss-factor on the number of people you follow, rather than a truer indication of timeline-update-frequency. What if I follow 4k accounts, but each of those accounts only posts once a decade? My timeline would be become unnecessarily lossy.
The funny thing is that all of the centralization in Bluesky is defended as being necessary to provide things like global search and all replies in a thread, things that Mastodon simply punts on in the name of decentralization. But then ultimately, Bluesky has to relax those goals after all.
ought to be possible to reverse-engineer it by following a large number of active accounts and seeing what percentage of their posts actually hit your feed
An interesting solution to a challenging problem. Thank you for sharing it.
I must admit, I had some trouble following the author's transition from "celebrity" with many followers to "bot" with many follows. While I assume the work done for a celebrity to scatter a bunch of posts would be symmetric to the work done for a commensurate bot to gather a bunch of posts, I had the impression that the author was introducing an entirely different concept in "Lossy Timelines."
That’s quite interesting and a challenge I have not thought of. I understand the need for a solution and I believe this works reasonably well, but I am wondering what is happening to users that follow a lot of accounts with below-average activity. This may naturally happen on new social media platforms with people trying out the service and possibly abandoning it.
The „reasonable limit“ is likely set to account for such an effect, but I am wondering if a per-user limit based on the activity of the accounts one follows will be an improvement on this approach.
To help avoid the hot shard problem, I wonder how capping followers per "timeline" would perform. Especially each user would have a separate timeline per 1000 followers and the client would merge them. You could still do the lossy part, if necessary, by only loading a percent of the actual timelines. That wouldn't help the celebrity problem but it was already acknowledged earlier that the solution to that is to not fan out celebrity accounts.
In the fanout design, why not dynamically move on to the next 10,000-user page as soon as all tasks for the current page are either queued or processing? Would that approach improve throughput, or could it introduce issues like resource contention?
On a related note, I am pretty confident that one of the main reasons the WWW succeeded where previous attempts failed was that it very specifically allowed 404s.
The solution to this problem is known and implemented already: the social web should be distributed between thousands of pods which should contain at the maximum a few thousands users.
Diaspora is already working like this for 15 years. It is technically harder to build initially but it then divide all the problems (maintenance, moderation, load, censorship, trust of the owner...) Which makes the network much more resilient.
Bluesky knows that and they are allowing other people to host their software but they are really not pushing for it and it highly doubt that the experience of a user on a small external pod is the same than on bluesky.com.
This particular problem will still exist for a fediverse server. You follow 10k people? Nice, now you're getting ddos'd by their activities. Though, most fediverse servers being monolithic applications definitely helps.
the use of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served a good implementations in hot reload scenarios
The typical problem of a centralized infrastructure.
Indeed:
> This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability
Note that all of this reflects design decisions on Bluesky's closed-source "AppView" server—any federated servers interacting with Bluesky would need to construct their own timelines, and do not get the benefit of the work described here.
As others have noted, the appview is open source. The dataplane has two implementations, one in postgres and another in scylla. The scylla dataplane is closed, the postgres one is open.
The interesting next stage for the postgres implementation is to create a sync engine for partial syncs of the network, so that an appview can run affordably. We ran some benches on the current state of the postgres implementation and found we could index 300k users on a $100/mo vps. I think with a couple of weeks of optimization that could reach 1mm users.
This is great to hear—my current understanding of the most recent state of the art on the topic is https://alice.bsky.sh/post/3laega7icmi2q which mentions that the self-hosted appview is not yet open source. So I'm glad to hear the situation has changed in the past 3 months.
It was open source (except the Scylla database layer) from the beginning, AFAIK - that blog post just says that they haven't set it up yet, because that's the hardest part to run
The statement "any federated servers interacting with Bluesky" is ambiguous, because Bluesky's federated model means there's many different types of servers, and one user's view of what a "federated server" could be vastly different from another.
Federated PDS-s (which is probably the closest to what people mean when they say they want to federate on bluesky) would not need to reconstruct timelines if their users use the bsky.app appview.
Thanks, that's a fair point that I was overlooking. When I say a "federated server", I don't just mean a self-hosted PDS, I mean a third party app that potentially has its own lexicon and design decisions. Creating a robust third-party app that can meaningfully interact with the Bluesky network is still a very difficult engineering challenge, which I think this article does a good job demonstrating—that was the tension I was trying to underscore in my comment. Bluesky may be solving those engineering challenges for those clients who are satisfied with Bluesky's frontend and AppView, but every single other app built on top of ATProto will have to resolve those same challenges. This is directly downstream from Bluesky's "global firehose" topology and various design decisions that stem from that.
> We did a backend rewrite from postgres to scylla and it has a bunch of deployment specific stuff, but is functionally identical to the open source postgres version. Its not really a "v2" in terms of new features, we just made it make use of our hardware really well[1]
It's not coming, it never went away… As I understand it, the "business layer" with all the logic is above the data later, shared by the Postgres and Scylla versions, and the data layer just makes queries to the database. I think they are using the Postgres version locally for development.
What is not are the extra services they run to provide a better and faster UX. Even if it was open source, it likely costs 10s of thousands to run per month (they have moved largely to "onprem" hardware instead of the cloud aiui)
AppView is a specific term of art within the Bluesky federation architecture: https://atproto.com/guides/glossary#app-view, you were incorrect in identifying the public frontend repo as the AppView.
A frontend is (can be) part of an App View. It is quite literally the app you view the network through. There can also be headless app views and app views which have no backend
App View is a bit fuzzy of a term. To me it seems like a combination of frontend, backend, custom lexicon, and supporting services. There isn't really another place in the spec or design where clients or browsers fit in, which do in fact provide a view of the network via an app.
when I read the spec it seemed like the operator of an AppView & Relay would be most in need of compensation for their hosting costs due to the amount of demand on those components so I believe the spec allows an operator to implement their own AppView & monetize it as that operator sees fit, so that they can afford to operate the service and maybe even make money off of it so that they can make it their full time jobs.
It seems this way to me as well. ATProto fundamentally changes how monetization works in social media by removing lockin. It's going to be interesting to see what emerges from this design decision.
Another interesting way to view ATProto is that it could be a collection of headless features and network browsers that leverage those feature providers.
what else? profit by means of doing work that benefits first and foremost the private proprietors of the closed source
if they gave it away (which used to be unfeasible until the digital era) they feel they’re loosing their valuable effort which they’re wont on concentrating, not diluting.
My thinking has evolved on this topic significantly as of late. My current thinking is we should create a secure gossip network on top of the Bluesky API, and forgot about all the DAG-CBOR stuff that gets stripped from the Jetstream. Hash the posts on the gossip layer and if posts change then diff them. This is all prep for when X billionaire buys out Bluesky then we just pop some signing key crypto on top of this gossip layer and wow! It's distributed!
So the system design puts the burden on what seems to be synchronous, not queued, writes to get easy reads. I usually prefer simpler cheaper writes at the cost of more complicated reads as the reads scale and parallelize better.
I understand that it's a different point, but how can someone write a whole essay called "When imperfect systems are good" without once mentioning Gabriel or https://en.wikipedia.org/wiki/Worse_is_better?
Anecdotally, I ran into a similar solution "by chance".
Long ago, I worked for a dating site. Our CTO at the time was a "guest of honor" who was brought in by a family friend who was working in the marketing at the time. The CTO was a university professor who took on a job as a courtesy (he didn't need the money nor fame, he had enough of both, and actually liked teaching).
But he instituted a lot of experimental practices in the company. S.a. switching roles every now and then (anyone in the company could apply for a different role except administration and try themselves wearing a different hat), or having company-wide discussions of problems where employees would have to prepare a presentation on their current work (that was very unusual at the time, but the practice became more institutional in larger companies afterwards).
Once he announced a contest for the problem he was trying to solve. Since we were building a dating site, the obvious problem was matching. The problem was that the more properties there were to match on, the longer it would take (beside other problems that is). So, the program was punishing site users who took time to fill out the questionnaires as well as they could and favored the "slackers".
I didn't have any bright ideas on how to optimize the matching / search for matches. So, ironically, I asked "what if we just threw away properties beyond certain threshold randomly?" I was surprised that my idea received any traction at all. And the answer was along the lines of "that would definitely work, but I wouldn't know how to explain this behavior to the users". Which, at the time, I took to be yet another eccentricity of the old man... but hey, the idea stuck with me for a long time!
The answer to that reply is you don't need to explain it to your users. People are used to fuzzy/best-effort sort of matching, especially when it's specifically presented as a "matching algorithm" instead of a "filter".
> This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post.
Seriously? Isn't this the nut of your problem right here?
What alternative design did you have in mind, given that a Twitter-like data model of individual follows is likely a strict product requirement?
There are obviously other ways of doing it (doing the timeline propagation in a batch job, fanning out the reads rather than the writes), but they've got their own problems. Probably worse ones.
You'd still have exactly the same hot write path, it'd just have maybe 50% of the load. That could be a legit optimization, but not having it hardly warrants an incredulous "seriously?" like the OP's.
(And the same for the inverse hybrid strategy of quarantining the writes of highly followed users and handling their fan-out at read time. A neat optimization, and maybe even absolutely once you have accounts with 100M followers. But the vast majority of posts would still be handled via the original strategy.)
An airline reservation system has to be perfect (no slack in today's skies), a hotel reservation can be 98% perfect so long as there is some slack and you don't mind putting somebody up in a better room than they paid for from time to time.
A social media system doesn't need to be perfect at all. It was clear to me from the beginning that Bluesky's feeds aren't very fast, not like they are crazy slow, but if it saves money or effort it's no problem if notifications are delayed 30s.
It's funny because from my experience airline systems are very imperfect (timing wise).
I (unwisely) tried to purchase an Icelandair ticket via the Chase travel portal. I would get a reservation number, go buy seats on Icelandair's website, and a few days later the entire reservation would vanish into the ether. Rinse and repeat 3x.
I can't remember the exact verbiage, but basically tickets can be "reserved" and "booked". One means the ticket is allocated, and one means the ticket is actually paid for. I eventually sat on the phone with an executive support person as they booked the ticket and got it all the way through. It turns out Chase reserves a ticket on an airline but as an SLA of ~3 days to actually pay for the ticket. Icelandair's requires a ticket to be paid with in 24 hours, so it was timing out.
Airlines are far from perfect. They overbook flights and sometimes have to ask people leave and pay them for the inconvenience. My wife and I once got $1000 a piece and a hotel and food voucher to volunteer to take a flight the next day on a layover in Atlanta.
As far as your particular situation, the number one rule of using a third party portal to book flights or hotels is - don’t.
I understand that Iceland Air is not a transfer partner of Chase. But even in that case, I would just wait to use my points until I could use a transfer partner.
On the earning side if paying cash, the difference between 2x/3x points when booking directly and 5x when going through the portal just isn’t worth the risk.
Overbooking is not a mistake, though. People miss flights for many reasons, and the airlines predict this with impressive accuracy, to the point that they can afford to pay tremendous sums for being wrong and yet still come out ahead.
Think about other ad-supported sites. If you're an engineer working on an ad-supported product, the perfect consistency you strive for in your code is not the product. The product is the sum of all of the content the user sees. And the costs of the tradeoffs you make are paid for by ads.
Am I willing to see 10x more ads for perfect consistency? Definitely not.
No, overbooking is a business decision justified by the fact that, statistically, not all passengers will actually show up for their flight, and lower load factors cost money.
However, my understanding is that airlines have much more sophisticated per-flight and per-passenger models that calculate the predicted no-show factor based on the historical rates for that particular route (e.g. you're more likely to get more no-shows in business class flying from NYC to SF compared to holiday travelers with a reservation on the Florida Keys)
That blows my mind, I would expect maybe 1 or 2 passengers per plane at most. I'm trying to think of what factors would cause that many no-shows, it has to be mostly missed connections?
I can't imagine spending hundreds of dollars and just not showing up.
Delays getting to the airport and missing the plane. Cancelations with full refund. "Hidden city" ticketing. Layover delays. Businesses booking blocks. Flexible flights ( https://www.travelperk.com/guides/flexible-travel/flexible-f... ). Changing / rebooking flights for an earlier or later time.
A friend of mine works for a Management Consultancy firm and they have full flex tickets if they miss the 8pm flight home they can take the next one or fly back the next morning. All without additional fees.
So I believe business travel is the biggest factor when it comes to missed flights.
Side note: His employer is the biggest client of a major European airline.
No additional fees, but the cost of the ticket is typically sky-high. In many cases the company could purchase 3 restricted-fare tickets for less than the unrestricted fare. (And also, the consultants want to get home, they HATE staying extra nights, so they have inherent motivation to catch the original flight). You'd think that management consulting firms, you know, who are supposed to be good at optimization and reducing costs, would have figured this out.
* A massive bulk of flex rates is different price than a single flex ticket
* when I travelled to a single city with 20+ colleagues for several years, our nightly hotel rate was less than quarter of published rate. I don't think we got anything like that on Airlines, but nevertheless, bulk gets discounts.
* I forget details but between specific frequent cities which had hourly flights, we had full flexibility and we used this all the time. So we might catch a flight hour earlier if there was one available, or hour later if need be than booked.
Basically to everybody's point, business travel is very different than vacation travel and intuition from once a year personal trip don't apply.
* Hotels are required by law to have a published "rack rate" and it's common for both leisure and business travelers to book at more than half, or more, off those rates. Airlines don't have that.
* Yes of course there are negotiated discounts for major corporations- but full flexibility is still far more expensive than restricted tickets, just like business is still far more expensive than economy.
* Most airlines offer guaranteed same-day changes, or at least priority standby, to high-status loyalty members (which most consultants are) so buying full-flex tickets to get an hour earlier/later flexibility is redundant - basically, paying for a benefit you already have.
... or perhaps they prioritize ensuring their consultants feel like they're being looked after - replacing them when they find another job is very expensive.
What part of my comment suggests consultants aren't "looked after?" The idea that the consultants' personal time is so low on the scale of priorities that the company prefers to pay 3x or 4x for plane tickets to ensure they are available to spend extra time with clients, rather than their families, suggests that the company is specifically NOT looking after the consultants' well-being.
That the consultants' personal time is high on the scale is why they pay for the flex-fare tickets. So you miss an 8pm flight for one reason or another - it happens, so "don't miss the flight" isn't a valid retort.
Do I feel better taken care of if I can just take the following morning's flight at no additional cost or hassle, or if I now need to contact someone at Amex Global Business Travel, have them try to get me on a flight, have another expense, potentially not be able to get on the next flight or the one after that, etc.
The company can start with heavily-discounted restricted tickets and eat them when a consultant needs to extend the trip. Today it's even easier because airlines have eliminated change fees - so essentially any ticket can be canceled and those funds credited to a future flight. It's all still "no cost or hassle" to the consultant since they aren't paying for the flights to begin with.
Similar reasons people buy overnight sleeper tickets from the west coast to asia for sky-high prices. That person has to be there, and they have to be rested. If they are negotiating 100m+ dollar contracts its a no-brainer.
Intercontinental business-class tickets have a tangible benefit, guaranteed lie-down seat and relaxing accommodations that enable rest, as you correctly point out. Flexible airline tickets have no tangible benefit, it is simply risk mitigation (avoiding wasting a nonrefundable ticket if a person can't make a flight) and like other risk-mitigation products such as insurance, extended warranties, car rental protection, etc, it is priced so that the cost exceeds the benefit.
And as stated elsewhere, the majority of consultants are relatively junior people whose role has nothing to do with negotiating contracts.
Vast majority of consultants are working on engagements where "the deal" was negotiated weeks/months earlier.
The goal isn't to eliminate flexibility, it's understanding probabilities. If fully-flexible/refundable flights are 3x the fare of restricted, then in aggregate, the company could have a chunk of consultants throw out their original reservations and rebook later flights, and still save money. Yes- sometimes consultants need to stay longer than planned, but in an age of prioritizing "work-life balance," most consultants are encouraged to stick to their schedule and get back home when originally planned.
Keep in mind they sell a lot of tickets where one of the features that allows for a premium price is that they allow late cancellations or changes to other flights. Holiday travelers are pretty "reliable", but business travelers might have changed needs at the drop of a hat (say you meet another prospective client on a business trip and decide to stay another day to fit in a face-to-face meeting).
Remember lots of business travelers have connections or flex schedules. When I had to go to the West coast for business, I usually have full fare tickets and book a later flight. If I had flexibility, I’d switch to an earlier plane or first class.
I think you'll have to pay a team millions to figure that out, it is unlikely to be a static rate but rather decided based on multiple traits like time of year, time of flight, distance of flight, cost of ticket, etc.
They probably just have an SSRS report that prints out in a few dozen offices automatically on some schedule.
I'm not trying to be pedantic but this is table stakes stuff. I know we're supposed to shy away from saying things like this but compared to the other engineering that airlines have to do, this is easy. It costs - at most, including wages - a few tens of thousands of dollars yearly to come up with these figures. It's a fraction of the salary of one United Airlines BA.[0] This cost might go up if one of the senior developers convinces their boss that this needs to be a machine learning model but unless they're resume pumping it's going to be at most PCA and a regression.
This is not a team of people working for months on this one thing.
Miscommunication leads to bad outcomes. One missed message out of order could easily lead to a fight, a lawsuit, a flash mob, threats of violence - that then need to be taken seriously, swatting, DOXxing, etc...
Msg 1: I hate ___insert_controversal_person_category_here___
Msg 2: Is the kind of statement that really sets me off
Msg 1 has a very different meaning if you don't see Msg 2.
It’s really impressive how well Bluesky is performing. It really feels like a throwback to older social media platforms with its simplicity and lack of dark-patterns. I’m concerned that all the great work on the platform, protocol, etc won’t shine in the long term as they eventually need to find a revenue source.
I love Mastodon but I have to admit that BlueSky has clearly out-engineered them. Of course they started with much more expertise and resources. I hope ActivityPub compatibility soon to unite the two
They've done an incredible job running with an extremely low headcount and crazy efficient use of hardware. It would be easy to 10x their expenses if they were blindly following the standard cloud deployment playbook. Hopefully this level of efficiency mean they don't have to work as hard and can stay pre-revenue, a pure play, for a very long time.
I completely agree with this... but without profit, people can't get paid, and they'll stop building. I do hate this incredibly need for growth, of course, but financial growth is necessary to pay people and give them raises and allow them to have upward mobility at the company.
I hope Bluesky is able to find a model that works for them AND for consumers. (I do know it's an open protocol, so it'll live on without Bluesky itself! However, as this post shows, it's a lot of work to build on the prototype... so if not them, who? And if someone else, how will they become sustainable?)
It's semantics but I like to separate money from profits. You need money to pay people and to survive but you don't need to be raking in endlessly growing piles of it. This is something that was really demoralizing about working for a big company, they could be making like 50000000000 a year in just profits but still be ruthless in getting more. Like I just want to make a product I'm proud of and I'm happy living a simple life, I am happier now making less money but not feeling like I'm endlessly milking customers.
Once you take on investors, that’s not an option. VCs expect rapid growth and an exit - statistically through acquisition, but occasionally an IPO.
Once you go public, then you have investor pressure and can be subject to activist investors unless the founder has controlling interests like in the case of Meta and Google.
Basically why I don't like to work for any public company anymore. You are there to be exploited by moneyed interests that most often don't even have any positive effect on the products itself.
At the same time I feel like a lot of companies grow much larger than they need to be simply because of bigger is better mentality. How many of Uber's 30,000ish employees are involved with making sure the app and backend database are working properly? Are they really doing 600 times more work than Craigslist at connecting sellers with buyers?
Like, sure, they don't need every single one of those 30,000... but they have to have ground teams in every city in the world. Connections with every airport. Connections with almost every restaurant in the world. Customer support and safety (okay I know they don't nail this, but still). They need to pay out drivers in each country. The app needs to work in hundreds of countries, all with different laws, currencies, languages and more. Some places let you pick up anywhere, others require specific locations. And that's not even including marketing, partnerships, HR, finance, etc.
I don't think the employees are the problem with Uber, it's the shareholders. They need to make X back, so that delta is where drivers get squeezed.
Aren't you actually arguing in favor of profit-driven behavior? You're not disagreeing with profit as a motivator, you're questioning if the 30,000 employees is the maximal way to achieve profit.
> but without profit, people can't get paid, and they'll stop building
I wholeheartledly disagree. People build things all the time for things other than profit. In fact, most of the greatest things ever built were a loss for those who built them.
Dignity is the best motivator. Profit only supercedes dignity when dignity is not on offer.
Lichess, is it bad? It basically solves the whole problem. If well-designed distributed social media site could be something like that. Donations are enough to support one guy at least.
I totally get/relate to your perspective, but to be the annoying leftie in your ear:
A) Sustainable revenue is a requirement for any company, yes, but the unlimited (above-inflation) growth demanded by most large corporations is absolutely not. Lots and lots of companies operate for a long time without expecting massive growth, raises n' all. MBAs pejoratively call such companies "lifestyle businesses"--as in "just pays for people to live"--but I'd call them "normal, healthy companies".
B) More fundamentally: the idea that a social media network can only be built by a single corporation owned by investors is an omnipresent, yet extremely toxic, assumption. Mastodon represents another extreme end of the capital<->labor spectrum where anyone can contribute to the network at any time with their own instance, but I think Bluesky is a hint of a less-pure--and therefor more feasible--future.
To use the language of my favorite dream, Chomskian Anarcho-Syndicalism: imagine a social media network organized by a democratic non-profit entity akin to the Python or Linux Foundations, that then contracts out work to a hierarchy of smaller, purpose-built teams ("syndicates"), each of which may in turn contract w/ other teams. Each team would have to attract talent and negotiate enough income to pay them sufficiently still, of course, but there would be no team leader to make a surplus profit from the system -- any "surplus" would stay at the non-profit level, and thus necessarily be reinvested back into the product.
In the current system, the reason Bluesky didn't do this off the bat is obvious: no one would loan them startup funds, as ownership investment is the de facto universal way to start up an unproven venture. But we can dream bigger and better, IMHO; both on a smaller scale by building upon already-proven open protocols like AT Proto, and on a larger scale by structuring the state & economy to support this kind of model equally, if not primarily.
All of the big tech companies today are the result of 100s of smaller, well intentioned tech companies that got acquired into these behemoths.
I always look at how WhatsApp played out as the company. They were the good guys, and didn't want to get acquired. Zuckerberg, almost bankrupt FB at the time giving into all of the ridiculous demands WhatsApp made. No one at WhatsApp thought it was going to happen, until it did and did result in a once-in-a-lifetime transfer of wealth to several hundred employees.
WhatsApp was acquired for $20 billion in cash and stock. Facebook was worth $170 billion at the time. The stock part of the acquisition wouldn’t have any real financial impact on Facebook.
If Bluesky ever gets close to becoming a serious threat to Meta's walled garden, the effort to fight back against them will take a lot of capital. Just the legal battles alone will cost a fortune.
Wikipedia isn't a threat to anyone, they just have to generate enough capital to exist.
Yes! If the venture capitalists that are already involved stick to their stated principles and don't demand eternal growth (which... fingers crossed?), I think bsky has an extremely feasible, promising future.
They've intentionally kept a low footprint to keep expenses down, and while income via donation is out of the picture (unless AT Proto grows into a full ecosystem, I suppose?), cosmetics are a tried-and-true model for supporting something that most users use for free, but that some power users spend all day on and want shiny stuff for. They'll probably end up exploring Discord-esque paywalled features for power users as well, which isn't necessarily ideal but is leagues better than getting on the currently-dying vicious cycle of Display Ads, IMO.
Alternative reading: It took 17 years for venture capitalists to figure out the pay-for-attention model, at which point they immediately applied it to everything they could.
Nah, the pay-for-attention model (the bluetick thing) was Musk; other than dating/hookup sites (and even then, they usually _ration_ it) I think it’s the only significant social media site to do it. No VCs involved.
It’s not like it’s a new idea; dating sites have been trying it since the 90s. But it, ah, a _bad_ model; it is why any popular post on Twitter is now drowned in irrelevant replies from idiot blueticks. It’s a reasonable rule of thumb that, if you have to pay to have your posts amplified, they’re probably not good enough for anyone to want to read them.
> That was the second moment I thought, uh, nope. This is literally repeating all the mistakes we made as a company. This is not a protocol that’s truly decentralized. It’s another app. It’s another app that’s just kind of following in Twitter’s footsteps, but for a different part of the population.
> Everything we wanted around decentralization, everything we wanted in terms of an open source protocol, suddenly became a company with VCs and a board. That’s not what I wanted, that’s not what I intended to help create.
Your actions' consequences are not limited to benefiting from the thing like it would for a product - with social media, you improve the networking effect for the soon-to-be bad. (Nothing against bluesky, I don't know or think it will do so)
Not everything good becomes bad. That premise is wrong.
Bluesky accepted VC money. For a social platform that means its death certificate has already been signed.
What you're ignoring with that framing is that we can use social media that operates outside the VC startup pipeline and doesn't have enshittification baked in from the start.
The fact that you have to be on “the” relay to meaningfully participate on the network.
If you instead claim that users can always choose to use other 3P relays, then you immediately lose all the nice things that Bluesky is able to do well today (search, discoverability, a “discover” algorithm). Indeed, you fall back to the same old problems that every other decentralized social network has.
Bluesky is just a shittier version of Nostr, except that the people over at Nostr don’t pretend.
It's not that it is "better" but that the choice is individual, not up to the mastodon server. In Mastodon, you trade Elon for some other group of individuals, so what happens if they make decision on moderation or content you do not agree with?
ATProto is designed around accounts that are independent of data host, application, and moderation, all in the name of giving users individual control over these things. It's like if every Mastodon user ran their own server, but without the overhead
> does not account for users banned from Bluesky by Bluesky for breaking the ToS or PDS abuse.
Then you are missing the point. I am asking how much censorship power the largest node in the network has.
If being blocked by the largest provider means 95% of users can't see me anymore then the situation is strictly worse than Mastodon vs ActivityPub-at-large.
What did you do to get banned? Most people who make this claim are unwilling to share the behavior that led to it. Bans are not arbitrary, nor is anyone obligated to host or show your speech
ATProto is young compared to the alternatives, give the network time to build out an ecosystem and the resilience we need.
First you made incorrect and misleading statements. Then when I point it out you try to rhetorically shift blame on to me.
To answer your misleading and irrelevant question, I have never been banned by Bluesky. What did you do to get banned on Mastodon? Do you understand that "moderation" and "banned for violating the ToS" are one and the same?
ATProto is not any better than ActivityPub (ie Mastodon) regarding the impact of network effects on moderation. The "big few" retain their outsized influence with both protocols, at least as far as I am aware. If there is a technical misunderstanding about this on my part then do please point it out.
Moreover, the current situation is significantly worse on ATProto because the vast majority of the network currently relies on a single provider.
To be clear, I am not opposed to ATProto. I am only opposed to inaccurate technical comparisons.
The question remains misleading and irrelevant whether it was addressed to me or the room at large.
I am well aware that ATProto makes improvements to both account migration and end user moderation. Those are of course good things in my view. As previously stated I have noting against ATProto.
This entire subthread started because I responded to your claim that "It's like if every Mastodon user ran their own server". That is highly misleading at best, outright wrong at worst. It is only equivalent to that scenario if every ATProto user manages their own account as well as every single other required service, at which point you might as well just have run your own Mastodon instance as it likely would have been both simpler and less resource intensive.
The reality is that this is never expected to happen. ATProto is expected to be like Mastodon, with the improvements that you can share a single account between multiple servers and there are tools to apply additional filtering so that you don't have to see (as much of) things you don't like. Those are certainly welcome improvements but they do nothing to address the outsized effects on censorship that large nodes have in federated networks.
Purely in theory, you could simultaneously use a single account with a full set of politically correct ATProto services (such as Bluesky) and a competing set of politically incorrect ones operated by someone else. However, if history is anything to go off of then this will quickly get you banned by either one or both of the operators. Merely interacting with those guilty of wrongthink has always been an easy way to get kicked off of a surprisingly large number of Mastodon servers.
Even supposing that the above doesn't happen to get you banned by either node, the potential still remains that it could. Whether or not a large node chooses to exercise its outsized influence doesn't change the fact that it possesses it. ATProto doesn't fix that.
Every user in ATProto gets their own database that amounts to a tar file (technically sqlite with car export format)
This is nothing like having a single server for every user. Perhaps you are confusing Bluesky (one app) with ATProtocol the shared network? There are already independent servers and apps operating separate from Bluesky
Create an account you own instead of having someone else run it. Maybe you can get your data, maybe you can ask Bluesky for a review (there were bugs and scaling issues against bot networks that cause false positives)
I'm not seeing that handle resolve in the normal places. Do you have the DID? You should use a custom domain so that you can control the the reference and lookup.
You can run your own PDS and manage complete account lifecycle
So after you're banned from Bluesky you create another account on a different server and hope the admins of your original server, which still hosts all the people you want to follow, don't block your new account from interacting with their server?
You said it was different from Mastodon, but how is this different from Mastodon?
To be fair, with ActivityPub your account is always tied to a given server.
However as you have rightly noticed, the person you are responding to here is being disingenuous. Controlling the account doesn't fix the fundamental issue caused by network effects. An account that can't interact with a large portion of the network is no different than Mastoson in practice.
The Mastodon equivalent would be running your own single user instance which gets defederated by one of the big players.
Twitter was always... not great (there's a reason it was affectionately known as the Hellsite), but it had 16 years of being _tolerable_ for most people (the real exodus only really started with Musk's changes, though there had been a couple of smaller ones previously, mostly over Twitter messing with the API).
Frankly, if I get 16 years out of Bluesky before having to move onto the next one, I can live with that. Social networks _die_; it has always been so. USENET, livejournal, Tumblr, twitter... nothing lasts forever.
My bluesky feed is somehow even more abhorrent than my twitter one, except that instead of right wing hate it's Facebook memes about "reading banned books"
Bluesky's 'Discover' feed (the default algo feed that you get when you create your account) is based on _likes_, not follows, so if you never like anything you'll get random nonsense.
You can try using other algo feeds from here: https://bsky.app/feeds and remove the discover feed, or of course you could just use the chronological one.
Isn't that the whole point of bs? Empowering users to take their data where they want. It's completely open-sourced and well-documented. If someone buys bluesky you can move all your data to a different service that follows the same protocol
But the users are not. The app and the AppView can be bought and then you are in control of what people can see. Good luck trying to move them to the alternative.
Can I move my followers/following graph as well? Moving the actual content is barely a consolation prize if you lose your entire audience in the process.
I honestly am annoyed to use websites and services like this. Annoys the crap out of me and everyone else, but since it's petty much forced down their throats, the "eventually" is "eventually everyone stops complaining".
I wonder why timelines aren't implemented as a hybrid gather-scatter choosing strategy depending on account popularity (a combination of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served).
When you have a celebrity account, instead of fanning out every message to millions of followers' timelines, it would be cheaper to do nothing when the celebrity posts, and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline. When millions of followers do that, it will be cheap read-only fetch from a hot cache.
This is probably what we'll end up with in the long-run. Things have been fast enough without it (aside from this issue) but there's a lot of low-hanging fruit for Timelines architecture updates. We're spread pretty thin from a engineering-hours standpoint atm so there's a lot of intense prioritization going on.
Just to be clear, you are a Bluesky engineer, right?
off-topic: how has been dealing with the influx of new users after X political/legals problems aftermath? Did you see an increase in toxicity around the network? And how has you (Bluesky moderation) dealing with it.
Maybe this would be helpful:http://daslab.seas.harvard.edu/datacalculator/
That's insightful. Keep up the good work!
> and later when serving each follower's timeline, fetch the celebrity's posts and merge them into the timeline
I think then you still have the 'weird user who follows hundreds of thousands of people' problem, just at read time instead of write time. It's unclear that this is _better_, though, yeah, caching might help. But if you follow every celeb on Bluesky (and I guarantee you this user exists) you'd be looking at fetching and merging _thousands_ of timelines (again, I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users).
Given the nature of the service, making read predictably cheap and writes potentially expensive (which seems to be the way they've gone) seems like a defensible practice.
> I suppose you could just throw up your hands and say "not doing that", and just skip most or all of the celebs for problem users
Random sampling? It's not as though the user needs thousands of posts returned for a single fetch. Scrolling down and seeing some stuff that's not in chronological order seems like an acceptable tradeoff.
You might mix the approaches based on some cut off point
This problem is discussed in the beginning of the Designing Data-Intensive Applications book. It's worth a read!
Do you know the name of the problem or strategy used for solving the problem? I'd be interested in looking it up!
I own DDIA but after a few chapters of how database work behind the scenes, I begin to fall asleep. I have trouble understanding how to apply the knowledge to my work but this seems like a useful thing with a more clear application.
Yes, we used the Yahoo! “Feeding Frenzy” paper as the basis for the design of Haplocheirus (the timeline service).
Why do they "insert" even non-celebrity posts into each follower's timeline? That is not intuitive to me.
To serve a user timeline in single-digit milliseconds, it is not practical for a data store to load each item in a different place. Even with an index, the index itself can be contiguous in disk, but the payload is scattered all over the place if you keep it in a single large table.
Instead, you can drastically speed up performance if you are able to store data for each timeline somewhat contiguously on disk.
Think of it as pre-rendering. Of pre-rendering and JIT collecting, pre-rendering means more work but it's async, and it means the timeline is ready whenever a user requests it, to give a fast user experience.
(Although I don't understand the "non-celebrity" part of your comment -- the timeline contains (pointers to) posts from whoever someone follows, and doesn't care who those people are.)
Perhaps I misunderstanding, I thought the actual content of each tweet was being duplicated to every single timeline who followed the author, which sounded extremely wasteful, especially in the case of someone who has 200 million followers.
From the linked article: "Additionally, a reference to your post is 'fanned out' to your followers so they can see it in their Timelines."
So not the content, just a sort of link to it.
At some point they'll end up just doing the Bieber rack [1]. It's when a shard becomes so hot that it just has to be its own thing entirely.
[1] - https://www.themarysue.com/twitter-justin-bieber-servers/
@bluesky devs, don't feel ashamed for doing this. It's exactly how to scale these kinds of extreme cases.
I've stood up machines for this before I did not know they had a name, and I worked at the mouse company and my parking spot was two over from a J. Beibe'rs spot.
So now we have Slashdot effect, HN hug, and its not Clarkson its... Stephen Fry effect? Maybe can be Cross-Discipline - there's a term for when lots of UK turns their kettles on at the same time.
I should make a blog post to record all the ones I can remember.
TV Pickup aka the Half Time Kettle Effect.
https://en.wikipedia.org/wiki/TV_pickup
Given that BlueSky is funded by Twitter, I'm assuming they know a lot more than us on how Twitter architects systems.
We never actually had a literal “Bieber Box”, but the joke took off.
Hot shards were definitely an issue, though.
Its so crazy.
Thanks a lot for sharing this link.
As a systems enthusiast I enjoy articles like this. It is really easy to get into the mindset of "this must be perfect".
In the Blekko search engine back end we built an index that was 'eventually consistent' which allowed updates to the index to be propagated to the user facing index more quickly, at the expense that two users doing the exact same query would get slightly different results. If they kept doing those same queries they would eventually get the exact same results.
Systems like this bring in a lot of control systems theory because they have the potential to oscillate if there is positive feedback (and in search engines that positive feedback comes from the ranker which is looking at which link you clicked and giving it a higher weight) and it is important that they not go crazy. Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly.
Reading this description of how user's timelines are sharded and the same sorts of feedback loops (in this case 'likes' or 'reposts') sounds like a pretty interesting problem space to explore.
I guess I hadn’t considered that search engines could be reranking pages on the fly as I click them. I’ve been seeing my DuckDuckGo results shuffle around for a while now thinking it’s an awful bug.
Like I click one page, don’t find what I want, and go back thinking “no, I want that other result that was below” and it’s an entirely different page with shuffled results, missing the one that I think might have been good.
That's connected with a basic usability complaint about current web interfaces, that ads and recommended content aren't stable. You very well might want to engage with an ad after you are done engaging what you wanted to engage with but you might never see it again. Similarly, you might see two or three videos that you want to click on on the side of a YouTube video you're watching but you can only click on one (though if you are thinking ahead you can open these in another tab.)
On top of that immediate frustration, the YouTube style interface here
https://marvelpresentssalo.com/wp-content/uploads/2015/09/id...
collects terrible data for recommendations because, even though it gives them information that you liked the thumbnail for a video, they can't come to any conclusion about whether or not you liked any of the other videos. TikTok, by focusing on one video at a time, collects much better information.
> though if you are thinking ahead you can open these in another tab
or add it to the "Watch Later" playlist :) so you can watch it...later.
I don't use DDG, but in my (very limited, just now) testing it doesn't seem to shuffle results unless you reload the page in some way. Is it possible you're browser is reloading the page when you go back? If so, setting DDG to open links in new tabs might fix this problem.
Interesting. Maybe something in my configuration is affecting it. I’ll have to look into it
This behavior started happening for me in the last few months. If I click on a result, then go back, I have different search results.
I've found a workaround, though – click back into the DDG search box at the top of the page and hit enter. This then returns the original search results.
Hi - I work on search at DuckDuckGo. Do you mind sharing a bit more detail about this issue? What steps would allow us to reproduce what you're seeing?
> Some of the most interesting, and most subtle, algorithm work was done keeping that system "critically damped" so that it would converge quickly.
Looking back at my early work with microservices I'm wondering how much time I would have saved by just manually setting a tongue weight.
Similar to how Google images loads lower quality blurred thumbnails towards the bottom of the window at first so that the user thinks they loaded faster
This is less a question of perfection and one of trade off's. Laws of physics put a limit on how efficiently you can keep data in NYC and London in perfect sync, so you choose CAP-style trade-offs. There are also $/SLO trade-offs. Each 9 costs more money.
I like your example it is very interesting. If I get to work on (or even hear someone in my team is working on) such interesting problems and I can hear about it, I get happy.
Interesting problems are rare because like a house you might talk about brick vs. Timber frame once, but you'll talk about cleaning the house every week!
Would you be willing to share more about how you guys did click ranking at Blekko? It's an interesting problem.
What became of Blekko?
> It was acquired by IBM in March 2015, and the service was discontinued.
— https://en.wikipedia.org/wiki/Blekko
Perhaps GP has a more interesting answer though.
That's the correct answer, IBM wanted the crawler mostly to feed Watson. Building a full search engine (crawler, indexer, ranker, API, web application) for the English language was a hell of an accomplishment but by the time Blekko was acquired Google was paying out tens of billions of dollars to people to send them and only them their search queries. For a service that nominally has to live on advertising revenue getting humans to use it was the only way to be net profitable, and you can't spend billions buying traffic and hope to make it back on advertising as the #3 search engine in the English speaking markets.
There are other ways to monetize search (look at Kagi for example) than advertising. Blekko missed that window though. (too early, Google needed to get a crappy as it is today to make the value of a spam free search engine desirable)
Blekko was gone by the time I learned about it. Recently (past few years) I emailed someone who worked on Blekko to get his opinion on a search engine concept I still have yet to start. His advice was to not bother competing with Google (obviously) LOL!
I don’t know if anyone’s embarked on a P2P search engine but that’s essentially my concept. Anyhoo, thanks for the inspiration!
Peer to peer would be tough, you really need a 10G network connection to some tier 1 provider, and about 2500 machines to distribute the crawling/serving load. (that is if you want to do a full stack search engine). And while you can run that infrastructure for on the order of $100K/month (not counting depreciation) that means you need roughly $5K/day in revenue from that cluster. At $10 RPM ($10 revenue per thousand queries) you're looking at a minimum of 500,000 'real' search queries during 'English time' (roughly 7AM to 11PM GMT). That's 31,250 queries per hour or ~9 queries per second (average).
And that just pays to keep the lights on at the colocation center. If you're paying off the development costs (30 - 50 developers over 2 - 3 years) and the cost of an office somewhere. You'll want at least double that revenue or you'll go broke before you break even.
Ideally you are the 'go to' place for people looking to buy something as those queries make money. People researching Douglas Fairbanks for a high school essay consume queries but don't generate ad revenue.
It isn't for the faint of heart.
When you don't know what you don't know...wow.
I know "search is hard" in the general sense but context is lacking (not a lot of details online from ex-search teams). It's always been apparent to me that you must have some other high-grossing product if you want to get into search or video, if only to pay for the servers.
Thank you for providing your context!
Darknet Lantern is a decentralized searchable directory. It's probably not going to take off, but it could inspire something else. Servers spider other servers with the same software, and synchronized their data.
Yup, directory services are a lot easier to do peer-to-peer. Pinboard.in is a good shared directory (sort of Yahoo! without the editorial). They can yield excellent quality when you're searching for something that someone has 'indexed' with them, but poor recall when it comes to the set of all possible answers.
Doing it peer to peer without editorial allows sites to 'get into' the index easily which has its own plusses and minuses.
I’ve never heard of this before but it looks interesting. Thanks for the tip!
Not my Q but thanks for the interesting history.
Also, (for other readers), I'm a huge fan of Kagi. Highly recommended.
I really thought Neeva was gonna make it. I'm glad Kagi swooped in when they exited.
PID techniques useful?
I'm a bit confused.
The lossy timeline solution basically means you skip updating the feed for some people who are above the number of reasonable followers. I get that
Seeing them get 96% improvements is insane, does that mean they have a ton of users following an unreasonable number of people or do they just have a very low number for reasonable followers. I doubt it's the latter since that would mean a lot of people would be missing updates.
How is it possible to get such massive improvements when you're only skipping a presumably small % of people per new post?
EDIT: nvm, I rethought about it, the issue is that a single user with millions of follows will constantly be written to which will slow down the fanout service when a celebrity makes a post since you're going through many db pages.
When a system gets "overloaded", typically it enters exponential degradation of performance state, i.e. performs self ddos.
> Seeing them get 96% improvements is insane
TFA is talking about P99 tail latencies. It does not sound too insane to reduce tail latencies by extraordinary margins. Remember, it's just reshaping of latency distribution. In this case pathological cases get dropped.
> does that mean they have a ton of users following an unreasonable number of people
Look at the accounts of OnlyFans models, crypto influencers, etc. They follow thousands or even tens of thousands of accounts in the hope that we will follow them in return.
I don't see that accommodating this behavior is prosocial or technically desirable.
Can you think of a use case?
All sorts of bots want this sort of access, but whether there are legitimate reasons to grant it to them on a non-sharded basis is another question since a lot of these queries do not scale resources with O(n) even on a centralized server architecture.
Given enough time, you'll end up with a lot of legitimate users who follow a huge number of accounts but rarely interact with more than a handful, similar to how many long-time YouTubers have a very high subscriber:viewer ratio (that is, they have way more subscribers than you would expect given their average view count), and there's nothing inherently suspicious about it. People lose access to their accounts, make new accounts, die, get bored, or otherwise stop watching the content but never bother unsubscribing because the algorithm recognized this and stopped recommending the channel's uploads to them.
Bluesky doesn't have this problem yet because it's so young, so the outsized follow counts are mostly going to be from doomscrollers and outright malicious users, but even if it was exclusively malicious users, there is no perfect algorithm to identify them, much less do so before they start causing performance problems. Under those constraints, it makes sense to limit the potential blast radius and keep the site more usable for everyone.
From TFA:
> Generally, this can be dealt with via policy and moderation to prevent abusive users from causing outsized load on systems, but these processes take time and can be imperfect.
So it’s a case of the engineers accepting that, however hard they try to moderate, these sorts of cases will crop up and they may as well design their infrastructure to handle them.
> does that mean they have a ton of users following an unreasonable number of people
They do, there are groups of users on bluesky who follow inordinate numbers of other accounts to try and get follows back.
They were specifically looking at worst-case performance. P99 means 99th percentile, so they saw 96% improvement on the longest 1% of jobs.
Ok I'm curious: since this strategy sacrifices consistency, has anyone thoughts about something that is not full fan-out on reads or on writes ?
Let's imagine something like this: instead of writing to every user's timeline, it is written once for each shard containing at least one follower. This caps the fan-out at write time to hundreds of shards. At read time, getting the content for a given users reads that hot slice and filters actual followers. It definitely has more load but
- the read is still colocated inside the shard, so latency remains low
- for mega-followers the page will not see older entries anyway
There are of course other considerations, but I'm curious about what the load for something like that would look like (and I don't have the data nor infrastructure to test it)
Hmm. Twitter/X appears to do this at quite a low number, as the "Following" tab is incredibly lossy (some users are permanently missing) at only 1,200 followed people.
It's insanely frustrating.
Hopefully you're adjusting the lossy-ness weighting and cut-off by whether a user is active at any particular time? Because, otherwise, applying this rule, if the cap is set too low, is a very bad UX in my experience x_x
> It's _insanely_ frustrating.
> at only 1,200 followed people.
I follow like, 50 people on bluesky. Who is following 1,200 people? What kind of value do you even get out of your feed?
1200 people is really nothing, specially if you have a job tangentially related to social media (for example journalists). It's really simple, you are not the same type of user. You have 50 "acquaintances", they have 1200 "sources".
The article is talking about people who have following/follower counts in the millions. Those are dozens of writes per second in one feed and a fannout of potentially millions. Someone with 1200 followers, if everyone actually posts once a day (most people do not) gets... a rate of 0.138 writes per second.
They should be background noise, irrelevant to the discussion. That level of work is within reasonable expectation. What they're pointing out is that Twitter is aggressively anti-perfectionist for no good technical reason - so there must be a business reason for it.
Why are you following 1,200 people? What is the point of your home feed? What are you trying to see?
You are on the verge of discovering why non-chronological timelines exist :) It's not hard to imagine that there are 1200 people posting at least one thing a week that you would find interesting. The trouble is, if they also post 100 things that are not interesting, how does the software surface the interesting stuff without drowning you in the non-interesting stuff? How do you do that in a way that feels fair to the user ("I never see Friend X's posts because they're drowned out by the interesting stuff posted by the other 1199 people I follow")? It's tough!
Personally, I always use chronological. I like to be able to hop on, and mingle with whatever percent of people are online and posting at any particular time.
Replying to people right after they post is how you actually get to have conversations with people, and get to know them well, imo
They're all people that I know, and 98% of them are mutuals. I regularly go through my list of accounts I'm following every 5-6 months, and get rid of people who have disconnected from me, have gone inactive, or I don't really know.
I think the part you're missing here is that there are certain parts of twitter where the density of interconnection is really high, so you'll know people because you see them in the comments of a lot of threads, or through retweets.
It's really not that hard to end up knowing 1,000+ people if you engage with a group or "corner" of twitter. Even moreso if it's multiple corners. (Like, AI, but also just frens, but also a little political corner, people from specific cities, etc.)
> What are you trying to see?
A scrollable feed of accounts that post interesting (to me) content.
there are a lot of interesting people on twitter
I can come up with 100 people I'd want to follow on Twitter, and I don't even have an account. Don't dismiss other people's use-cases if you don't have or understand them.
> Additionally, beyond this point, it is reasonable for us to not necessarily have a perfect chronology of everything posted by the many thousands of users they follow, but provide enough content that the Timeline always has something new.
While I'm fine with the solution, the wording of this sentence led me to believe that the solution was going to be imperfect chronology, not dropped posts in your feed.
So, let's say I follow 4k people in the example and have a 50% drop rate. It seems a bit weird that if all (4k - 1) accounts I follow end up posting nothing in a day, that I STILL have a 50% chance that I won't see the 1 account that posts in a day. It seems to me that the algorithm should consider my feed's age (or the post freshness of my followers). Am I overthinking?
This feels like an edge case.
The "reasonable limit" is likely set based on experimentation, and thus on how much people post on average and the load it generates (so the real number is unlikely to be exactly "2000", IMHO).
If you follow a lot of people, how likely it is that their posting pattern is so different from the average? The more people you follow, the less likely that is.
So while you can end up in such situation in theory, it would need to be a very unusual (and rare) case.
I think the 'law of large numbers' says that it's very unlikely for you to follow 4k and have _none_ of them posting. You could artificially construct a counter-example by finding 4k open but silent accounts, but that's silly.
The other workaround is: follow everyone. Write some code to get what you want out of the jetstream event feed. https://docs.bsky.app/blog/jetstream
Yeah, this seems concerning to me. Maybe now as the platform is new this isn't much of an issue. But as accounts go inactive people will naturally collect "dead" accounts that they are still following. On Facebook it isn't uncommon of to have old accounts of sociable people naturally collect thousands of friends.
It seems that what they are trying to measure is "busy timelines" and it seems bag they could probably measure that more directly. For example what is the number of posts in the timeline over theast 24h? It seems that it should be fairly easy to use this as the metric for calculating drop rate.
Love reading these sorts of "technical problem + solution" pieces. The world does not need more content, in general, but it does need more of this kind of quality information sharing.
Anyone following hundreds of thousands of users is obviously a bot account scraping content. I'd ban them and call it a day.
However, I do love reading about the technical challenge. I think Twitter has a special architecture for celebrities with millions of followers. Given Bluesky is a quasi-clone, I wonder why they did not follow in these footsteps.
You don't need to follow anyone (or even have an account) to scrape content… Someone following a huge amount of accounts usually wants to get a lot of followers quickly this way through follow-backs.
Yes, and Starter Packs make this possible.
> Given Bluesky is a quasi-clone, I wonder why they did not follow in these footsteps.
There are only six users with over a million followers, and none with two million yet.
I'm sure they'll get there.
Maybe not hundreds of thousands but I'd follow anybody that looks remotely interesting and then primarily use customized feeds. E.g. if I wanna hear about union news, my personal irl network, etc I check that feed
if you want to scrape all the content, that's what the firehose is for, and it's allowed.
the only reason to mass-follow is for spam purposes.
This does assume that scrapers are smart, and often they're really not. They have infrastructure for scraping HTML from webpages at scale and that is the hammer they use for all nails. (e.g. Wikipedia has to fight off scraper traffic despite full archives being available as torrents, etc.)
In this case I agree though, they're all spammers and/or "clout farmers", or trying to make an account seem more authentic for future scams. They want to generate follow notifications in the hope that some will follow them back (and if they don't, they unfollow again after some interval).
100%. I ran a job board where we provided a nice machine readable XML feed of all of our jobs, but we had bots that insisted on using the standard search box. Searching by city using an alphabetized list.
Geographic search to was the most expensive thing they could have done and no matter what we did we couldn’t get them to use the XML feed.
I even tried returning a link to the feed when we detected a bot. No dice. They just kept working around the bot detection.
BlueSky has starter packs that allow you to mass follow in the click of a button. You join 10 starter packs in one day, you are following over 1000 people. Sometimes following others is the only way to get people to engage with your content.
Or just enforce a maximum number of followed accounts.
No matter how high you set a maximum limit for interactions on social media (followers, friends, posts, etc), someone will reach the limit and complain about it. I can see why Bluesky would prefer a "soft limit", where going above the limit will degrade the experience. It gives more flexibility to adjust things later, and prevents obnoxious complaints from power users with outsized influence.
I’m skeptical that the people who would complain about that wouldn’t find something else to complain about if you resolved the first complaint. I’d recommend implementing product features that you think are reasonable and accepting the fact that you will get complaints from people who disagree.
Potential solutions:
- Make it easy to systematically unfollow people (or degrade them to a different tier, see below, or sort them automatically into a different feed; maybe even allow automatic following of certain people, like your cities mayor or local ice cream parlors). Like based on recent activity, last engagement with a post, type of content (pictures, videos, links ...), on a schedule (e.g. follow for 3 yeard, follow until 2028), special status (family, friends, member of congress, member of city council, mayor...), number/ratio of common followers, regex expressions, recommendations by certain accounts, letter-to-word ratio, season, planetary alignment, weather, age, train departure time, side-chaing based on other accounts, force accounts to play russian unfollow roulette, urgency to pee, healthcare CEO life expectancy derivative, ... or any combination of these.
- Allow different tiers of following someone. Like friends (never unfollow, always fetch updates), family (never unfollow, rate limit high-energy uncles), news (filter based on urgency or current topics of interest), politicians (highlight as untrustworthy, attach link to donation and board membership disclosure, attach term-limit and next election countdown), local businesses (hard rate limit, attach opening hours), bookmark (never unfollow, no updates), ... maybe multiple tiers in each category and allow those being followed to either temporarily boost their tier (or tiers of certain posts) or e.g. once per year.
- Allow people from exempting some of their posts from not being shown to some of their followers. E.g. two per week and an additional 5 per month.
- Allow people to choose which followers should be given a higher priority when writing posts to their feeds.
Nice problem to have, though. Over on Nostr they're finding it a real struggle to get to the point where you're confident you won't miss replies to your own notes, let alone replies from other people in threads you haven't interacted with.
The current solution is for everyone to use the same few relays, which is basically a polite nod to Bluesky's architecture. The long-term solution is—well it involves a lot of relay hint dropping and a reliance on Japanese levels of acuity when it comes to picking up on hints (among clinets). But (a) it's proving extreme slow going and (b) it only aims to mitigate the "global as relates to me" problem.
AWS has a cool general approach to this problem (one badly behaving user effecting others on their shard)
https://aws.amazon.com/builders-library/workload-isolation-u...
The basic idea is to assign each user to multiple shards, decreasing the changes of another user sharing all their shards with the badly behaving user.
Fixing this issue as described in the article makes sense, but if they did shuffle sharding in the first place it would cover any new issues without effecting many other users.
I think shuffle sharding is beneficial for read-only replica cases, not for writing scenarios like this. You'll have to write to the primary and not to a "virtual node". Right? Or am I understand it incorrectly? I just read that article now.
When I go directly to a user's profile and see all their posts, sometimes one of their posts isn't in my timeline where it should be. I follow less than 100 users on Bluesky, but I guess this explains why I occasionally don't see a user's post in my timeline.
Lossy indeed.
If another user you follow reposted or replied to a post, it can affect its order in your following feed. You shouldn't be seeing any loss as described in the article from following only 100 users.
I've experienced it with "first-party" posts, not replies. A post wouldn't show in my timeline but would on the user's profile. This is the official android app, but there has been an update or two so I'll have to double check again
Are you using an app, website, or combination?
Various clients (I’m writing one) interpret the timeline differently, as a feed that shows literally everything includes could things that most people would find undesirable or irrelevant. (replies to strangers, replies to replies to replies, etc)
I'm using the official android app. There has been an update or two so I'll have to confirm it's still happening
I am a bit perplexed though as to why they have implemented fan-out in a way that each "page" is blocking fetching further pages, they would not have been affected by the high tail latencies if they had not done this,
"In the case of timelines, each “page” of followers is 10,000 users large and each “page” must be fanned out before we fetch the next page. This means that our slowest writes will hold up the fetching and Fanout of the next page."
Basically means that they block on each page, process all the items on the page, and then move on to the next page. Why wouldn't you rather decouple page fetcher and the processing of the pages?
A page fetching activity should be able to continuously keep fetching further set of followers one after another and should not wait for each of the items in the page to be updated to continue.
Something that comes to mind would be to have a fetcher component that fetches pages, stores each page in S3 and publishes the metadata (content) and the S3 location to a queue (SQS) that can be consumed by timeline publishers which can scale independently based on load. You can control the concurrency in this system much better, and you could also partition based on the shards with another system like Kafka by utilizing the shards as keys in the queue to even "slow down" the work without having to effectively drop tweets from timelines (timelines are eventually consistent regardless).
I feel like I'm missing something and there's a valid reason to do it this way.
I interpreted this as a batch write, e.g. "write these 10k entries and then come back". The benefit of that is way less overhead versus 10k concurrent background routines each writing individual rows to the DB. The downside is, as you've noted, that you can't "stream" new writes in as older ones finish.
There's a tradeoff here between batch size and concurrency, but perhaps they've already benchmarked it and "single-threaded" batches of 10k writes performed best.
I found it odd to base the loss-factor on the number of people you follow, rather than a truer indication of timeline-update-frequency. What if I follow 4k accounts, but each of those accounts only posts once a decade? My timeline would be become unnecessarily lossy.
The funny thing is that all of the centralization in Bluesky is defended as being necessary to provide things like global search and all replies in a thread, things that Mastodon simply punts on in the name of decentralization. But then ultimately, Bluesky has to relax those goals after all.
True. In context though Bluesky can tweak the volume knob as and when they see fit, whereas for Mastodon it's stuck where it is.
This design makes sense if you didn’t previously have any limit on the number of people an account could follow. But why not have a limit?
people get so up in arms when you suggest there might be a limit on how many people they can follow.
Interesting! I wonder what value they chose for the `reasonable_limit`.
It's 4k: https://bsky.app/profile/jaz.bsky.social/post/3likncbqutk2y
ought to be possible to reverse-engineer it by following a large number of active accounts and seeing what percentage of their posts actually hit your feed
An interesting solution to a challenging problem. Thank you for sharing it.
I must admit, I had some trouble following the author's transition from "celebrity" with many followers to "bot" with many follows. While I assume the work done for a celebrity to scatter a bunch of posts would be symmetric to the work done for a commensurate bot to gather a bunch of posts, I had the impression that the author was introducing an entirely different concept in "Lossy Timelines."
That’s quite interesting and a challenge I have not thought of. I understand the need for a solution and I believe this works reasonably well, but I am wondering what is happening to users that follow a lot of accounts with below-average activity. This may naturally happen on new social media platforms with people trying out the service and possibly abandoning it.
The „reasonable limit“ is likely set to account for such an effect, but I am wondering if a per-user limit based on the activity of the accounts one follows will be an improvement on this approach.
To help avoid the hot shard problem, I wonder how capping followers per "timeline" would perform. Especially each user would have a separate timeline per 1000 followers and the client would merge them. You could still do the lossy part, if necessary, by only loading a percent of the actual timelines. That wouldn't help the celebrity problem but it was already acknowledged earlier that the solution to that is to not fan out celebrity accounts.
In the fanout design, why not dynamically move on to the next 10,000-user page as soon as all tasks for the current page are either queued or processing? Would that approach improve throughput, or could it introduce issues like resource contention?
On a related note, I am pretty confident that one of the main reasons the WWW succeeded where previous attempts failed was that it very specifically allowed 404s.
A simpler option is to put a limit on the number of accounts one’s can follow. Who needs to follow more than 4k followers if not bots?
The solution to this problem is known and implemented already: the social web should be distributed between thousands of pods which should contain at the maximum a few thousands users. Diaspora is already working like this for 15 years. It is technically harder to build initially but it then divide all the problems (maintenance, moderation, load, censorship, trust of the owner...) Which makes the network much more resilient. Bluesky knows that and they are allowing other people to host their software but they are really not pushing for it and it highly doubt that the experience of a user on a small external pod is the same than on bluesky.com.
This particular problem will still exist for a fediverse server. You follow 10k people? Nice, now you're getting ddos'd by their activities. Though, most fediverse servers being monolithic applications definitely helps.
> some of them will do abnormal things like… well… following hundreds of thousands of other users.
Sounds like Bluesky Pro.
the use of fan-out to followers and a lazy fetch of popular followed accounts when follower's timeline is served a good implementations in hot reload scenarios
I think something like this was a FB engineering interview (several years ago), just for instagram feeds.
Are users informed that they follow too many creators and now they will not see every post on their timelines?
The typical problem of a centralized infrastructure.
Indeed:
> This means each user gets their own Timeline partition, randomly distributed among shards of our horizontally scalable database (ScyllaDB), replicated across multiple shards for high availability
"Lossy timelines" have already been implemented in ActivityPub and Mastodon by design. Will Bluesky ever catch up? It remains to be seen.
Principle: Progress over perfection.
Note that all of this reflects design decisions on Bluesky's closed-source "AppView" server—any federated servers interacting with Bluesky would need to construct their own timelines, and do not get the benefit of the work described here.
As others have noted, the appview is open source. The dataplane has two implementations, one in postgres and another in scylla. The scylla dataplane is closed, the postgres one is open.
The interesting next stage for the postgres implementation is to create a sync engine for partial syncs of the network, so that an appview can run affordably. We ran some benches on the current state of the postgres implementation and found we could index 300k users on a $100/mo vps. I think with a couple of weeks of optimization that could reach 1mm users.
This is great to hear—my current understanding of the most recent state of the art on the topic is https://alice.bsky.sh/post/3laega7icmi2q which mentions that the self-hosted appview is not yet open source. So I'm glad to hear the situation has changed in the past 3 months.
It was open source (except the Scylla database layer) from the beginning, AFAIK - that blog post just says that they haven't set it up yet, because that's the hardest part to run
This is not true. Third party PDSes are fully supported by our app view, and our app view generates timelines for all the users on those PDSes.
What does this have to do with third party app views?
The statement "any federated servers interacting with Bluesky" is ambiguous, because Bluesky's federated model means there's many different types of servers, and one user's view of what a "federated server" could be vastly different from another.
Federated PDS-s (which is probably the closest to what people mean when they say they want to federate on bluesky) would not need to reconstruct timelines if their users use the bsky.app appview.
Thanks, that's a fair point that I was overlooking. When I say a "federated server", I don't just mean a self-hosted PDS, I mean a third party app that potentially has its own lexicon and design decisions. Creating a robust third-party app that can meaningfully interact with the Bluesky network is still a very difficult engineering challenge, which I think this article does a good job demonstrating—that was the tension I was trying to underscore in my comment. Bluesky may be solving those engineering challenges for those clients who are satisfied with Bluesky's frontend and AppView, but every single other app built on top of ATProto will have to resolve those same challenges. This is directly downstream from Bluesky's "global firehose" topology and various design decisions that stem from that.
You didn't specify what kind of federated servers you were thinking about
What reason does Bluesky give for not opening up their AppView code?
Another notable component that is closed source is the discovery feed generator, where at least there is some reason.
I asked this and got
> We did a backend rewrite from postgres to scylla and it has a bunch of deployment specific stuff, but is functionally identical to the open source postgres version. Its not really a "v2" in terms of new features, we just made it make use of our hardware really well[1]
[1]: https://bsky.app/profile/iame.li/post/3l7e3jfqit22s
Thanks, so are both the Postgres and Scylla versions maintained in terms of new features?
I wasn't aware that AppView v1 was open source, and the most recent info I'm aware of on the topic is https://alice.bsky.sh/post/3laega7icmi2q, https://github.com/bluesky-social/atproto/discussions/2961 and https://docs.bsky.app/docs/advanced-guides/federation-archit..., and everything I've heard about Bluesky was that open source appview is "still coming".
It's not coming, it never went away… As I understand it, the "business layer" with all the logic is above the data later, shared by the Postgres and Scylla versions, and the data layer just makes queries to the database. I think they are using the Postgres version locally for development.
The App View frontend is open source: https://github.com/bluesky-social/social-app
Much of the backend is open source as well: https://github.com/bluesky-social/atproto/tree/main/packages
What is not are the extra services they run to provide a better and faster UX. Even if it was open source, it likely costs 10s of thousands to run per month (they have moved largely to "onprem" hardware instead of the cloud aiui)
That's the frontend code, it doesn't include the backend API services, which are closed source.
the backend (the AppView) can be found here:
https://github.com/bluesky-social/atproto/tree/main/packages...
there are various supporting services written in Go as well
https://github.com/bluesky-social/indigo
Which is what I said in the second sentence
AppView is a specific term of art within the Bluesky federation architecture: https://atproto.com/guides/glossary#app-view, you were incorrect in identifying the public frontend repo as the AppView.
The glossary specifically calls out the UI as part of an app view. Can you explain why it is not according to you?
A frontend is (can be) part of an App View. It is quite literally the app you view the network through. There can also be headless app views and app views which have no backend
this is not correct
that's not the appview, that's the client
App View is a bit fuzzy of a term. To me it seems like a combination of frontend, backend, custom lexicon, and supporting services. There isn't really another place in the spec or design where clients or browsers fit in, which do in fact provide a view of the network via an app.
"UI" is part of the definition they give in the glossary
https://atproto.com/guides/glossary#app-view
when I read the spec it seemed like the operator of an AppView & Relay would be most in need of compensation for their hosting costs due to the amount of demand on those components so I believe the spec allows an operator to implement their own AppView & monetize it as that operator sees fit, so that they can afford to operate the service and maybe even make money off of it so that they can make it their full time jobs.
It seems this way to me as well. ATProto fundamentally changes how monetization works in social media by removing lockin. It's going to be interesting to see what emerges from this design decision.
Another interesting way to view ATProto is that it could be a collection of headless features and network browsers that leverage those feature providers.
what else? profit by means of doing work that benefits first and foremost the private proprietors of the closed source
if they gave it away (which used to be unfeasible until the digital era) they feel they’re loosing their valuable effort which they’re wont on concentrating, not diluting.
My thinking has evolved on this topic significantly as of late. My current thinking is we should create a secure gossip network on top of the Bluesky API, and forgot about all the DAG-CBOR stuff that gets stripped from the Jetstream. Hash the posts on the gossip layer and if posts change then diff them. This is all prep for when X billionaire buys out Bluesky then we just pop some signing key crypto on top of this gossip layer and wow! It's distributed!
isnt that ssb?
reverse-ssb
So the system design puts the burden on what seems to be synchronous, not queued, writes to get easy reads. I usually prefer simpler cheaper writes at the cost of more complicated reads as the reads scale and parallelize better.
you're underestimating the read load, by a lot
I understand that it's a different point, but how can someone write a whole essay called "When imperfect systems are good" without once mentioning Gabriel or https://en.wikipedia.org/wiki/Worse_is_better?
Anecdotally, I ran into a similar solution "by chance".
Long ago, I worked for a dating site. Our CTO at the time was a "guest of honor" who was brought in by a family friend who was working in the marketing at the time. The CTO was a university professor who took on a job as a courtesy (he didn't need the money nor fame, he had enough of both, and actually liked teaching).
But he instituted a lot of experimental practices in the company. S.a. switching roles every now and then (anyone in the company could apply for a different role except administration and try themselves wearing a different hat), or having company-wide discussions of problems where employees would have to prepare a presentation on their current work (that was very unusual at the time, but the practice became more institutional in larger companies afterwards).
Once he announced a contest for the problem he was trying to solve. Since we were building a dating site, the obvious problem was matching. The problem was that the more properties there were to match on, the longer it would take (beside other problems that is). So, the program was punishing site users who took time to fill out the questionnaires as well as they could and favored the "slackers".
I didn't have any bright ideas on how to optimize the matching / search for matches. So, ironically, I asked "what if we just threw away properties beyond certain threshold randomly?" I was surprised that my idea received any traction at all. And the answer was along the lines of "that would definitely work, but I wouldn't know how to explain this behavior to the users". Which, at the time, I took to be yet another eccentricity of the old man... but hey, the idea stuck with me for a long time!
The answer to that reply is you don't need to explain it to your users. People are used to fuzzy/best-effort sort of matching, especially when it's specifically presented as a "matching algorithm" instead of a "filter".
> This process involves looking up all of your followers, then inserting a new row into each of their Timeline tables in reverse chronological order with a reference to your post.
Seriously? Isn't this the nut of your problem right here?
What alternative design did you have in mind, given that a Twitter-like data model of individual follows is likely a strict product requirement?
There are obviously other ways of doing it (doing the timeline propagation in a batch job, fanning out the reads rather than the writes), but they've got their own problems. Probably worse ones.
Wouldn't a hybrid approach makes sense?
Periodically classify users as hot/cold based on their activity, build hot-follower timelines on write, and build cold-follower timelines on read.
You'd still have exactly the same hot write path, it'd just have maybe 50% of the load. That could be a legit optimization, but not having it hardly warrants an incredulous "seriously?" like the OP's.
(And the same for the inverse hybrid strategy of quarantining the writes of highly followed users and handling their fan-out at read time. A neat optimization, and maybe even absolutely once you have accounts with 100M followers. But the vast majority of posts would still be handled via the original strategy.)
An airline reservation system has to be perfect (no slack in today's skies), a hotel reservation can be 98% perfect so long as there is some slack and you don't mind putting somebody up in a better room than they paid for from time to time.
A social media system doesn't need to be perfect at all. It was clear to me from the beginning that Bluesky's feeds aren't very fast, not like they are crazy slow, but if it saves money or effort it's no problem if notifications are delayed 30s.
It's funny because from my experience airline systems are very imperfect (timing wise).
I (unwisely) tried to purchase an Icelandair ticket via the Chase travel portal. I would get a reservation number, go buy seats on Icelandair's website, and a few days later the entire reservation would vanish into the ether. Rinse and repeat 3x.
I can't remember the exact verbiage, but basically tickets can be "reserved" and "booked". One means the ticket is allocated, and one means the ticket is actually paid for. I eventually sat on the phone with an executive support person as they booked the ticket and got it all the way through. It turns out Chase reserves a ticket on an airline but as an SLA of ~3 days to actually pay for the ticket. Icelandair's requires a ticket to be paid with in 24 hours, so it was timing out.
(Replying to both you and the parent poster)
Airlines are far from perfect. They overbook flights and sometimes have to ask people leave and pay them for the inconvenience. My wife and I once got $1000 a piece and a hotel and food voucher to volunteer to take a flight the next day on a layover in Atlanta.
As far as your particular situation, the number one rule of using a third party portal to book flights or hotels is - don’t.
I understand that Iceland Air is not a transfer partner of Chase. But even in that case, I would just wait to use my points until I could use a transfer partner.
On the earning side if paying cash, the difference between 2x/3x points when booking directly and 5x when going through the portal just isn’t worth the risk.
Overbooking is not a mistake, though. People miss flights for many reasons, and the airlines predict this with impressive accuracy, to the point that they can afford to pay tremendous sums for being wrong and yet still come out ahead.
> afford to pay tremendous sums for being wrong
Or they can just haul you forcefully from the flight you paid for:
https://en.wikipedia.org/wiki/2017_United_Express_passenger_...
Case in point! Imagine how much United(Express) settled this for.
This is ridiculous. There must have been some amount of money that would have convinced four people to deplane voluntarily. $800 was ridiculously low.
Especially for a free service!
Think about other ad-supported sites. If you're an engineer working on an ad-supported product, the perfect consistency you strive for in your code is not the product. The product is the sum of all of the content the user sees. And the costs of the tradeoffs you make are paid for by ads.
Am I willing to see 10x more ads for perfect consistency? Definitely not.
Does the fact that an airline booking system must be perfect explain why so many flights are overbooked or cancelled?
No, overbooking is a business decision justified by the fact that, statistically, not all passengers will actually show up for their flight, and lower load factors cost money.
What is the "no show" rate?
A 2019 study of 5 European airports in 2019 had no-show rates of 14.4%: https://www.ozion-airport.com/product/comparative-analysis-n...
However, my understanding is that airlines have much more sophisticated per-flight and per-passenger models that calculate the predicted no-show factor based on the historical rates for that particular route (e.g. you're more likely to get more no-shows in business class flying from NYC to SF compared to holiday travelers with a reservation on the Florida Keys)
That blows my mind, I would expect maybe 1 or 2 passengers per plane at most. I'm trying to think of what factors would cause that many no-shows, it has to be mostly missed connections?
I can't imagine spending hundreds of dollars and just not showing up.
Delays getting to the airport and missing the plane. Cancelations with full refund. "Hidden city" ticketing. Layover delays. Businesses booking blocks. Flexible flights ( https://www.travelperk.com/guides/flexible-travel/flexible-f... ). Changing / rebooking flights for an earlier or later time.
A friend of mine works for a Management Consultancy firm and they have full flex tickets if they miss the 8pm flight home they can take the next one or fly back the next morning. All without additional fees. So I believe business travel is the biggest factor when it comes to missed flights.
Side note: His employer is the biggest client of a major European airline.
No additional fees, but the cost of the ticket is typically sky-high. In many cases the company could purchase 3 restricted-fare tickets for less than the unrestricted fare. (And also, the consultants want to get home, they HATE staying extra nights, so they have inherent motivation to catch the original flight). You'd think that management consulting firms, you know, who are supposed to be good at optimization and reducing costs, would have figured this out.
* A massive bulk of flex rates is different price than a single flex ticket
* when I travelled to a single city with 20+ colleagues for several years, our nightly hotel rate was less than quarter of published rate. I don't think we got anything like that on Airlines, but nevertheless, bulk gets discounts.
* I forget details but between specific frequent cities which had hourly flights, we had full flexibility and we used this all the time. So we might catch a flight hour earlier if there was one available, or hour later if need be than booked.
Basically to everybody's point, business travel is very different than vacation travel and intuition from once a year personal trip don't apply.
* Hotels are required by law to have a published "rack rate" and it's common for both leisure and business travelers to book at more than half, or more, off those rates. Airlines don't have that.
* Yes of course there are negotiated discounts for major corporations- but full flexibility is still far more expensive than restricted tickets, just like business is still far more expensive than economy.
* Most airlines offer guaranteed same-day changes, or at least priority standby, to high-status loyalty members (which most consultants are) so buying full-flex tickets to get an hour earlier/later flexibility is redundant - basically, paying for a benefit you already have.
... or perhaps they prioritize ensuring their consultants feel like they're being looked after - replacing them when they find another job is very expensive.
What part of my comment suggests consultants aren't "looked after?" The idea that the consultants' personal time is so low on the scale of priorities that the company prefers to pay 3x or 4x for plane tickets to ensure they are available to spend extra time with clients, rather than their families, suggests that the company is specifically NOT looking after the consultants' well-being.
That the consultants' personal time is high on the scale is why they pay for the flex-fare tickets. So you miss an 8pm flight for one reason or another - it happens, so "don't miss the flight" isn't a valid retort.
Do I feel better taken care of if I can just take the following morning's flight at no additional cost or hassle, or if I now need to contact someone at Amex Global Business Travel, have them try to get me on a flight, have another expense, potentially not be able to get on the next flight or the one after that, etc.
The company can start with heavily-discounted restricted tickets and eat them when a consultant needs to extend the trip. Today it's even easier because airlines have eliminated change fees - so essentially any ticket can be canceled and those funds credited to a future flight. It's all still "no cost or hassle" to the consultant since they aren't paying for the flights to begin with.
Yeah, replace consultants with clients in that comment and you’ll have the right of it.
They have figured it out, the client is paying for it.
This is actually the most sensible response
> You'd think that management consulting firms, you know, who are supposed to be good at optimization and reducing costs, would have figured this out.
Maybe they've figured out that enabling their employees to stay a few extra hours without worry to finish a deal is worth it.
Similar reasons people buy overnight sleeper tickets from the west coast to asia for sky-high prices. That person has to be there, and they have to be rested. If they are negotiating 100m+ dollar contracts its a no-brainer.
Intercontinental business-class tickets have a tangible benefit, guaranteed lie-down seat and relaxing accommodations that enable rest, as you correctly point out. Flexible airline tickets have no tangible benefit, it is simply risk mitigation (avoiding wasting a nonrefundable ticket if a person can't make a flight) and like other risk-mitigation products such as insurance, extended warranties, car rental protection, etc, it is priced so that the cost exceeds the benefit.
And as stated elsewhere, the majority of consultants are relatively junior people whose role has nothing to do with negotiating contracts.
Vast majority of consultants are working on engagements where "the deal" was negotiated weeks/months earlier.
The goal isn't to eliminate flexibility, it's understanding probabilities. If fully-flexible/refundable flights are 3x the fare of restricted, then in aggregate, the company could have a chunk of consultants throw out their original reservations and rebook later flights, and still save money. Yes- sometimes consultants need to stay longer than planned, but in an age of prioritizing "work-life balance," most consultants are encouraged to stick to their schedule and get back home when originally planned.
They aren’t following the same rules as you, they’re on negotiated rates/agreements.
Keep in mind they sell a lot of tickets where one of the features that allows for a premium price is that they allow late cancellations or changes to other flights. Holiday travelers are pretty "reliable", but business travelers might have changed needs at the drop of a hat (say you meet another prospective client on a business trip and decide to stay another day to fit in a face-to-face meeting).
European airports in 2019: A lot of these would be 10 euro Ryanair/Easyjet/Whatever flights, probably.
(The really ultra cheap Ryanair flights are less of a thing now, but in 2019 they were very much a factor)
Remember lots of business travelers have connections or flex schedules. When I had to go to the West coast for business, I usually have full fare tickets and book a later flight. If I had flexibility, I’d switch to an earlier plane or first class.
I'm sure other factors such as sudden illness and migrateable tickets make a sizeable chunk too.
I think you'll have to pay a team millions to figure that out, it is unlikely to be a static rate but rather decided based on multiple traits like time of year, time of flight, distance of flight, cost of ticket, etc.
The airline has literally all of the data on this, they definitely do not have to pay a team millions.
They probably do pay millions of dollars in wages for business analysts to figure out what this rate is on their flights.
They probably just have an SSRS report that prints out in a few dozen offices automatically on some schedule.
I'm not trying to be pedantic but this is table stakes stuff. I know we're supposed to shy away from saying things like this but compared to the other engineering that airlines have to do, this is easy. It costs - at most, including wages - a few tens of thousands of dollars yearly to come up with these figures. It's a fraction of the salary of one United Airlines BA.[0] This cost might go up if one of the senior developers convinces their boss that this needs to be a machine learning model but unless they're resume pumping it's going to be at most PCA and a regression.
This is not a team of people working for months on this one thing.
[0] https://www.glassdoor.com/job-listing/analyst-revenue-manage...
> airline reservation system has to be perfect (no slack in today's skies)
The slack just gets moved. Airlines oversell by about 8 percent. All systems need some slack in them. Isn't that kinda Bob's Law or something?
Miscommunication leads to bad outcomes. One missed message out of order could easily lead to a fight, a lawsuit, a flash mob, threats of violence - that then need to be taken seriously, swatting, DOXxing, etc...
Msg 1: I hate ___insert_controversal_person_category_here___
Msg 2: Is the kind of statement that really sets me off
Msg 1 has a very different meaning if you don't see Msg 2.
This can already happen without help from the platform.
Sure, but that doesn't mean the platform should make it worse.
Trying to have a conversation on flaky platform is hell.
It’s really impressive how well Bluesky is performing. It really feels like a throwback to older social media platforms with its simplicity and lack of dark-patterns. I’m concerned that all the great work on the platform, protocol, etc won’t shine in the long term as they eventually need to find a revenue source.
I love Mastodon but I have to admit that BlueSky has clearly out-engineered them. Of course they started with much more expertise and resources. I hope ActivityPub compatibility soon to unite the two
They've done an incredible job running with an extremely low headcount and crazy efficient use of hardware. It would be easy to 10x their expenses if they were blindly following the standard cloud deployment playbook. Hopefully this level of efficiency mean they don't have to work as hard and can stay pre-revenue, a pure play, for a very long time.
Absolutely. The profit motive is the root of most evil. It is a shame that so many are trained to believe it is the only motive available.
I completely agree with this... but without profit, people can't get paid, and they'll stop building. I do hate this incredibly need for growth, of course, but financial growth is necessary to pay people and give them raises and allow them to have upward mobility at the company.
I hope Bluesky is able to find a model that works for them AND for consumers. (I do know it's an open protocol, so it'll live on without Bluesky itself! However, as this post shows, it's a lot of work to build on the prototype... so if not them, who? And if someone else, how will they become sustainable?)
It's semantics but I like to separate money from profits. You need money to pay people and to survive but you don't need to be raking in endlessly growing piles of it. This is something that was really demoralizing about working for a big company, they could be making like 50000000000 a year in just profits but still be ruthless in getting more. Like I just want to make a product I'm proud of and I'm happy living a simple life, I am happier now making less money but not feeling like I'm endlessly milking customers.
That is not semantics. It is fundamental economics.
Once you take on investors, that’s not an option. VCs expect rapid growth and an exit - statistically through acquisition, but occasionally an IPO.
Once you go public, then you have investor pressure and can be subject to activist investors unless the founder has controlling interests like in the case of Meta and Google.
Basically why I don't like to work for any public company anymore. You are there to be exploited by moneyed interests that most often don't even have any positive effect on the products itself.
At the same time I feel like a lot of companies grow much larger than they need to be simply because of bigger is better mentality. How many of Uber's 30,000ish employees are involved with making sure the app and backend database are working properly? Are they really doing 600 times more work than Craigslist at connecting sellers with buyers?
I'm an Uber hater, but... yes.
Like, sure, they don't need every single one of those 30,000... but they have to have ground teams in every city in the world. Connections with every airport. Connections with almost every restaurant in the world. Customer support and safety (okay I know they don't nail this, but still). They need to pay out drivers in each country. The app needs to work in hundreds of countries, all with different laws, currencies, languages and more. Some places let you pick up anywhere, others require specific locations. And that's not even including marketing, partnerships, HR, finance, etc.
I don't think the employees are the problem with Uber, it's the shareholders. They need to make X back, so that delta is where drivers get squeezed.
You cannot compare uber to Craigslist.
Uber takes on so much more responsibility of the transaction. Setting price, handling disputes, real time coordination, etc.
Aren't you actually arguing in favor of profit-driven behavior? You're not disagreeing with profit as a motivator, you're questioning if the 30,000 employees is the maximal way to achieve profit.
> but without profit, people can't get paid, and they'll stop building
I wholeheartledly disagree. People build things all the time for things other than profit. In fact, most of the greatest things ever built were a loss for those who built them.
Dignity is the best motivator. Profit only supercedes dignity when dignity is not on offer.
Profit supercedes dignity when one needs to eat, because one cannot eat dignity.
Being able to spend a significant amount of time and effort on passion projects is a luxury most people can't afford.
Yes, but there is a path, and it's simplicity.
Lichess, is it bad? It basically solves the whole problem. If well-designed distributed social media site could be something like that. Donations are enough to support one guy at least.
I totally get/relate to your perspective, but to be the annoying leftie in your ear:
A) Sustainable revenue is a requirement for any company, yes, but the unlimited (above-inflation) growth demanded by most large corporations is absolutely not. Lots and lots of companies operate for a long time without expecting massive growth, raises n' all. MBAs pejoratively call such companies "lifestyle businesses"--as in "just pays for people to live"--but I'd call them "normal, healthy companies".
B) More fundamentally: the idea that a social media network can only be built by a single corporation owned by investors is an omnipresent, yet extremely toxic, assumption. Mastodon represents another extreme end of the capital<->labor spectrum where anyone can contribute to the network at any time with their own instance, but I think Bluesky is a hint of a less-pure--and therefor more feasible--future.
To use the language of my favorite dream, Chomskian Anarcho-Syndicalism: imagine a social media network organized by a democratic non-profit entity akin to the Python or Linux Foundations, that then contracts out work to a hierarchy of smaller, purpose-built teams ("syndicates"), each of which may in turn contract w/ other teams. Each team would have to attract talent and negotiate enough income to pay them sufficiently still, of course, but there would be no team leader to make a surplus profit from the system -- any "surplus" would stay at the non-profit level, and thus necessarily be reinvested back into the product.
In the current system, the reason Bluesky didn't do this off the bat is obvious: no one would loan them startup funds, as ownership investment is the de facto universal way to start up an unproven venture. But we can dream bigger and better, IMHO; both on a smaller scale by building upon already-proven open protocols like AT Proto, and on a larger scale by structuring the state & economy to support this kind of model equally, if not primarily.
All of the big tech companies today are the result of 100s of smaller, well intentioned tech companies that got acquired into these behemoths.
I always look at how WhatsApp played out as the company. They were the good guys, and didn't want to get acquired. Zuckerberg, almost bankrupt FB at the time giving into all of the ridiculous demands WhatsApp made. No one at WhatsApp thought it was going to happen, until it did and did result in a once-in-a-lifetime transfer of wealth to several hundred employees.
WhatsApp was acquired for $20 billion in cash and stock. Facebook was worth $170 billion at the time. The stock part of the acquisition wouldn’t have any real financial impact on Facebook.
On the other hand, running something like BlueSky is not terribly expensive. A foundation with a reasonable endowment can do that indefinitely.
Initially, it can be funded by selling tools that do analytics or by donations (like Wikipedia).
If Bluesky ever gets close to becoming a serious threat to Meta's walled garden, the effort to fight back against them will take a lot of capital. Just the legal battles alone will cost a fortune.
Wikipedia isn't a threat to anyone, they just have to generate enough capital to exist.
Yes! If the venture capitalists that are already involved stick to their stated principles and don't demand eternal growth (which... fingers crossed?), I think bsky has an extremely feasible, promising future.
They've intentionally kept a low footprint to keep expenses down, and while income via donation is out of the picture (unless AT Proto grows into a full ecosystem, I suppose?), cosmetics are a tried-and-true model for supporting something that most users use for free, but that some power users spend all day on and want shiny stuff for. They'll probably end up exploring Discord-esque paywalled features for power users as well, which isn't necessarily ideal but is leagues better than getting on the currently-dying vicious cycle of Display Ads, IMO.
This is hardly the first instance of "if only venture capitalists light their money on fire, we can have nice things."
Spoiler: Venture capitalists don't light their money on fire, and we can't have nice things.
There's no reason Bluesky has emulate what FB Newsfeed and Twitter/X did to solve engagement by promoting certain items over others.
At the very least, they do have hindsight to learn from.
There's the profit motive. It's funded by venture capital, so it has to grow at all costs (check) and then cash out.
Twitter didn't start doing it (the pay-for-attention model) until 17 years after it was founded, so, I mean, bsky will probably be okay for a _while_.
Alternative reading: It took 17 years for venture capitalists to figure out the pay-for-attention model, at which point they immediately applied it to everything they could.
Nah, the pay-for-attention model (the bluetick thing) was Musk; other than dating/hookup sites (and even then, they usually _ration_ it) I think it’s the only significant social media site to do it. No VCs involved.
It’s not like it’s a new idea; dating sites have been trying it since the 90s. But it, ah, a _bad_ model; it is why any popular post on Twitter is now drowned in irrelevant replies from idiot blueticks. It’s a reasonable rule of thumb that, if you have to pay to have your posts amplified, they’re probably not good enough for anyone to want to read them.
Bluesky is a private for-profit company that has taken $37M in venture capital.
https://www.piratewires.com/p/interview-with-jack-dorsey-mik...
> That was the second moment I thought, uh, nope. This is literally repeating all the mistakes we made as a company. This is not a protocol that’s truly decentralized. It’s another app. It’s another app that’s just kind of following in Twitter’s footsteps, but for a different part of the population.
> Everything we wanted around decentralization, everything we wanted in terms of an open source protocol, suddenly became a company with VCs and a board. That’s not what I wanted, that’s not what I intended to help create.
"Hot Shards in Your Area" - 10/10 heading
[stub for offtopicness]
I don’t understand the infatuation with blue sky. The minute they need money it’ll go the way of the Reddit and twitter.
If everything good is assumed to eventually become bad, why not use things while they are good and then immediately move on when it becomes bad?
Your actions' consequences are not limited to benefiting from the thing like it would for a product - with social media, you improve the networking effect for the soon-to-be bad. (Nothing against bluesky, I don't know or think it will do so)
Not everything good becomes bad. That premise is wrong.
Bluesky accepted VC money. For a social platform that means its death certificate has already been signed.
What you're ignoring with that framing is that we can use social media that operates outside the VC startup pipeline and doesn't have enshittification baked in from the start.
People want the old Twitter, and Bluesky is close to that. It also cosplays being decentralized to people who don’t look too closely.
What makes it not decentralised?
https://dustycloud.org/blog/how-decentralized-is-bluesky/
The approximately a million dollars a year that it costs to run another copy.
The fact that you have to be on “the” relay to meaningfully participate on the network.
If you instead claim that users can always choose to use other 3P relays, then you immediately lose all the nice things that Bluesky is able to do well today (search, discoverability, a “discover” algorithm). Indeed, you fall back to the same old problems that every other decentralized social network has.
Bluesky is just a shittier version of Nostr, except that the people over at Nostr don’t pretend.
People seem to lark on and on about how it has better "default moderation" than Mastodon.
It's not that it is "better" but that the choice is individual, not up to the mastodon server. In Mastodon, you trade Elon for some other group of individuals, so what happens if they make decision on moderation or content you do not agree with?
ATProto is designed around accounts that are independent of data host, application, and moderation, all in the name of giving users individual control over these things. It's like if every Mastodon user ran their own server, but without the overhead
Are you suggesting the "big few" can't largely censor a given account?
I don't see how ATProto is doing noticeably better than the scenario where a large ActivityPub instance blocks your external account.
Generally, yes. Currently, because Bluesky requires the use of their labeler if you use their app, this could happen.
Two points of note
1. You can participate in Bluesky without the Bluesky app, so you can remove this requirement by using an alternative app
2. The most blocked account is blocked by around 0.25% of the full network (https://clearsky.app/)
This second point does not account for users banned from Bluesky by Bluesky for breaking the ToS or PDS abuse.
> does not account for users banned from Bluesky by Bluesky for breaking the ToS or PDS abuse.
Then you are missing the point. I am asking how much censorship power the largest node in the network has.
If being blocked by the largest provider means 95% of users can't see me anymore then the situation is strictly worse than Mastodon vs ActivityPub-at-large.
What did you do to get banned? Most people who make this claim are unwilling to share the behavior that led to it. Bans are not arbitrary, nor is anyone obligated to host or show your speech
ATProto is young compared to the alternatives, give the network time to build out an ecosystem and the resilience we need.
First you made incorrect and misleading statements. Then when I point it out you try to rhetorically shift blame on to me.
To answer your misleading and irrelevant question, I have never been banned by Bluesky. What did you do to get banned on Mastodon? Do you understand that "moderation" and "banned for violating the ToS" are one and the same?
ATProto is not any better than ActivityPub (ie Mastodon) regarding the impact of network effects on moderation. The "big few" retain their outsized influence with both protocols, at least as far as I am aware. If there is a technical misunderstanding about this on my part then do please point it out.
Moreover, the current situation is significantly worse on ATProto because the vast majority of the network currently relies on a single provider.
To be clear, I am not opposed to ATProto. I am only opposed to inaccurate technical comparisons.
I was using the indeterminate "you" for a hypothetical question, one that gets asked often
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
https://news.ycombinator.com/newsguidelines.html
ATProto moderation is technically and fundamentally different from ActivityPub: https://bsky.social/about/blog/03-12-2024-stackable-moderati...
The question remains misleading and irrelevant whether it was addressed to me or the room at large.
I am well aware that ATProto makes improvements to both account migration and end user moderation. Those are of course good things in my view. As previously stated I have noting against ATProto.
This entire subthread started because I responded to your claim that "It's like if every Mastodon user ran their own server". That is highly misleading at best, outright wrong at worst. It is only equivalent to that scenario if every ATProto user manages their own account as well as every single other required service, at which point you might as well just have run your own Mastodon instance as it likely would have been both simpler and less resource intensive.
The reality is that this is never expected to happen. ATProto is expected to be like Mastodon, with the improvements that you can share a single account between multiple servers and there are tools to apply additional filtering so that you don't have to see (as much of) things you don't like. Those are certainly welcome improvements but they do nothing to address the outsized effects on censorship that large nodes have in federated networks.
Purely in theory, you could simultaneously use a single account with a full set of politically correct ATProto services (such as Bluesky) and a competing set of politically incorrect ones operated by someone else. However, if history is anything to go off of then this will quickly get you banned by either one or both of the operators. Merely interacting with those guilty of wrongthink has always been an easy way to get kicked off of a surprisingly large number of Mastodon servers.
Even supposing that the above doesn't happen to get you banned by either node, the potential still remains that it could. Whether or not a large node chooses to exercise its outsized influence doesn't change the fact that it possesses it. ATProto doesn't fix that.
>It's like if every Mastodon user ran their own server
No, it's like every Mastodon user used the same server, and all the coordination is done by one server that nobody can replicate.
Every user in ATProto gets their own database that amounts to a tar file (technically sqlite with car export format)
This is nothing like having a single server for every user. Perhaps you are confusing Bluesky (one app) with ATProtocol the shared network? There are already independent servers and apps operating separate from Bluesky
You have the opportunity to demonstrate this. I am banned from Bluesky. (They didn't tell me why - just a generic "you violated community guidelines")
Tell me, concretely, how people can choose to continue following me, even though I am banned.
Profile: immibis.bsky.social
Create an account you own instead of having someone else run it. Maybe you can get your data, maybe you can ask Bluesky for a review (there were bugs and scaling issues against bot networks that cause false positives)
I'm not seeing that handle resolve in the normal places. Do you have the DID? You should use a custom domain so that you can control the the reference and lookup.
You can run your own PDS and manage complete account lifecycle
So after you're banned from Bluesky you create another account on a different server and hope the admins of your original server, which still hosts all the people you want to follow, don't block your new account from interacting with their server?
You said it was different from Mastodon, but how is this different from Mastodon?
I have my own domain already attached, I can point it at any server and my identity on the network remains the same.
When you use a *.bsky.social handle, you have not made yourself independent and resilient to arbitrary decision by the org that manages that service
So that's the same as on Mastodon. What's the difference again?
To be fair, with ActivityPub your account is always tied to a given server.
However as you have rightly noticed, the person you are responding to here is being disingenuous. Controlling the account doesn't fix the fundamental issue caused by network effects. An account that can't interact with a large portion of the network is no different than Mastoson in practice.
The Mastodon equivalent would be running your own single user instance which gets defederated by one of the big players.
Follow the instructions under "Self-hosting PDS" here: https://github.com/bluesky-social/pds
Twitter was always... not great (there's a reason it was affectionately known as the Hellsite), but it had 16 years of being _tolerable_ for most people (the real exodus only really started with Musk's changes, though there had been a couple of smaller ones previously, mostly over Twitter messing with the API).
Frankly, if I get 16 years out of Bluesky before having to move onto the next one, I can live with that. Social networks _die_; it has always been so. USENET, livejournal, Tumblr, twitter... nothing lasts forever.
My bluesky feed is somehow even more abhorrent than my twitter one, except that instead of right wing hate it's Facebook memes about "reading banned books"
Bluesky's 'Discover' feed (the default algo feed that you get when you create your account) is based on _likes_, not follows, so if you never like anything you'll get random nonsense.
You can try using other algo feeds from here: https://bsky.app/feeds and remove the discover feed, or of course you could just use the chronological one.
But when everything I see is awful, what do I like? I think that's my classic issue with feed style websites
You follow _people_, and ignore the 'Discover' feed, is generally my personal solution to this. YMMV.
I would be so much more interested in Bluesky if it were technically impossible for a random super rich guy to buy and bend it to his whims.
Isn't that the whole point of bs? Empowering users to take their data where they want. It's completely open-sourced and well-documented. If someone buys bluesky you can move all your data to a different service that follows the same protocol
You can move your data, but whoever controls the AppView controls what people see “on Bluesky”.
Yeah but anyone can make a GreenSky. The source code is all there
But the users are not. The app and the AppView can be bought and then you are in control of what people can see. Good luck trying to move them to the alternative.
Can I move my followers/following graph as well? Moving the actual content is barely a consolation prize if you lose your entire audience in the process.
Centrally-controlled social media platforms are not a good thing, period. Neither Twitter/X, nor BlueSky. Let's not fete them.
I honestly am annoyed to use websites and services like this. Annoys the crap out of me and everyone else, but since it's petty much forced down their throats, the "eventually" is "eventually everyone stops complaining".
Bluesky is the Conservative Dad Beer of "left" short form social media.
I implore everyone to use something better like Mastodon or maybe minds
"Hot Shards in Your Area"... I died
I don’t see much call for blusky anymore….
I mean: https://bskycharts.edavis.dev/static/dynazoom.html?cgiurl_gr...
Posts/sec are just off record levels.
Record seems to be ~115, while current is ~63, or 55%.
Yeah, there was a short spike directly after the US elections, but, ignoring that, post rates are stable to somewhat rising.
So it’s bluesky this week? What about mastodon? What about threads? I thought X was going to die.
I’ll just stay with X, that’s where everybody is, contrary to the bluesky cheerleaders here.