This article shares details on two versions of Meta's 24,576-GPU data center scale cluster. These clusters support the company's current and next-generation AI models as well as AI research and development across GenAI and other areas. Meta's long-term vision is to build artificial general intelligence that is open and built responsibly to be widely available so that everyone can benefit from it. Its infrastructure is designed to power this ambition.
Wednesday, March 13, 2024The Vision Pro may just be an over-engineered devkit. There is some genius and audacity behind some of Apple's hardware decisions - gaze and pinch was a major industry 'a-ha' moment. However, the device's software/content story is dull and unimaginative and most people likely won't use the Vision Pro for watching media. The Vision Pro's debut was the best thing that ever happened to Meta's Quest brand as it elevated VR to a level of mainstream consumer curiosity, set new standards and pricing envelopes for VR devices, and created a competitor for Meta teams to chase.
Meta showed off a brief demo of how users will eventually be able to connect their accounts to the fediverse during the FediForum conference on Tuesday. The company's integration will let users share their posts across different platforms through Threads. It currently allows users to post to other services that interoperate through ActivityPub, but they can't view replies and can only see the likes they get, a fact Meta is working hard to change. A video showing how the integration currently works is available in the article.
Smartwatches weren't even mainstream yet when Facebook bought Oculus 10 years ago. VR is still not massively adopted in 2024, but it has gone mainstream several times. This article contains an overview of where Meta is heading next with the Quest 3, AI, smart glasses, and AR glasses. Meta is bringing its own generative AI features to its Ray-Ban smart glasses later this month. The glasses will use their onboard cameras to recognize things in the real world.
Meta's Threads has introduced developer documentation and a sign-up sheet for interested parties ahead of its API's public launch planned for June. The documentation details the API's current limitations and endpoints. Developers can use the API to track analytics and publish posts and media, among other things. Threads accounts are limited to 250 API-published posts and 1,000 replies every 24 hours, with a text post limit of 500 characters.
Meta's Threads has introduced developer documentation and a sign-up sheet for interested parties ahead of its API's public launch planned for June. The documentation details the API's current limitations and endpoints. Developers can use the API to track analytics and publish posts and media, among other things. Threads accounts are limited to 250 API-published posts and 1,000 replies every 24 hours, with a text post limit of 500 characters.
Meta's Threads has introduced developer documentation and a sign-up sheet for interested parties ahead of its API's public launch planned for June. The documentation details the API's current limitations and endpoints. Developers can use the API to track analytics and publish posts and media, among other things. Threads accounts are limited to 250 API-published posts and 1,000 replies every 24 hours, with a text post limit of 500 characters.
Meta plans to release an initial version of its next-generation Llama 3 large language model within the next month. The company will release a number of different models with different capabilities and versatilities during the course of the year. Llama 3 will be able to answer a wider range of questions compared to its predecessor, including questions regarding more controversial topics. Meta has not released any details about the model's size, but it is expected to have about 140 billion parameters - the biggest Llama 2 model has 70 billion.
The influence of the printing press on the dissemination of information and the formation of nation-states mirrors the impact of the Internet, particularly through Aggregators like Google and Meta. However, the emergence of generative AI challenges the Aggregator model by providing singular answers that reflect embedded worldviews, potentially undermining the universal appeal that underpins Aggregator economics and suggesting a shift towards personalized AI to maintain their dominance.
Tuesday, March 5, 2024This blog post from Meta outlines the infrastructure being used to train Llama 3. It talks through storage, networking, Pytorch, NCCL, and other improvements. This will lay the foundation for Meta's H100s coming online throughout the rest of this year.
AI advancement poses a significant threat to the adtech industry, with its ability to filter out ads, potentially reducing the $1 trillion in annual revenue that companies like Google, Meta, and TikTok currently enjoy. This series explores how AI disrupts the crucial ad inventory by appealing to consumers' desire for ad-free content and questions the stability of Big Tech's business models. The analysis delves into AI's effects on ad consumption, which has implications for major players like OpenAI, Microsoft, Apple, Meta, and Alphabet.
Meta has released a native PyTorch library for fine-tuning language models.
Meta is introducing AI features to its Ray-Ban smart glasses, including translation, and the identification of objects, animals, and monuments, with activation through voice commands.
Researchers from Meta have been teasing a new optimizer on X. They have now released the code along with various integrations. The optimizer has no LR schedule, which means you don’t need to know the full number of training steps beforehand. It has been shown empirically to work on a wide variety of problems including language models.
Meta has confirmed plans to release Llama 3, the next generation of its large language model for generative AI assistants, within the next month.
The development of AI, particularly large language models like GPT-3, is heavily reliant on vast amounts of data, with companies like Meta and Google racing to gather more as high-quality online data may run out by 2026. Tech giants are employing controversial methods, including using YouTube data and considering the purchase of publishers, to fuel their AI advancements. The use of 'synthetic' data is a potential solution, though it carries the risk of amplifying AI errors.
Meta has confirmed plans to release Llama 3, the next generation of its large language model for generative AI assistants, within the next month.
The development of AI, particularly large language models like GPT-3, is heavily reliant on vast amounts of data, with companies like Meta and Google racing to gather more as high-quality online data may run out by 2026. Tech giants are employing controversial methods, including using YouTube data and considering the purchase of publishers, to fuel their AI advancements. The use of 'synthetic' data is a potential solution, though it carries the risk of amplifying AI errors.
Meta has announced the next generation of its AI accelerator chip. Its development focused on chip memory (128GB at 5nm) and throughput (11 TFLOPs at int8).
Meta plans to release an initial version of its next-generation Llama 3 large language model within the next month. The company will release a number of different models with different capabilities and versatilities during the course of the year. Llama 3 will be able to answer a wider range of questions compared to its predecessor, including questions regarding more controversial topics. Meta has not released any details about the model's size, but it is expected to have about 140 billion parameters - the biggest Llama 2 model has 70 billion.
Wednesday, April 10, 2024SapFix is Meta’s deployment of automated end-to-end fault fixing. It makes debugging easy by automatically generating fixes for specific issues and proposing those fixes to engineers for approval and deployment to production. SapFix has been used for many of Meta’s core apps, each containing millions of lines of code. It uses automated program repair algorithms that identify, analyze, and patch known software bugs without human intervention.
Thursday, March 7, 2024Meta’s outage on Tuesday underscored the risk for businesses overly reliant on social media. It emphasized the need for backup plans like email databases and the importance of diversifying communication channels. Small business experts suggest leveraging outages to collect customer contact information.
Meta’s react-strict-dom (open sourced last week) is an experimental integration of React DOM and StyleX that standardizes the development of styled React components, both web and native. This article goes into the historical background of building a universal codebase that shares components between Web and Native and how react-strict-dom changes the way we can do it. You might need to refresh the page to get it to load!
Meta has announced new ads updates that primarily focus on retailers and those using its automated Advantage+ campaigns. Advantage+ creative optimizations can now automatically optimize video ads for viewing on Reels and dynamically create ad variations in order to personalize them to each user. The platform also offers new dynamic formats, including Reminder ads on Instagram, improved promo code ads, and ads with product tags.
Meta’s build systems follow a two-step process. The first step generates instructions for what to build and the second step executes the build. This separation makes them inflexible, as you can't add new files without re-running the first step. "End-to-end" (E2E) systems combine both steps, allowing for dynamic target discovery. In these systems, cloud caching, early cutoff, suspending schedulers, and minimal builds are all features easier to implement correctly.
Meta's video calling products rely on bandwidth estimation (BWE) and congestion control for optimal performance. Its hand-tuned system was complex and difficult to maintain, so its team developed an ML approach for network characterization and optimization, replacing hand-tuned rules to improve efficiency and reliability. The ML system analyzes network signals to classify network types and applies optimized settings for BWE and network resiliency.
Since March, Instagram and Threads have instituted a new default setting that limits political content you see from people you’re not following. Hundreds of creators, convened by GLAAD and Accountable Tech, have signed an open letter demanding that Instagram make the political content limit an opt-in feature, rather than on by default. Instagram describes political content as anything related to laws, elections, or social topics. For political creators, these limits impact their livelihood since they make it harder for them to reach new audiences.
OpenAI and Meta are teasing the next iterations of their AI models, expected to feature enhanced reasoning and planning capabilities. Dubbed GPT-5 and Llama 3, the models aim to advance toward artificial general intelligence, with vague release timelines and application details. The tech community remains skeptical given the history of overhyped AI promises with limited substantive evidence.
Meta's AI assistant is being integrated into search boxes within its apps. It will start appearing directly in the main Facebook feed and users can chat with it using Meta's messaging apps. The assistant is also accessible via a standalone website. It runs on Meta's new Llama 3 model, which outperforms competing models of its class on key benchmarks. The assistant integrates real-time search results from both Bing and Google and can generate images.
Meta has released an 8B and 70B model with dramatically improved performance, particularly in reasoning, context length, and code. It is still training a 400B parameter model, which will match Opus in performance. These models are easily the most powerful available open models.