TII Releases Falcon 2-11B: The First AI Model of the Falcon 2 Family Trained on 5.5T Tokens with a Vision Language Model

The Technology Innovation Institute (TII) in Abu Dhabi has introduced Falcon, a cutting-edge family of language models available under the Apache 2.0 license. Falcon-40B is the inaugural “truly open” model, boasting capabilities on par with many proprietary alternatives. This development marks a significant advancement, offering many opportunities for practitioners, enthusiasts, and industries alike.

Falcon2-11B, crafted by the TII, is a causal decoder-only model boasting 11 billion parameters. It has been meticulously trained on a vast corpus exceeding 5 trillion tokens, amalgamating RefinedWeb data with meticulously curated corpora. This model is accessible under the TII Falcon License 2.0, a permissive software license inspired by Apache 2.0. Notably, the license includes an acceptable use policy, fostering the responsible utilization of AI technologies.

Falcon2-11B, a causal decoder-only model, is trained to predict the next token in a causal language modeling task. It’s based on the GPT-3 architecture but incorporates rotary positional embeddings, multiquery attention, FlashAttention-2, and parallel attention/MLP decoder-blocks, distinguishing it from the original GPT-3 model.

The Falcon family includes Falcon-40B and Falcon-7B models, with the former excelling on the Open LLM Leaderboard. Falcon-40B requires ~90GB GPU memory, still less than LLaMA-65B. Falcon-7B needs only ~15GB, enabling accessible inference and fine-tuning even on consumer hardware. TII offers instruct variants optimized for assistant-style tasks. Both models are trained on vast token datasets, predominantly from RefinedWeb, with publicly available extracts. They employ multiquery attention, enhancing inference scalability by reducing memory overheads. This design facilitates robust optimizations like statefulness, making Falcon models formidable contenders in the language model landscape.

Research advocates using large language models as a foundation for specialized tasks like summarization and chatbots. However, caution is urged against irresponsible or harmful use without thorough risk assessment. Falcon2-11B, trained on multiple languages, may not generalize well beyond them and can carry biases from web data. Recommendations include fine-tuning for specific tasks and implementing safeguards for responsible production use.

To recapitulate, the introduction of Falcon by the Technology Innovation Institute presents a groundbreaking advancement in the field of language models. Falcon-40B and Falcon-7B offer remarkable capabilities, with Falcon-40B leading the charge on the Open LLM Leaderboard. Falcon2-11B, with its innovative architecture and extensive training, further enriches the Falcon family. While these models hold immense potential for various applications, responsible usage is paramount. Vigilance against biases and risks, alongside conscientious fine-tuning for specific tasks, ensures their ethical and effective deployment across industries. Thus, Falcon models represent a promising frontier in AI innovation, poised to reshape numerous domains responsibly.