{"articles":[{"slug":"coheres-north-mini-code-megakernel-serving-engine","title":"Cohere's North Mini Code Megakernel Serving Engine","subtitle":null,"summary":"Today, Cohere presents a serving engine for North Mini Code built around a decode megakernel: BF16 on a single H100, 1.25× - 1.41× faster than vLLM end-to-end. Explore the code behind the serving engine on GitHub. Most LLM serving stacks still treat each forward pass as a sequence of kernels: launch QKV, wait; launch attention, wait; launch the MoE, wait. Each launch is fine on its own. The problem is the waiting in between. At small batch sizes, the GPU spends a surprising fraction of every decode step waiting rather than computing.","content_type":"blog_post","language":"en","canonical_url":"https://cohere.com/blog/megakernels","author":{"name":"Xiaochun Tong, Conway Zhu, Donglu Wang","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Cohere","url":"https://cohere.com/blog/","listing_slug":"cohere","listing":{"slug":"cohere","name":"Cohere","listing_type":"company","url":"https://listedstartups.com/companies/cohere"}},"topics":[{"name":"AI","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"LLMs","slug":"llms","url":"https://listedarticles.com/topics/llms"},{"name":"Performance","slug":"performance","url":"https://listedarticles.com/topics/performance"},{"name":"Machine Learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"},{"name":"Infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":5534,"reading_minutes":24,"published_at":"2026-09-08T12:00:00.000Z","added_at":"2026-09-19T00:10:02.879Z","updated_at":"2026-09-19T00:10:02.879Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":false},"profile_url":"https://listedarticles.com/articles/coheres-north-mini-code-megakernel-serving-engine","markdown_url":"https://listedarticles.com/articles/coheres-north-mini-code-megakernel-serving-engine.md","example":false,"citation":"Xiaochun Tong, Conway Zhu, Donglu Wang, Cohere. \"Cohere's North Mini Code Megakernel Serving Engine.\" 8 Sept 2026. https://cohere.com/blog/megakernels (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://cohere.com/blog/megakernels"},"snippet":null,"score":null}],"total":1,"count":1,"next_offset":null,"has_more":false,"query":{"q":null,"content_type":null,"topic":null,"publisher":"cohere","about":null,"author":null,"language":null,"sort":"newest","limit":20,"offset":0}}