Which model fits which computer

What Metis measures

In Setup, the Computer step measures this computer once, and nothing is installed. It reads these things:

• Graphics cards, each with its memory. A card is counted when the installed build drives it, and then it is marked “Runs models”. A card the build does not drive says “left out: the installed build runs NVIDIA cards only”. With no card at all, the step says “Metis found no graphics card.”
• The processor and how many threads it has.
• The memory, in GB.
• The disks, each with its free space.
• The network links and their speed.

Above the figures, Metis says how many graphics cards can run models and how much memory they have between them. Under “What was read, and where it went” are each card’s compute class, where each figure came from, and everything Metis could not read. Metis also times how fast this computer’s memory can be read. If that was not run, the step lists it among the figures that were not read.

Where a model can run

What decides whether a model fits is where its files can sit while it runs. Metis counts the memory of the graphics cards it can use, then the memory of the computer. Setup says where a model would run:

• “all of it on your graphics card, the fastest place it can be”.
• “spread across your graphics cards”.
• “on your graphics card with part of it in the computer’s memory, slower than one that fits the card alone”.
• “on the processor alone: slower, and no graphics card needed”.

A model that does not fit the card whole, and spills into the computer’s memory instead, is slower again. Metis keeps those for background work, and does not offer them as the model you talk to: “The models that fit this computer suit only background work, not Metis’s everyday work, so there is nothing here to download.”

On a computer with several cards, Setup’s model step has a closed “Advanced: one model across the cards”. Sharing one model across the cards can give it more room for a long chat; choosing one there starts Setup with that model spread across them.

In Setup

On the Model step, Metis offers the models that fit this computer. Each offer shows its name, about how many seconds it takes to answer, its size and how many pages of context it holds, with the badge “Fits this computer”, or “Tight fit” where part of it waits in memory. The first is “Recommended”, and the others are under “Other models that fit this computer”. “The numbers behind these” opens the technical figures.

When the models differ in what they are for, Setup groups them: “General tasks”, “Chatting” and “Coding”. A model that can look at pictures says so on its offer, “Can look at pictures”, as a note and not as a group. Each group shows its best first and folds the rest under “More for” that group. The largest model that fits whole on your card comes first, and a model that can use Metis’s tools comes before one that cannot. Each offer says which it is: “Calls tools”, “Tools not checked” where its file does not say, or “Chat only: can’t use Metis’s tools”, which means it can answer in a chat but cannot read, edit or run anything for you.

Terms of art are underlined, and show their meaning when you hover over them or tap them:

• “The context window is how much the model can hold in mind at once”: your question, the document you gave it, and its own answer so far. Anything older than the window falls out.
• “How fast it writes an answer” is decode. “How fast it reads what you gave it before it starts answering” is prefill.
• “The quantization is how tightly the model’s numbers are packed to fit in memory.”

If no model fits, Setup says “None of the models Metis knows fits this computer, so there is nothing to download.”

In Models

Models has four tabs: Your models, Computers, Get models and Tools. Get models lists models you can download, each with a fit badge: “Fits” (on your cards), “Tight fit” (part of it waits in memory), “Too big”, or “Fit not checked”.

• “Fit is worked out the way Layout places a model”: the Computers tab’s own placement, on your cards first, then in memory.
• A model that runs with part in memory says how much of it sits on your cards and how much in memory; one too big says what it needs.
• Its speed is in “words a second”, marked “estimated” until Metis has timed the model on this computer, then “measured”. Where no speed can be worked out, it says “no estimate”, and the “?” beside it says why.
• An estimate comes from the nearest model already run on this computer when one of the same family has a measured speed and the file fits on your cards: Metis scales that speed by the bytes each reads a token, and the “?” says “Scaled from a model measured here” with the figure it started from. Otherwise it comes from the maker’s published memory speed for your card. A listing gives Metis a file’s size and its family, not its quantisation or whether it is a mixture, so Metis takes the file as one that reads all of itself.
• Each estimate is kept until that file has its first measured speed, and the page then says how far estimates here have been off, as a percentage: the largest miss so far, and how many have been checked. Until one has been checked it says “No estimate has been checked against a measurement here yet.”

The room the chat needs is not counted: “the context needs room too.” A size that is not exact is marked “estimate”. The list can be sorted by “Best for this computer”. If this computer has not been measured, the badge says “Fit not checked”, and “Read again” measures it.

Under the search, “Filters” opens For (“Chat and writing”, “Reading pictures” and “Search (embeddings)” narrow the list), Fit, Size (“Any size”, “Under 10 GB”, “10 to 30 GB”, “Over 30 GB”) and, folded under “More”, licence, family and file type. The list opens on models that fit on your cards or run with part in memory; tick “Too big” to see the rest. The Fit filter reads the same placement as the badge, never a size rule of its own. Each filter in use is a chip, and pressing it takes that one off; “Clear all” takes them all off. When “No model on this page matches the filters in use.”, the list names the filter whose removal brings models back, and one press does it. The filters are kept in the address, so a reload or Back keeps them, and on a phone they open as a sheet from the bottom over a dimmed page; Escape, a press on the dim page or “Done” closes it and puts you back on the Filters button.

Get models draws one list at a time, from “Hugging Face” or from “Pirate Face”, and a switch above the search changes which. The two lists are never mixed. Metis remembers the one you chose, on every device that opens it, and your search and filters stay on the list you switch to.

On the Computers tab, a layout you are drawing has one badge, the worst of its parts: “everything fits”, “fits, tight” or “won’t fit”. “Figures are estimates”: they are worked out from each model’s file, and Launch measures them. Launch stays yours to press even when something won’t fit: “it will most likely fail there, and what’s running now comes back by itself.”