Hochmut kommt vor dem oom
Eines Tages wollte ich Ollama, das Open Source Projekt für das Betreiben von Sprachmodellen auf eigener Hardware, auf meinem Pixel 7 Smartphone betreiben. Ich verwende ja schon seit einiger Zeit Termux auf dem Handy und dem Tablet. Eine einerseits sehr lustige, aber auch etwas “kramperte” Angelegenheit. Ich habe mich also auf die Suche nach einem solchen Unterfangen, bevorzugt samt Anleitung. Ich wurde fündig, und machte mich sogleich ans Werk. Und wollte endgültig die Frage beantwortet haben: Kann mein Pixel 7 LLMs betreiben? Und wenn ja, wie schnell?
”Man bringe den Terminal”
Die folgenden Code-Blöcke zeigen diesen kleinen Ausflug ins Land von pkg, von den Niederungen von git bis in die golang-Höhen. Aber wie der Titel schon verrät, es ist eine Tragödie in einem Akt.
Welcome to Termux!
Docs: https://termux.dev/docs
Donate: https://termux.dev/donate
Community: https://termux.dev/community
Working with packages:
- Search: pkg search <query>
- Install: pkg install <package>
- Upgrade: pkg upgrade
Subscribing to additional repositories:
- Root: pkg install root-repo
- X11: pkg install x11-repo
For fixing any repository issues,
try 'termux-change-repo' command.
Report issues at https://termux.dev/issues
~ $ termux-setup-storage
~ $ pkg upgrade
No mirror or mirror group selected. You might want to select one by running 'termux-change-repo'
Testing the available mirrors:
[*] (10) https://packages-cf.termux.dev/apt/termux-main: ok
[*] (1) https://mirror.textcord.xyz/termux/termux-main: bad
[*] (1) https://mirror.nevacloud.com/applications/termux/termux-main: ok
[*] (1) https://mirrors.nguyenhoang.cloud/termux/termux-main: ok
[*] (1) https://mirror.twds.com.tw/termux/termux-main: ok
[*] (1) https://linux.domainesia.com/applications/termux/termux-main: ok
[*] (1) https://tmx.xvx.my.id/apt/termux-main: ok
[*] (1) https://mirror.freedif.org/termux/termux-main: bad
[*] (1) https://mirrors.cbrx.io/apt/termux/termux-main: ok
[*] (1) https://mirrors.in.sahilister.net/termux/termux-main/: ok
[*] (1) https://mirror.albony.in/termux/termux-main: ok
[*] (1) https://mirror.bardia.tech/termux/termux-main: bad
[*] (1) https://mirrors.sdu.edu.cn/termux/termux-main: ok
[*] (1) https://mirrors.sau.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.nju.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.sustech.edu.cn/termux/apt/termux-main: ok
[*] (1) https://mirrors.pku.edu.cn/termux/termux-main/: ok
[*] (1) https://mirror.sjtu.edu.cn/termux/termux-main/: ok
[*] (1) https://mirror.iscas.ac.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.ustc.edu.cn/termux/termux-main: ok
[*] (1) https://mirrors.aliyun.com/termux/termux-main: bad
[*] (1) https://mirrors.qvq.net.cn/termux/termux-main: bad
[*] (1) https://mirrors.cqupt.edu.cn/termux/termux-main: bad
[*] (1) https://mirrors.zju.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.hust.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.bfsu.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirror.nyist.edu.cn/termux/apt/termux-main: bad
[*] (1) https://mirrors.tuna.tsinghua.edu.cn/termux/apt/termux-main: bad
[*] (1) https://ftp.agdsn.de/termux/termux-main: bad
[*] (4) https://grimler.se/termux/termux-main: bad
[*] (1) https://ro.mirror.flokinet.net/termux/termux-main: bad
[*] (1) https://mirrors.cfe.re/termux/termux-main: bad
[*] (1) https://mirror.termux.dev/termux-main: bad
[*] (1) https://termux.3san.dev/termux/termux-main: bad
[*] (1) https://mirror.bouwhuis.network/termux/termux-main: bad
[*] (1) https://mirror.accum.se/mirror/termux.dev/termux-main: bad
[*] (1) https://termux.mentality.rip/termux-main: bad
[*] (1) https://is.mirror.flokinet.net/termux/termux-main: bad
[*] (1) https://md.mirrors.hacktegic.com/termux/termux-main: bad
[*] (1) https://termux.astra.in.ua/apt/termux-main: bad
[*] (1) https://mirror.mwt.me/termux/main: bad
[*] (1) https://mirrors.medzik.dev/termux/termux-main: bad
[*] (1) https://packages.termux.dev/apt/termux-main: bad
[*] (1) https://mirror.sunred.org/termux/termux-main: bad
[*] (1) https://mirrors.de.sahilister.net/termux/termux-main: bad
[*] (1) https://mirror.leitecastro.com/termux/termux-main: bad
[*] (1) https://termux.cdn.lumito.net/termux-main: bad
[*] (1) https://nl.mirror.flokinet.net/termux/termux-main: bad
[*] (1) https://ftp.fau.de/termux/termux-main: bad
[*] (1) https://termux.librehat.com/apt/termux-main: bad
[*] (1) https://mirror.autkin.net/termux/termux-main: bad
[*] (1) https://mirror.polido.pt/termux/termux-main: bad
[*] (1) https://dl.kcubeterm.com/termux-main: bad
[*] (1) https://termux.danyael.xyz/termux/termux-main: bad
[*] (1) https://mirror.mwt.me/termux/main: bad
[*] (1) https://mirrors.utermux.dev/termux/termux-main: bad
[*] (1) https://mirror.vern.cc/termux/termux-main: bad
[*] (1) https://plug-mirror.rcac.purdue.edu/termux/termux-main: bad
[*] (1) https://mirror.quantum5.ca/termux/termux-main: bad
[*] (1) https://mirror.fcix.net/termux/termux-main: bad
[*] (1) https://mirror.csclub.uwaterloo.ca/termux/termux-main: bad
[*] (1) https://mirror.endianness.com/termux/termux-main: bad
[*] (1) https://mirrors.rda.run/termux/termux-main: bad
[*] (1) https://repository.su/termux/termux-main/: bad
[*] (1) http://mirror.mephi.ru/termux/termux-main: bad
Picking mirror: (5) /data/data/com.termux/files/usr/etc/termux/mirrors/default
Get:1 https://packages-cf.termux.dev/apt/termux-main stable InRelease [14.0 kB]
Get:2 https://packages-cf.termux.dev/apt/termux-main stable/main aarch64 Packages [533 kB]
Fetched 547 kB in 4s (133 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
49 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
libandroid-selinux
The following packages will be upgraded:
apt bash ca-certificates command-not-found coreutils
curl debianutils diffutils dpkg ed findutils gpgv
inetutils ldns less libandroid-support libassuan
libc++ libcurl libgcrypt libgmp libgnutls
libgpg-error liblz4 liblzma libnettle libnghttp2
libnghttp3 libsmartcols libssh2 libtirpc libunbound
libunistring lsof nano ncurses openssh
openssh-sftp-server openssl patch readline termux-am
termux-auth termux-keyring termux-tools util-linux
xxhash xz-utils zstd
49 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 16.3 MB of archives.
After this operation, 1946 kB of additional disk space will be used.
Do you want to continue? [Y/n]
{...}
Configuration file '/data/data/com.termux/files/usr/etc/apt/sources.list'
==> File on system created by you or by a script.
==> File also in package provided by package maintainer.
What would you like to do about it ? Your options are:
Y or I : install the package maintainers version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : start a shell to examine the situation
The default action is to keep your current version.
*** sources.list (Y/I/N/O/D/Z) [default=N] ? y
Installing new version of config file /data/data/com.termux/files/usr/etc/apt/sources.list ...
(Reading database ... 5116 files and directories currently installed.)
{...}
Installing new version of config file /data/data/com.termux/files/usr/etc/ssh/ssh_config ...
Installing new version of config file /data/data/com.termux/files/usr/etc/ssh/sshd_config ...Als nächstes werden die Pakete git, cmake und golang installiert:
~ $ pkg install git cmake golang
No mirror or mirror group selected. You might want to select one by running 'termux-change-repo'
Checking availability of current mirror:
[*] https://packages-cf.termux.dev/apt/termux-main/: ok
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
clang jsoncpp libarchive libcompiler-rt libexpat
libffi libllvm libuv libxml2 lld llvm make
ndk-sysroot rhash
Suggested packages:
perl
The following NEW packages will be installed:
clang cmake git golang jsoncpp libarchive
libcompiler-rt libexpat libffi libllvm libuv libxml2
lld llvm make ndk-sysroot rhash
0 upgraded, 17 newly installed, 0 to remove and 0 not upgraded.
Need to get 130 MB of archives.
After this operation, 800 MB of additional disk space will be used.
Do you want to continue? [Y/n]
{...}Jetzt geht es ans Eingemachte, nämlich an das Ollama-Repository:
~ $ git clone --depth 1 https://github.com/ollama/ollama.git
Cloning into 'ollama'...
remote: Enumerating objects: 897, done.
remote: Counting objects: 100% (897/897), done.
remote: Compressing objects: 100% (700/700), done.
Receiving objects: 26% (234/897), 652.00 KiB | 1.24 MiBReceiving objects: 27% (243/897), 652.00 KiB | 1.24 MiBReceiving objects: 28% (252/897), 652.00 KiB | 1.24 MiBReceiving objects: 29% (261/897), 652.00 KiB | 1.24 MiBReceiving objects: 30% (270/897), 652.00 KiB | 1.24 MiBReceiving objects: 30% (271/897), 652.00 KiB | 1.24 MiBReceiving objects: 31% (279/897), 1.02 MiB | 1023.00 KiReceiving objects: 32% (288/897), 1.02 MiB | 1023.00 KiReceiving objects: 33% (297/897), 1.02 MiB | 1023.00 KiReceiving objects: 34% (305/897), 1.02 MiB | 1023.00 KiReceiving objects: 35% (314/897), 1.02 MiB | 1023.00 KiReceiving objects: 36% (323/897), 1.02 MiB | 1023.00 KiReceiving objects: 37% (332/897), 1.02 MiB | 1023.00 KiReceiving objects: 38% (341/897), 1.02 MiB | 1023.00 KiReceiving objects: 39% (350/897), 1.02 MiB | 1023.00 KiReceiving objects: 40% (359/897), 1.02 MiB | 1023.00 KiReceiving objects: 41% (368/897), 1.02 MiB | 1023.00 KiReceiving objects: 42% (377/897), 1.02 MiB | 1023.00 KiReceiving objects: 43% (386/897), 1.02 MiB | 1023.00 KiReceiving objects: 44% (395/897), 1.63 MiB | 1.08 MiB/sremote: Total 897 (delta 168), reused 742 (delta 163), pack-reused 0 (from 0)
Receiving objects: 100% (897/897), 5.54 MiB | 2.30 MiB/s, done.
Resolving deltas: 100% (168/168), done.In den Projektordner wechseln und go generate ./... ausführen:
~ $ cd ollama
~/ollama $ go generate ./...
go: downloading github.com/google/uuid v1.6.0
go: downloading golang.org/x/crypto v0.31.0
go: downloading google.golang.org/protobuf v1.34.1
go: downloading github.com/containerd/console v1.0.3
go: downloading github.com/mattn/go-runewidth v0.0.14
go: downloading github.com/d4l3k/go-bfloat16 v0.0.0-20211005043715-690c3bdd05f1
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/nlpodyssey/gopickle v0.3.0
go: downloading github.com/spf13/cobra v1.7.0
go: downloading golang.org/x/sync v0.10.0
go: downloading golang.org/x/term v0.27.0
go: downloading github.com/pdevine/tensor v0.0.0-20240510204454-f88f4562727c
go: downloading github.com/dlclark/regexp2 v1.11.4
go: downloading golang.org/x/image v0.22.0
go: downloading github.com/x448/float16 v0.8.4
go: downloading github.com/emirpasic/gods/v2 v2.0.0-alpha
go: downloading golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa
go: downloading github.com/gin-gonic/gin v1.10.0
go: downloading golang.org/x/text v0.21.0
go: downloading gonum.org/v1/gonum v0.15.0
go: downloading github.com/agnivade/levenshtein v1.1.1
go: downloading github.com/gin-contrib/cors v1.7.2
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading golang.org/x/sys v0.28.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading golang.org/x/net v0.25.0
go: downloading github.com/apache/arrow/go/arrow v0.0.0-20211112161151-bc219186db40
go: downloading github.com/chewxy/hm v1.0.0
go: downloading github.com/chewxy/math32 v1.11.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/google/flatbuffers v24.3.25+incompatible
go: downloading go4.org/unsafe/assume-no-moving-gc v0.0.0-20231121144256-b99613f794b6
go: downloading gorgonia.org/vecf32 v0.9.0
go: downloading gorgonia.org/vecf64 v0.9.0
go: downloading github.com/go-playground/validator/v10 v10.20.0
go: downloading github.com/pelletier/go-toml/v2 v2.2.2
go: downloading github.com/ugorji/go/codec v1.2.12
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/xtgo/set v1.0.0
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/golang/protobuf v1.5.4
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading github.com/go-playground/universal-translator v0.18.1
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading github.com/leodido/go-urn v1.4.0
go: downloading github.com/go-playground/locales v0.14.1Und dann go build ., gibt es nichts schöneres als am Smartphone in einer Konsole etwas zu Compilen? Ich denke auch nicht:
~/ollama $ go build .
# github.com/ollama/ollama/discover
gpu_info_cudart.c:61:13: warning: comparison of different enumeration types ('cudartReturn_t' (aka 'enum cudartReturn_enum') and 'enum cudaError_enum') [-Wenum-compare]
# github.com/ollama/ollama/ml/backend/ggml/ggml/src
ggml-backend-reg.cpp:438:1: warning: non-void function does not return a value in all control paths [-Wreturn-type]
~/ollama $ ./ollama serve &
[1] 28496
~/ollama $ Couldnt find '/data/data/com.termux/files/home/.ollama/id_ed25519'. Generating new private key.
Your new public key is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5TROLOLOLoliJGpD3fbCe0aekl3SY50jyn7D4PO/lYNpmOoj7Lr
2025/02/22 12:19:01 routes.go:1205: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/data/data/com.termux/files/home/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-02-22T12:19:01.072Z level=INFO source=images.go:432 msg="total blobs: 0"
time=2025-02-22T12:19:01.072Z level=INFO source=images.go:439 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
{...}
time=2025-02-02T12:19:01.073Z level=INFO source=routes.go:1256 msg="Listening on 127.0.0.1:11434 (version 0.0.0)"
time=2025-02-02T12:19:01.077Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-02-02T12:19:01.080Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-02-02T12:19:01.080Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="7.3 GiB" available="658.1 MiB"
{...}“No RAM today, my random access’ gone away”
Und hier nahm meine Smartphone-Ollama-Karriere ein jähes Ende. Es war der RAM! Das Hightech-Kastl für die Hosentasche hat mit seinen 8 GB Arbeitsspeicher einfach nicht genug frei für Ollama. Nicht einmal das kleinste Modell ward mir vergönnt:
~/ollama $ ./ollama run llama3.2:3b --verbose
(...)
pulling 34bb5ab01051... 100% ▕▏ 561 B
verifying sha256 digest
writing manifest
success
[GIN] 2025/02/02 - 12:43:08 | 200 | 47.173014ms |
127.0.0.1 | POST "/api/show" ⠋
time=2025-02-02T12:43:08.944Z level=INFO source=server.go:97 msg="system memory" total="7.3 GiB" free="701.3 MiB" free_swap="0 B"
time=2025-02-02T12:43:08.945Z level=WARN source=server.go:125 msg="model request too large for system" requested="2.8 GiB" available=735334400 total="7.3 GiB" free="701.3 MiB" swap="0 B"
time=2025-02-02T12:43:08.946Z level=INFO source=sched.go:429 msg="NewLlamaServer failed"
model=/data/data/com.termux/files/home/.ollama/models/blobs/sha256-dde5aa3fc5ffc17176b5e8bdc82f587b24b2678c6c66101bf7da77af9f7ccdff error="model requires more system memory (2.8 GiB) than is available (701.3 MiB)" [GIN] 2025/02/02 - 12:43:08 | 500 | 144.352824ms | 127.0.0.1 | POST "/api/generate"
Error: model requires more system memory (2.8 GiB) than is available (701.3 MiB)Keine Ahnung weshalb ein mobiles Betriebssystem wie Android nach einem Neustart gleich einmal 5GB vom Arbeitsspeicher belegt. Das ist ja ärger als im Hause Microsoft. Somit ist mir vorerst der Weg zur mobilen Textgenerierung (im Volksmund auch als “Gschichtlsdrucker” bekannt) verbaut und der Weg zu einem 16GB-Smartphone zumindest mental frei, wenn auch das Budget noch nicht genehmigt ist.
Damit endet dieses kleine Tragödie. Vorerst.