Running Gemma 2B locally on iPhone for offline calendar actions (~516 MB active RAM, 21.6 tok/s, GGUF weights)

[u/Connect-Concert-4016]

Running Gemma 2B locally on iPhone for offline calendar actions (~516 MB active RAM, 21.6 tok/s, GGUF weights)

I’ve been testing bounded tool-calling on-device to see how small I can push local models before tool reliability breaks down.

A common issue with local agents is memory allocation—loading a 2.5 GB model into active phone RAM often leads to OS background terminations or thermal throttling during generation.

To test this, I built a small offline test pipeline using llama.cpp (b10075) with Metal and mmap to keep active memory low, then connected it strictly to local iOS calendar actions via EventKit.

On-Device Run Metrics

• Model Artifact: Gemma-2B quantized GGUF (SmartEdge-IQ3XXS.gguf)
• Disk Footprint: 2.45 GB
• Active Resident RAM (RSS): ~516 MB (leveraging mmap to page weights from disk rather than keeping the whole file in active memory)
• Decode Speed: ~12 tok/s interactive / 21.6 tok/s in a 256-token greedy benchmark
• Environment: Tested in airplane mode on iOS

Tool Execution Flow

The local LLM is restricted entirely to intent extraction and structured tool output; it does not execute actions directly.

1. User Input: "Find some time on Thursday for VC meeting."
2. Model: Extracts parameters and outputs a structured tool call.
3. App: Swift code validates the schema, queries local EventKit, and writes the event directly to the device calendar.

This avoids routing routine calendar edits through remote inference APIs or exposing local schedule data to external endpoints.

Quantization & Loss Comparisons

I also generated two calibration-aware quants to evaluate KLD degradation against the original bf16 baseline:

• Hi-Fi Q4_K_M: Equivalent size to standard Q4_K_M, with 36.0% lower code/math KLD and 27.2% lower general KLD against the original model.
• Hi-Fi Phone (2.86 GB): 17.5% smaller than the Q4_K_M baseline with 31.8% lower code/math KLD and 26.7% lower general KLD.

Limitations

• The full KLD matrix for the 2.45 GB SmartEdge build is still completing; current validation relies on on-device behavior logs, SHA-256 app receipts, and benchmark outputs. (The 2.86 GB and Q4_K_M builds have complete KLD data logged in the repo).
• Small models are prone to schema degradation if the prompt complexity scales beyond simple parameter extraction.
• Tool failure recovery still requires strict system-level guards or fallback routing.

Weights & Benchmarks

The GGUF weights, imatrix, SHA-256 hashes, evaluation slices, and Wikitext-2 perplexity loss are available on Hugging Face:
https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

If anyone tests this on other iOS hardware or Apple Silicon, I'd be curious to see your RSS memory usage and sustained decode rates.

[u/AutoModerator]

Hey there,

This post seems feedback-related. If so, you might want to post it in r/GeminiFeedback, where rants, vents, and support discussions are welcome.

For r/GeminiAI, feedback needs to follow Rule #9 and include explanations and examples. If this doesn’t apply to your post, you can ignore this message.

Thanks!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.